A deep learning-based tobacco variety automatic classification method and system

Abstract

The application discloses a tobacco variety automatic classification method and system based on deep learning, and belongs to the technical field of tobacco detection. Visible light images and near-infrared spectra of tobacco leaves are collected, preprocessed and enhanced, then weighted features are extracted and spliced and fused through an image branch with mixed gradient domain attention and a spectrum branch with wave band sparse attention; a deep support vector data description single-class classifier is trained for each known variety, and parallel integration is realized, open set recognition and modular incremental updating are realized; the integrated model is used as a teacher, a lightweight student model is obtained through knowledge distillation and quantization compression, and is deployed to an embedded device; a confidence weighted fusion rule is used for real-time recognition to output a final classification result. The application fuses multi-modal information, can recognize unknown varieties, supports flexible updating of a variety library, is suitable for real-time operation on an edge side, and is high in accuracy and strong in robustness.

Description

Technical Field

[0001] This invention belongs to the field of tobacco detection technology, specifically relating to an automatic tobacco variety classification method and system based on deep learning. Background Technology

[0002] Traditional methods for identifying tobacco varieties primarily rely on manual experience or chemical composition analysis. Manual judgment depends on expert visual and tactile experience, is highly subjective, inefficient, and lacks standardized criteria. While chemical composition analysis, such as chromatography and mass spectrometry, offers high precision, it is cumbersome, time-consuming, and destructive, failing to meet the demands of large-scale, rapid detection. In recent years, visible light imaging and near-infrared spectroscopy have been explored for tobacco variety identification, but existing solutions often employ single-modal data, failing to comprehensively characterize the coupling differences between the external morphology and internal chemical composition of tobacco leaves. Furthermore, traditional classification models are closed-set classifications, forcibly classifying unknown tobacco varieties not included in the training set into a known category, leading to serious misclassifications. In addition, complex deep learning models are difficult to deploy directly on portable embedded devices, limiting their application in real-time field detection. Therefore, there is an urgent need for an automatic tobacco variety classification method that integrates multimodal information, supports open-set recognition, and is lightweight and deployable. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this invention provides an automatic tobacco variety classification method and system based on deep learning.

[0004] The objective of this invention can be achieved through the following technical solutions: In a first aspect embodiment of the present invention, a method for automatic tobacco variety classification based on deep learning is provided, comprising: Visible light images and near-infrared spectral data of multiple known tobacco varieties were collected, and a dataset was formed after preprocessing and data augmentation. The dual-branch feature extraction network includes an image branch and a spectral branch; the image branch uses a convolutional neural network and introduces a hybrid gradient domain attention module to output weighted image features; the spectral branch uses a one-dimensional convolutional neural network or a lightweight multilayer perceptron and introduces a band attention and sparse constraint module to output weighted spectral features. The weighted image features and the weighted spectral features are concatenated and fused to obtain the fused features; For each known variety, a single-class classifier based on deep support vector data description is trained, and all single-class classifiers are integrated in parallel to obtain a variety identification model; the variety is determined as known or unknown based on the comparison results of the distance and radius from the fusion features of the sample to be identified to the center of each hypersphere; when adding or eliminating varieties, the corresponding single-class classifier is added or removed accordingly. Using the variety identification model as the teacher model and the lightweight neural network as the student model, knowledge from the teacher model is transferred to the student model through knowledge distillation. The trained student model is then quantized, compressed, and deployed to an embedded device. Visible light images and near-infrared spectra of tobacco leaves are acquired simultaneously and input into a student model deployed on an embedded device to obtain image recognition results and spectral recognition results. The final classification result is obtained by using a confidence-weighted fusion rule.

[0005] In a further embodiment of the present invention, the hybrid gradient domain attention module specifically includes: extracting texture gradient maps of the input feature map in parallel using five edge detection operators: Sobel, Scharr, Laplace, Roberts, and Prewitt; concatenating the five gradient maps in the channel dimension; performing feature extraction and channel restoration through convolutional layers to obtain an intermediate feature map; and then encoding the intermediate feature map using a pyramid module to generate an attention weight map; the pyramid module includes multiple convolutional layers, max pooling layers, and deconvolutional layers.

[0006] In a further embodiment of the present invention, the band attention and sparsity constraint module specifically includes: generating original attention weights for each band using a two-layer fully connected neural network, mapping them to a preset interval via an activation function; sorting the generated attention weights by value and setting a weight threshold, assigning weights smaller than the weight threshold to preset values ​​to obtain sparsified band weights; multiplying the sparsified band weights by the original spectral data to obtain weighted spectral features; wherein the weight threshold is determined according to a preset number or proportion of retained bands.

[0007] In a further embodiment of the present invention, the training loss function of the single-class classifier described by deep support vector data includes a first term and a second term. The first term is used to constrain the distance between the fused features of all samples and the center of the preset hypersphere, and the second term is the regularization term of the network weights. The radius of the hypersphere is taken as the preset quantile of the distance from all samples in the training set to the center of the hypersphere.

[0008] In a further embodiment of the present invention, the loss function of knowledge distillation includes the cross-entropy loss of the student model on the real label and the soft label cross-entropy loss of the student model and each teacher model under a preset temperature parameter, wherein the loss terms of different teacher models are multiplied by their corresponding balance weights; when the training sample belongs to the variety corresponding to a certain teacher model, the balance weight corresponding to that teacher model is greater than the balance weights of other teacher models.

[0009] In a further embodiment of the present invention, the confidence weighted fusion rule is as follows: the confidence vectors corresponding to the image recognition result and the spectral recognition result are multiplied by a preset first weight and a preset second weight, respectively, and then added together to obtain a fused confidence vector, wherein the sum of the first weight and the second weight is a set value; when the variety categories corresponding to the image recognition result and the spectral recognition result are consistent, the category and the corresponding value in the fused confidence vector are output; when they are inconsistent, the category corresponding to the mode with higher confidence is output.

[0010] In a further embodiment of the present invention, the preprocessing of the visible light image includes: image size scaling, pixel normalization, brightness equalization, and noise reduction; the preprocessing of the near-infrared spectral data includes: sliding window smoothing, baseline correction, multivariate scattering correction, and derivative processing; the data enhancement includes: random cropping, rotation, brightness adjustment, noise addition, and linear superposition of the image; and translation, addition of random noise, and linear superposition of similar samples of the spectrum.

[0011] In a further embodiment of the present invention, the system operation and anomaly management steps are also included: after the device is powered on, it automatically completes hardware initialization, model loading and spectral sensor connection verification; a built-in anomaly capture daemon process captures anomalies in the camera, spectral sensor and model inference process and automatically restarts; the image results, spectral curves, fusion results, timestamps and device numbers of each recognition are automatically saved to local storage; and a built-in dataset balancing verification module is used to analyze the distribution of the number of samples in each category before model training and output data bias warnings.

[0012] In a further embodiment of the present invention, the wavelength range of the near-infrared spectrum is the near-infrared band, the embedded device is an embedded development board, and the student model is deployed after quantization and compression.

[0013] In a second aspect embodiment of the present invention, a deep learning-based automatic tobacco variety classification system is provided for executing a deep learning-based automatic tobacco variety classification method according to a first aspect embodiment of the present invention, comprising: Data acquisition unit: includes a visible light camera and a near-infrared spectral sensor, used to acquire visible light images and near-infrared spectral data of tobacco leaves; Embedded main control unit: It adopts an embedded development board, runs an embedded operating system, and deploys a lightweight classification model after knowledge distillation and quantization compression; Preprocessing unit: used to perform preprocessing and data augmentation on the acquired visible light images and near-infrared spectral data; The dual-modal feature extraction and fusion unit includes a hybrid gradient domain attention module for the image branch, a band attention and sparse constraint module for the spectral branch, and a feature fusion layer, which are used to extract and fuse weighted image features and weighted spectral features respectively. Open set classification and incremental learning unit: Multiple single-class classifiers are constructed and integrated in parallel based on deep support vector data description, which are used to identify known tobacco varieties, reject unknown varieties, and support modular incremental updates of the variety library; Knowledge distillation and model compression unit: used to transfer the knowledge of the integrated complex teacher model to the lightweight student model, and to perform quantitative compression on the student model; Human-computer interaction and display unit: used to display recognition results, spectral curves, confidence levels and system status in real time; Data storage and log unit: used to store identification history, spectral data, operation logs and equipment information; Anomaly Protection and Ethics Compliance Unit: Used to achieve fault self-recovery, data bias verification, and access control.

[0014] The present invention has at least the following beneficial effects: 1. Simultaneously acquire visible light images and near-infrared spectra, extract texture and chemical features through a dual-branch attention network, and achieve a higher classification accuracy than single-modality models after fusion.

[0015] 2. By constructing a multi-class classifier in parallel based on deep support vector data description, it is possible to reliably reject unknown varieties, avoid forced classification errors, and ensure the credibility of detection results.

[0016] 3. When adding or removing varieties, only the corresponding single-class classifier needs to be added or removed independently, without retraining the overall model, thus reducing the maintenance cost and time of the variety library. Attached Figure Description

[0017] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of a method provided in one embodiment of the present invention; Figure 2 This is a flowchart of the operation of the hybrid gradient domain attention module provided in one embodiment of the present invention; Figure 3 This is a flowchart of the operation of the band attention and sparsity constraint module provided in one embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0020] Please refer to Figure 1-3 In a first aspect, the present invention provides an automatic tobacco variety classification method based on deep learning, comprising: Visible light images and near-infrared spectral data of multiple known tobacco varieties were collected, and a dataset was formed after preprocessing and data augmentation. The dual-branch feature extraction network includes an image branch and a spectral branch; the image branch uses a convolutional neural network and introduces a hybrid gradient domain attention module to output weighted image features; the spectral branch uses a one-dimensional convolutional neural network or a lightweight multilayer perceptron and introduces a band attention and sparse constraint module to output weighted spectral features. The weighted image features and the weighted spectral features are concatenated and fused to obtain the fused features; For each known variety, a single-class classifier based on deep support vector data description is trained, and all single-class classifiers are integrated in parallel to obtain a variety identification model; the variety is determined as known or unknown based on the comparison results of the distance and radius from the fusion features of the sample to be identified to the center of each hypersphere; when adding or eliminating varieties, the corresponding single-class classifier is added or removed accordingly. Using the variety identification model as the teacher model and the lightweight neural network as the student model, knowledge from the teacher model is transferred to the student model through knowledge distillation. The trained student model is then quantized, compressed, and deployed to an embedded device. Visible light images and near-infrared spectra of tobacco leaves are acquired simultaneously and input into a student model deployed on an embedded device to obtain image recognition results and spectral recognition results. The final classification result is obtained by using a confidence-weighted fusion rule.

[0021] In tobacco agriculture and industrial production, different tobacco varieties possess distinct chemical compositions and sensory qualities, directly impacting their economic value and formulation applications. Traditional tobacco variety identification relies primarily on manual experience and chemical analysis. The former is heavily influenced by subjective factors and inefficient, while the latter is time-consuming, destructive, and costly. With the development of spectral imaging and deep learning technologies, multimodal data fusion has become an effective way to improve classification accuracy. However, existing technologies often face two major challenges: first, there is a non-linear correlation between the appearance and internal components of tobacco leaves, making it difficult for a single modality to comprehensively represent varietal differences; second, new varieties may emerge in real-world applications not included in the training set, and traditional classification models can only force them into a known category, leading to misclassification. Furthermore, the high model complexity makes deployment on edge computing devices difficult, affecting real-time performance and portability. Therefore, there is an urgent need for an automatic tobacco variety classification method that can comprehensively utilize visible light images and near-infrared spectra, support open set recognition, and be lightweight and deployable.

[0022] In this embodiment, firstly, visible light images and near-infrared spectral data of multiple known tobacco varieties are acquired. For example, 10 common varieties, such as K326, Yunyan 87, and Honghua Dajinyuan, are selected, and 500 fresh tobacco leaf samples are collected for each variety. Visible light images are captured using an industrial camera in a standard light source box with a resolution of 224×224 pixels; near-infrared spectra are acquired using a portable spectrometer in the 900-1700nm band at 2nm intervals. The images are scaled to a uniform size, pixel values ​​are normalized to [0,1], histogram equalization is used to correct brightness unevenness, and median filtering is applied for noise reduction. The spectral data are then processed sequentially using sliding window smoothing, polynomial baseline correction, multivariate scattering correction, and first derivative processing. In the data augmentation stage, each image is randomly rotated ±15°, randomly cropped to 200×200 and then scaled back to the original size, the brightness factor is randomly adjusted between 0.8 and 1.2, Gaussian noise is added, and the image is linearly superimposed with another image of the same variety with a weight of 0.5. Each spectrum was shifted ±2 bands along the wavelength axis, random noise with an amplitude of 0.01 was added, and it was linearly superimposed with similar spectra. The final result was a dataset containing 3000 enhanced samples for each variety.

[0023] Secondly, a dual-branch feature extraction network is constructed. The image branch uses ResNet-34 as the backbone network, and a hybrid gradient domain attention module is introduced after its last residual block. This module first uses five operators—Sobel, Scharr, Laplace, Roberts, and Prewitt—in parallel to extract the gradient information of the input feature map. After concatenating the channels of the five gradient maps, it is reduced to the original number of channels by a 3×3 convolution. Then, a three-layer pyramid structure is used to generate spatial attention weights, which are multiplied element-wise with the original feature map to output a weighted image feature vector. The spectral branch uses a one-dimensional convolutional neural network: three convolutional layers, each followed by batch normalization and ReLU activation, and then connected to a band attention and sparse constraint module. This module uses global average pooling to compress spectral features into a 256-dimensional vector. It generates the original weights for each band through two fully connected layers, and maps them to [0,1] using Sigmoid. The weights are sorted in descending order, and a weight threshold corresponding to a retention ratio of 30% is set. Weights below the threshold are reset to 0 to obtain sparse band weights. The weighted spectral features are then multiplied band by band by band of the input spectrum and output as weighted spectral features.

[0024] Then, the weighted image features and weighted spectral features are concatenated along the channel dimension to obtain a fusion feature vector with dimension 768.

[0025] Next, a single-class classifier based on deep support vector data description is trained for each known variety. Each classifier consists of a three-layer fully connected network, and the loss function during training is the sum of the squared distances of all samples to the center c of a preset hypersphere plus a weighted L2 regularization term. The radius R of the hypersphere is taken as the 95th percentile of the distances from all samples in the training set to the center. All single-class classifiers are ensembled in parallel: the fused features of the samples to be identified are input into each classifier. If there exists a classifier whose distance from the feature to its center is ≤ R, it is determined to be the corresponding known variety; if all distances are greater than their respective radii, it is determined to be an unknown variety. When a new variety needs to be added, only a new single-class classifier is trained and added to the ensemble; when a variety is eliminated, the corresponding classifier is directly removed.

[0026] Next, using the integrated variety identification model as the teacher model, a lightweight student model was designed: The MobileNetV3-Small architecture was used to process images, with the spectral branch employing two one-dimensional depthwise separable convolutions, a fusion layer outputting 192 dimensions, and finally a 10-dimensional fully connected layer. During knowledge distillation, the temperature parameter T=3, and the loss function included the student model's cross-entropy loss on the true labels, as well as the sum of the KL divergences between the soft outputs of the student model and the soft outputs of each teacher model. For samples belonging to variety i, the balanced weight of the teacher model corresponding to variety i was set to 0.9, and the weights of the other teacher models were all set to 0.1 / 9≈0.0111. After training, the student model was compressed using 8-bit integer quantization and deployed to an embedded development board based on ARM Cortex-A72.

[0027] Finally, in practical applications, visible light images and near-infrared spectra of tobacco leaves are simultaneously acquired and input into the deployed student model to obtain image recognition results and spectral recognition results. Assuming image weight α = 0.4 and spectral weight β = 0.6 (α + β = 1), the two confidence vectors are weighted and summed to obtain a fused confidence vector. If the categories identified by the image and spectral recognitions are consistent, the category and fused confidence are output; otherwise, the category corresponding to the mode with the higher confidence is output.

[0028] In a further embodiment of the present invention, the hybrid gradient domain attention module specifically includes: extracting texture gradient maps of the input feature map in parallel using five edge detection operators: Sobel, Scharr, Laplace, Roberts, and Prewitt; concatenating the five gradient maps in the channel dimension; performing feature extraction and channel restoration through convolutional layers to obtain an intermediate feature map; and then encoding the intermediate feature map using a pyramid module to generate an attention weight map; the pyramid module includes multiple convolutional layers, max pooling layers, and deconvolutional layers.

[0029] In this embodiment, the hybrid gradient domain attention module is embedded in the intermediate layer of the image branch convolutional neural network, receiving the feature map output from the preceding convolutional layer as input. First, five classic edge detection operators are used in parallel to process the input feature map: Sobel, Scharr, Laplace, Roberts, and Prewitt. Each operator calculates the gradient response of the feature map in the horizontal and vertical directions, and then synthesizes the gradient magnitude map. The Sobel operator is computationally simple and efficient, providing good response to general edges; the Scharr operator is more accurate than Sobel, providing more precise gradient estimation; the Laplace operator, being a second-order differential operator, provides more accurate edge localization and can detect more subtle texture changes; the Roberts operator uses a small kernel, is sensitive to detailed edges, and is suitable for capturing minute textures; the Prewitt operator is similar to Sobel but slightly different in terms of noise resistance. The five operators run in parallel, each independently outputting a gradient response map. Since the output dimensions of the five operators are the same, these gradient maps are sequentially concatenated along the channel dimension to form a multi-channel composite gradient feature map.

[0030] Next, the concatenated composite gradient feature map is processed by convolution. A convolutional layer with a moderate kernel size, five input channels, and the same number of output channels as the original feature map is restored. This convolutional layer adaptively learns the nonlinear combination relationship between the outputs of different edge detection operators, fusing the five gradient information into a more expressive intermediate feature map. Through training this layer, the network can automatically assign optimal operator combination weights to different tobacco varieties: for varieties with straight and clear veins, the Sobel and Prewitt operators may contribute more; for varieties with finely serrated leaf edges, the weights of the Laplace and Roberts operators will be increased accordingly. Batch normalization and a nonlinear activation function are then applied after the convolutional layer to accelerate convergence and enhance nonlinear expressive power.

[0031] Then, a pyramid module is used to encode the intermediate feature maps at multiple scales to generate an attention weight map. The pyramid module processes features at multiple scales simultaneously: one branch directly processes the intermediate feature map at its original size, preserving detailed information; another branch first reduces the feature map size by half through max pooling, then performs convolution and deconvolution to upsample it back to its original size, capturing a slightly larger area of ​​texture structure; a third branch further reduces the feature map to one-quarter of its original size, performs convolution and upsampling, obtaining a larger receptive field capable of perceiving the overall texture distribution pattern of the leaf. The feature maps output from the three branches are concatenated along the channel dimension, then compressed to one dimension through a convolutional layer. Finally, an activation function maps the value at each spatial location to a value between zero and one, forming the final attention weight map. The value at each position in this weight map represents the feature importance of the corresponding position in the original feature map.

[0032] Finally, the original input feature map is multiplied element-wise with the generated attention weight map, meaning that all channel features at each spatial location are multiplied by the same weight value. This enhances the more important texture regions while suppressing less important background or smooth areas. The weighted feature map is then fed into subsequent network layers, enabling the entire network to focus on the most discriminative texture details of the tobacco variety.

[0033] In practice, this module can be flexibly inserted into multiple different locations within a convolutional neural network. Insertion into shallow and mid-level layers typically yields better results because the feature maps at these levels retain more spatial texture information. During training, the parameters of this module are jointly optimized with the entire network, eliminating the need for separate pre-training.

[0034] In a further embodiment of the present invention, the band attention and sparsity constraint module specifically includes: generating original attention weights for each band using a two-layer fully connected neural network, mapping them to a preset interval via an activation function; sorting the generated attention weights by value and setting a weight threshold, assigning weights smaller than the weight threshold to preset values ​​to obtain sparsified band weights; multiplying the sparsified band weights by the original spectral data to obtain weighted spectral features; wherein the weight threshold is determined according to a preset number or proportion of retained bands.

[0035] In this embodiment, the band attention and sparsity constraint module is located at the front or middle of the spectral branch, receiving preprocessed raw spectral data or mid-layer spectral features as input. The input spectral data is a one-dimensional vector with a length equal to the number of wavelength points, where each element represents the spectral reflectance or absorbance value at that wavelength.

[0036] The first step of the module is to generate the original attention weights. The input spectral vector first undergoes a global pooling operation, compressing the entire spectrum into a fixed-length feature vector. This vector then passes through two fully connected layers: the first fully connected layer compresses the input dimension to a smaller dimension and uses a non-linear activation function to increase expressive power; the second fully connected layer restores the dimension to the same number of original spectral bands. This bottleneck structure forces the network to learn the non-linear relationships between spectral bands, rather than a simple linear weighting. After the second layer, each output value is mapped to a finite interval through an activation function, resulting in the original attention weight vector. Each element in this vector corresponds to an initial importance score for a spectral band, but at this point, the weights of all bands are mostly non-zero positive numbers, exhibiting a dense distribution.

[0037] The second step of the module is sparsification. All elements in the original attention weight vector are sorted in descending order of value. A weight threshold is determined based on a preset retention ratio or absolute retention quantity. For example, if the retention ratio is set at 30%, the smallest weight in the first 30% after sorting is used as the threshold; if the goal is to retain 100 out of 300 bands, the weight at the 100th position after sorting is used as the threshold. After obtaining the threshold, each element in the original weight vector is evaluated: if the element value is greater than or equal to the threshold, its original value remains unchanged; if the element value is less than the threshold, it is forcibly changed to zero. After this hard thresholding operation, only a few bands in the resulting sparse weight vector have non-zero weights, while the weights of most bands become zero. It is important to note that this hard thresholding operation is non-differentiable during neural network training. To address the problem of gradient non-backpropagation, a pass-through estimator technique is used: the sparsified weights are used during forward propagation, and the gradient is directly copied and compared through the threshold during backpropagation, meaning the gradient of the sparse weights is considered approximately equal to the gradient of the original dense weights. In this way, the network can still learn how to adjust the original weight values ​​so that the weights of important bands exceed the threshold and the weights of irrelevant bands fall below the threshold.

[0038] The third step of the module is to apply sparsity weights. The sparsified weight vector is multiplied element-wise with the original input spectral vector to obtain weighted spectral features. Since the weights of many bands are zero, the corresponding values ​​of these bands also become zero after multiplication, effectively removing these wavelengths from the spectrum. Subsequent network layers will only receive information from non-zero bands, thus achieving true feature selection. To further encourage the weight vector to converge towards extreme values, a regularization term can be added to the training loss function. This term has a small value when the weights are close to zero or close to one, and a larger value near 0.5, prompting the network to produce a more binarized sparse distribution.

[0039] In actual deployment, since the sparsity weights are fixed, the indexes of bands with non-zero weights can be pre-recorded. During the inference phase, only the spectral data of these bands are collected and calculated, skipping all zero-weight bands, thereby significantly reducing the number of collection points of the spectral sensor and the computational load of the subsequent network.

[0040] In a further embodiment of the present invention, the training loss function of the single-class classifier described by deep support vector data includes a first term and a second term. The first term is used to constrain the distance between the fused features of all samples and the center of the preset hypersphere, and the second term is the regularization term of the network weights. The radius of the hypersphere is taken as the preset quantile of the distance from all samples in the training set to the center of the hypersphere.

[0041] In this embodiment, for the k-th known variety, all training samples of that variety are collected, and each sample has been extracted into a fused feature vector by the aforementioned dual-branch network. A single-class classifier network is designed, which typically consists of several fully connected layers, to further map the input high-dimensional fused features to a feature space, the dimension of which can be reasonably selected according to the amount of data.

[0042] Before training begins, the center of the hypersphere needs to be defined. To avoid the network output degenerating into trivial solutions with all zeros, instead of setting the mean of the training sample features, a vector is sampled from a random distribution as the preset center, and this center remains fixed during training. The magnitude of this random vector is normalized so that it lies on the unit hypersphere, thus preventing features from being arbitrarily stretched or compressed. For classifiers of different varieties, different center vectors can be randomly sampled, or the same center can be used. Experiments show that randomly sampling different centers helps to further separate the feature distributions of different varieties.

[0043] The training loss function consists of two terms. The first term is the hypersphere constraint term, which calculates the average of the squared distances between the feature vectors of all training samples after being mapped by the network and the preset center. The core function of this term is to force the network to pull the sample features closer to the preset center, so that similar samples cluster within a compact region in the feature space. The second term is the regularization term for the network weights, usually the sum of the squares of the weights, used to control model complexity and prevent overfitting. The regularization coefficient is set to a small value to ensure that the main optimization direction remains feature compactness. The total loss function is the weighted sum of the first and second terms, and the network parameters are optimized end-to-end through the backpropagation algorithm. During training, it is not required that the features of all samples are strictly equal to the center, but a certain distribution range is allowed, but the overall tendency is towards clustering.

[0044] After training, the hypersphere radius for each classifier needs to be determined. The radius calculation does not use the maximum distance from the training samples to the center, as this maximum value is easily influenced by outliers and becomes abnormally large, resulting in an overly loose hypersphere that allows unknown samples to easily intrude. This embodiment uses a quantile-based method: the distances from all samples in the training set to the center are calculated, forming a distance list. These distances are then sorted from smallest to largest, and the distance value at a preset quantile position is taken as the radius. This quantile is usually set to a relatively high value, such as 95% or 90%. This means that the vast majority of training samples can fall inside the hypersphere, while a few samples with larger distances are allowed to be considered normal variants outside the boundary. By adjusting the quantile value, the conservatism of the model can be flexibly controlled: a higher quantile results in a larger radius, increasing the acceptance rate of known samples but decreasing the ability to reject unknown samples; a lower quantile results in a smaller radius, a more compact hypersphere, and enhanced ability to reject unknown samples, but may falsely reject some known samples at the boundary.

[0045] During the classification phase, for a sample to be identified, its fused features are first extracted and then input into the single-class classifier network for each variety. The distance from the mapped features to the center of their respective hyperspheres is calculated. This distance is compared with the radius of the corresponding classifier. If at least one classifier satisfies the condition that the distance is not greater than the radius, the sample is considered to belong to the variety with the smallest relative distance to the radius; if the distance is greater than the radius for all classifiers, it is determined to be an unknown variety.

[0046] In a further embodiment of the present invention, the loss function of knowledge distillation includes the cross-entropy loss of the student model on the real label and the soft label cross-entropy loss of the student model and each teacher model under a preset temperature parameter, wherein the loss terms of different teacher models are multiplied by their corresponding balance weights; when the training sample belongs to the variety corresponding to a certain teacher model, the balance weight corresponding to that teacher model is greater than the balance weights of other teacher models.

[0047] In this embodiment, the knowledge distillation process uses a trained ensemble variety identification model as the teacher model and a lightweight neural network as the student model. The student model typically adopts a two-branch design similar to the teacher model, but with a significant reduction in the number of layers and channels. Its output layer is a standard fully connected layer with an activation function, and the output dimension is equal to the total number of known varieties. After appropriate nonlinear transformation, the predicted probability of each category is obtained.

[0048] During distillation, each training sample carries its true variety label. First, soft label information for distillation needs to be extracted from the teacher model. For a given input sample, it is fed into each single-class classifier in the teacher model. Each classifier outputs the distance from the sample's features to the center of its hypersphere. This distance is converted into a similarity score by taking the maximum value of zero or one and subtracting the ratio of distance to radius. A higher score is given when the sample is completely within the hypersphere, and a lower score is given when it is outside the hypersphere. This results in a score vector whose length is equal to the number of known varieties. To obtain a smoother soft label distribution, a temperature parameter is introduced to soften the score vector. This involves dividing each score by the temperature and then normalizing it to obtain a temperature-scaled probability distribution. Higher temperatures result in a flatter probability distribution with smaller differences between categories, which is beneficial for the student model to learn the similarity structure between categories; lower temperatures result in a sharper distribution. Higher temperatures are typically used during distillation.

[0049] The student model also produces a predicted probability distribution under the same temperature parameters. The soft-label loss term in the distillation loss measures the difference between the probability distributions of the student model and the teacher model at high temperatures, typically measured using divergence. However, the claims specifically state that the loss terms of different teacher models are multiplied by their respective balance weights, meaning that the contribution to different variety categories is multiplied by different coefficients when calculating the soft-label loss. The rule for setting the balance weights is: when the true label of a training sample belongs to a certain variety, the balance weight of the teacher model corresponding to that variety is set to a larger value, while the balance weights of the teacher models for all other varieties are set to a smaller value, and the sum of all balance weights is a fixed value. Specifically, the positive class weight is significantly higher than the negative class weight; for example, the positive class weight can be set to close to 0.9, while the sum of all negative class weights is 0.1, with each negative class weight being a very small decimal. The physical meaning of this setting is that for the current sample, only the teacher classifier of its true variety provides reliable discriminative knowledge, and the outputs of other teacher classifiers should not have a significant impact on the learning of the student model.

[0050] In addition to the soft-label loss, the total distillation loss also includes a hard-label loss term, which is the cross-entropy loss of the student model to the true labels at room temperature. This term ensures that the student model can correctly learn the discrete class assignment of the samples. The two losses are weighted and summed by a balancing coefficient, which typically takes a value between 0.7 and 0.9, so that the soft-label loss dominates, fully conveying the knowledge of the teacher model.

[0051] In actual training, the student model is trained using the same dataset as the teacher model, but the data augmentation strategy can be relatively simplified. After training, the student model can be deployed to embedded devices.

[0052] In a further embodiment of the present invention, the confidence weighted fusion rule is as follows: the confidence vectors corresponding to the image recognition result and the spectral recognition result are multiplied by a preset first weight and a preset second weight, respectively, and then added together to obtain a fused confidence vector, wherein the sum of the first weight and the second weight is a set value; when the variety categories corresponding to the image recognition result and the spectral recognition result are consistent, the category and the corresponding value in the fused confidence vector are output; when they are inconsistent, the category corresponding to the mode with higher confidence is output.

[0053] In this embodiment, the student model deployed on the embedded device outputs two recognition results for each of the simultaneously acquired visible light image and near-infrared spectral data. Each result includes not only the determined variety category but also the confidence level corresponding to that determination. In classification neural networks, the confidence level is typically the maximum probability value obtained after normalization of the network output layer. This value, ranging from zero to one, reflects the model's degree of confidence in its judgment.

[0054] This embodiment sets two weighting coefficients, one for the image mode and one for the spectral mode. The sum of these two weights is a fixed value. The weighting coefficients can be determined based on prior knowledge or performance on a validation set. For example, if historical data indicates that the spectral mode is generally more reliable than the image mode, the spectral weight can be set higher than the image weight; conversely, if historical data indicates that the spectral mode is less reliable than the image mode, the spectral weight can be set higher. In some implementations, these two weights can be designed to be adaptively adjusted, for example, dynamically adjusting the image weight based on ambient light sensor readings, increasing the image weight when the lighting is good and decreasing the image weight when the lighting is insufficient.

[0055] For a single recognition task, image recognition results provide a predicted category A and a confidence score C_i, while spectral recognition results provide a predicted category B and a confidence score C_s. First, the confidence vectors of the image mode and the spectral mode are multiplied by their respective weights. Then, the two weighted vectors are summed element-wise to obtain a fused confidence vector. The dimension of this vector equals the number of known varieties, and each element corresponds to the weighted confidence score for one variety.

[0056] Next, it is determined whether the image prediction category A and the spectral prediction category B are the same. If they are the same, for example, A and B are both "Yunyan 87", then the category is finally output as the recognition result, and the confidence value of that category in the fused confidence vector is also output as the final confidence value. Since the two modalities agree, the fused confidence value is usually the sum of the two weighted confidence values, which is theoretically higher than the confidence value of any single modality, representing higher reliability.

[0057] If A and B are different, meaning the conclusions of the two modalities conflict, this embodiment adopts a strategy of prioritizing the modality with higher confidence. Specifically, it compares the image confidence levels C. iand spectral confidence C s The size of C. i Greater than C s The final output is the predicted category A of the image modality, and C. i As the final confidence level; if C s Greater than or equal to C i The final output is the predicted category B of the spectral mode, and is assigned the value C. s As the final confidence level. The logic behind this approach is that when there is disagreement between modalities, at least one modality's judgment is wrong, while the modality with higher confidence is more likely to be correct. Therefore, using the result of the high-confidence modality is safer than forcibly merging them.

[0058] In practical system implementation, this fusion rule can be extended with a threshold judgment step: if the fusion confidence level is lower than a certain lower threshold, the system can prompt "the recognition result is unreliable, and it is recommended to remeasure," avoiding drawing incorrect conclusions based on low-quality data. Furthermore, the system can record the occurrence of each modal conflict for subsequent analysis, such as for monitoring the operational status of spectral sensors or cameras.

[0059] In a further embodiment of the present invention, the preprocessing of the visible light image includes: image size scaling, pixel normalization, brightness equalization, and noise reduction; the preprocessing of the near-infrared spectral data includes: sliding window smoothing, baseline correction, multivariate scattering correction, and derivative processing; the data enhancement includes: random cropping, rotation, brightness adjustment, noise addition, and linear superposition of the image; and translation, addition of random noise, and linear superposition of similar samples of the spectrum.

[0060] In this embodiment, the preprocessing flow for visible light images includes four steps. The first step is image scaling, which uniformly adjusts all acquired raw images to a preset fixed size to meet the input layer requirements of the neural network. Appropriate interpolation algorithms are used during scaling to maintain the aspect ratio of the leaves without severe distortion and to ensure that the scaled image does not lose key texture information. The second step is pixel normalization, which maps the value of each pixel in the image from the original integer range to a smaller floating-point range, typically between zero and one. This accelerates the convergence speed of neural network training and avoids gradient instability caused by an excessively large input value range. The third step is brightness equalization, typically using histogram equalization or its improved algorithms to stretch the image contrast, making the difference between the leaf area and the background area more obvious, while correcting the overall brightness or darkness caused by different lighting angles. The fourth step is noise reduction, using algorithms such as median filtering or Gaussian filtering to remove salt-and-pepper noise or Gaussian noise introduced during image acquisition, while preserving leaf edge and texture details as much as possible.

[0061] For near-infrared spectral data, the preprocessing workflow is more specialized. The first step is sliding window smoothing, typically using the Savitzky-Gore smoothing algorithm to smooth the spectral curve of each sample, eliminating high-frequency random noise while preserving the shape and position of spectral peaks. The second step is baseline correction. Since baseline tilt may occur during spectral acquisition due to instrument background drift or sample scattering, methods such as polynomial fitting or asymmetric least squares are used to estimate and subtract the baseline, making the spectral curves flatter. The third step is multivariate scattering correction, used to eliminate light scattering effects caused by differences in tobacco leaf powder particle size or compaction. By regressing each spectrum to the average spectrum, the spectral data becomes more comparable. The fourth step is derivative processing, typically calculating the first or second derivative spectrum. Derivative processing can enhance the subtle variations in the spectrum, eliminate constant baseline shifts and linear drift, and highlight differences in the position and intensity of absorption peaks.

[0062] After the above preprocessing, data augmentation is performed to expand the training set. For visible light images, data augmentation methods include, but are not limited to, the following: random cropping, randomly cropping a sub-region from the original image and then scaling it back to its original size to simulate the observation perspective of different parts of the leaf; random rotation, randomly rotating the image within a certain angle range to simulate different leaf placement orientations; brightness adjustment, randomly increasing or decreasing the brightness of the image to simulate shooting conditions under different light intensities; adding noise, randomly adding a small amount of Gaussian noise or salt-and-pepper noise to the image to improve the model's robustness to sensor noise; and linear superposition, superimposing and mixing two different images of the same variety with random weights to generate new virtual samples, which can further expand sample diversity. All augmentation operations are performed randomly, and the augmented image shares the same category label as the original image.

[0063] For near-infrared spectroscopy, data augmentation methods include: wavelength axis shifting, randomly shifting the spectral curve a small number of units along the wavelength direction to simulate small deviations in wavelength calibration; adding random noise, adding a small amount of random noise to the spectral intensity values ​​to improve the model's tolerance to spectral acquisition noise; and linear superposition of similar samples, mixing two different spectra of the same variety with random weights to generate a new virtual spectrum. It is important to note that spectral augmentation should be performed while ensuring physical plausibility to avoid disrupting the basic spectral structure.

[0064] In a further embodiment of the present invention, the system operation and anomaly management steps are also included: after the device is powered on, it automatically completes hardware initialization, model loading and spectral sensor connection verification; a built-in anomaly capture daemon process captures anomalies in the camera, spectral sensor and model inference process and automatically restarts; the image results, spectral curves, fusion results, timestamps and device numbers of each recognition are automatically saved to local storage; and a built-in dataset balancing verification module is used to analyze the distribution of the number of samples in each category before model training and output data bias warnings.

[0065] In this embodiment, after the system is powered on, it automatically executes the hardware initialization process, including detecting whether the camera is turned on normally, setting image acquisition parameters, initializing the near-infrared spectral sensor and establishing a communication connection, loading the pre-deployed student model file into memory, and configuring the embedded device's clock and power management strategy. During the initialization process, if a hardware module fails to initialize, the system will retry several times according to the preset retry strategy. If it still fails, an error log is recorded and the system enters standby mode, while prompting the user to check the hardware connection through the display screen or indicator lights.

[0066] The system incorporates an exception handling daemon, which runs in the background as an independent thread or process, continuously monitoring three key stages: camera acquisition, spectral sensor acquisition, and model inference. For the camera, the daemon monitors for image acquisition timeouts, empty or abnormally sized returned image data, and attempts to reinitialize the camera driver upon detecting an exception. For the spectral sensor, the daemon periodically sends heartbeat commands to check communication port connectivity; if there are multiple consecutive unresponsive attempts, it attempts to restart the sensor and re-establish the connection. For the model inference stage, the daemon monitors memory usage, inference time exceeding set thresholds, and invalid output values; if an exception is detected, the model is reloaded. For severe exceptions that cannot be automatically recovered from, the daemon performs a soft reboot of the entire system.

[0067] After each recognition operation, the system automatically saves the relevant information to local storage. The saved information includes: the original visible light image and its thumbnail, the original near-infrared spectral curve data, the preprocessed spectral data, the fused recognition result and confidence levels for each category, the recognition timestamp, the device serial number and hardware version number, and the system's operating status at that time. This data is stored in a structured format for easy retrieval and analysis. Local storage uses a circular buffer management system, automatically overwriting the oldest historical records when storage space is insufficient. Data can also be exported via an external interface.

[0068] During the model training phase, the system's built-in dataset balancing and verification module analyzes the distribution of sample numbers for each category in advance. When the user prepares the training dataset, this module reads the labels of all samples, counts the number of samples for each variety, and calculates the maximum, minimum, mean, and coefficient of variation. If it finds that the number of samples for some varieties is significantly less than that of other varieties, for example, less than one-tenth of the largest class, or below a preset lower limit for absolute number, the module will output a data bias warning on the training interface, clearly indicating which varieties belong to the minority class and providing a possible impact assessment, such as the possibility that the recognition accuracy of minority class varieties may be lower. At the same time, the module will suggest countermeasures for the user, such as collecting more minority class samples, oversampling enhancement for the minority class, or using class weight adjustment loss functions. This module can also further calculate the feature space overlap between categories, and warn of difficult varieties that not only have few samples but also have features that are highly confused with the majority class.

[0069] In a further embodiment of the present invention, the wavelength range of the near-infrared spectrum is the near-infrared band, the embedded device is an embedded development board, and the student model is deployed after quantization and compression.

[0070] The near-infrared spectrum covers the near-infrared band, typically referring to the region from 780 nm to 2500 nm. Within this band, organic molecules in tobacco, such as hydrogen-containing groups, produce abundant overtone and combination frequency absorptions, carrying information about key chemical components such as total sugar, nicotine, total nitrogen, and moisture. To balance information richness and sensor cost, this embodiment preferably uses the short-wave near-infrared band of 900 nm to 1700 nm. This band allows the use of a lower-cost indium gallium arsenide detector while avoiding the influence of the strong moisture absorption peak. Of course, depending on specific application requirements, it can be extended to above 1800 nm, but the sensor cost will increase accordingly. The determination of the wavelength range should be based on the principle of effectively distinguishing the target tobacco variety, and usually requires preliminary experimental screening.

[0071] The embedded device uses a commercially available embedded development board. This board is based on a low-power processor architecture, such as the ARM Cortex series, and has sufficient computing power to run lightweight neural networks. It also integrates a camera interface, a communication interface, and general-purpose input / output interfaces for easy connection to a spectral sensor. An embedded operating system runs on the development board, with unnecessary services removed to reduce memory usage and startup time. To support quantized model inference, the development board's software environment needs to deploy a neural network inference framework optimized for embedded devices, supporting eight-bit integer operations.

[0072] The specific process for deploying the student model after quantization and compression is as follows: First, on the server side, a batch of calibration datasets is used to calculate the quantization parameters of the student model. The calibration data should be representative, covering various tobacco varieties and typical variations. The quantization process includes mapping the weights of each layer of the model from floating-point numbers to a set of discrete eight-bit integer values, while simultaneously calculating the dynamic range of activation values ​​for each layer, determining the scaling factor and zero-point offset. Weight quantization typically employs symmetric or asymmetric quantization schemes, compressing as much as possible while keeping accuracy loss controllable. Activation value quantization needs to be dynamically calculated during inference or pre-calculated using the calibration set. After quantization, the model weights are saved as eight-bit integers, while retaining the quantization parameters. Then, the quantized model file is deployed to the file system of the embedded development board. During device runtime, the inference framework loads the quantized model. During inference, the input data is first converted to integer representation, then all layer operations are completed within the integer domain, and finally, the output result is dequantized back to floating-point numbers for subsequent confidence calculations. Since integer operations are much faster than floating-point operations and consume less memory, the model inference speed is significantly improved.

[0073] To further optimize, a mixed-precision strategy can be adopted for certain layers of the model. This involves maintaining floating-point precision for sensitive layers and using quantized integers for most layers, achieving a balance between speed and accuracy. After quantization, a test set should be used to verify whether the quantization accuracy loss is within an acceptable range, typically requiring an accuracy decrease of no more than one percentage point.

[0074] In a second aspect embodiment of the present invention, a deep learning-based automatic tobacco variety classification system is provided for executing a deep learning-based automatic tobacco variety classification method according to a first aspect embodiment of the present invention, comprising: Data acquisition unit: includes a visible light camera and a near-infrared spectral sensor, used to acquire visible light images and near-infrared spectral data of tobacco leaves; Embedded main control unit: It adopts an embedded development board, runs an embedded operating system, and deploys a lightweight classification model after knowledge distillation and quantization compression; Preprocessing unit: used to perform preprocessing and data augmentation on the acquired visible light images and near-infrared spectral data; The dual-modal feature extraction and fusion unit includes a hybrid gradient domain attention module for the image branch, a band attention and sparse constraint module for the spectral branch, and a feature fusion layer, which are used to extract and fuse weighted image features and weighted spectral features respectively. Open set classification and incremental learning unit: Multiple single-class classifiers are constructed and integrated in parallel based on deep support vector data description, which are used to identify known tobacco varieties, reject unknown varieties, and support modular incremental updates of the variety library; Knowledge distillation and model compression unit: used to transfer the knowledge of the integrated complex teacher model to the lightweight student model, and to perform quantitative compression on the student model; Human-computer interaction and display unit: used to display recognition results, spectral curves, confidence levels and system status in real time; Data storage and log unit: used to store identification history, spectral data, operation logs and equipment information; Anomaly Protection and Ethics Compliance Unit: Used to achieve fault self-recovery, data bias verification, and access control.

[0075] In this embodiment, the system consists of multiple functional units that work together to form a complete hardware-software integrated solution.

[0076] The data acquisition unit comprises a visible light camera and a near-infrared spectral sensor. The camera uses an industrial-grade module with autofocus and a global shutter, achieving a resolution sufficient for texture detail extraction, and is equipped with a ring LED supplementary light source to ensure consistent illumination. The near-infrared spectral sensor employs a miniaturized, low-power module, covering the key near-infrared bands, and incorporates a temperature stabilization circuit to reduce the impact of ambient temperature on the spectrum. Both sensors are mechanically mounted on the same acquisition bracket to ensure consistent leaf position during synchronous acquisition.

[0077] The embedded main control unit is the core of the entire system, utilizing a high-performance embedded development board and running an embedded Linux operating system. The main control unit connects to the camera and spectral sensor via a data interface, to the touchscreen display via a display interface, and to a local memory card via a storage interface. The main control unit deploys a lightweight classification model that has undergone knowledge distillation and quantization compression, as well as a model inference engine.

[0078] The preprocessing unit runs as a software module on the main control unit, implementing visible light image preprocessing, near-infrared spectral preprocessing, and data augmentation functions. During the training phase, the preprocessing unit is responsible for offline augmentation to generate expanded datasets; during the inference phase, the preprocessing unit performs standardization, scaling, normalization, smoothing, and correction operations on the real-time acquired data.

[0079] The core algorithm is implemented in a dual-modal feature extraction and fusion unit. This unit includes a hybrid gradient domain attention module for the image branch, a band attention and sparse constraint module for the spectral branch, and a feature fusion layer. These modules are implemented as a computational graph in the deep learning framework and are executed automatically during model inference. The output of this unit is the fused feature vector, which is used by the classifier.

[0080] The Open Set Classification and Incremental Learning Unit implements ensemble classifiers based on deep support vector data description. This unit maintains a classifier registry, recording the single-class classifier corresponding to each known variety, along with its hypersphere center and radius. When a new variety is added, this unit calls the incremental learning submodule to train only the new classifier and register it; when a variety is removed, the corresponding entry is removed from the registry. During inference, this unit receives fused features, iterates through all registered classifiers for distance comparison, and outputs the recognition result and open set determination.

[0081] The knowledge distillation and model compression unit implements the teacher-student distillation and quantization compression process. This unit runs during the system's offline training phase to generate a lightweight student model deployed on the embedded main control unit. After training is complete, this unit does not consume resources during device runtime.

[0082] The human-computer interaction and display unit is responsible for user interaction. The touchscreen displays a real-time preview of the camera image, the current recognition result, confidence level bar charts for each category, and near-infrared spectral waveforms. Users can trigger recognition operations, view historical records, export data, and update the variety database through the interface. The system also provides a voice broadcast function, automatically announcing the variety name after recognition is complete.

[0083] The data storage and log unit manages local storage space. Each identified image, spectral curve, processed features, identification result, confidence level, timestamp, and device number are structurally stored. The storage unit also records system operation logs, including boot time, abnormal events, and hardware status, supporting retrieval and export by time range.

[0084] The Anomaly Prevention and Ethics Compliance unit implements anomaly management functions and adds an ethics compliance submodule. This submodule includes access control, such as restricting unauthorized personnel operations via passwords or biometrics; it also includes data bias auditing, periodically analyzing stored historical identification records to detect systematic biases towards certain varieties or sources of samples, and generating audit reports if found. This unit is also responsible for anonymizing user data and device information to ensure that sensitive information is not leaked during data export.

[0085] The automatic tobacco variety classification system constructed in this embodiment integrates the technical solutions of the aforementioned claims into a complete, practically deployable product, achieving a comprehensive system-level effect. First, the clear responsibilities and interfaces of each unit make the system easy to maintain and upgrade. For example, when a better feature extraction network emerges, only the implementation of the bimodal feature extraction and fusion unit needs to be replaced, while other units remain unchanged. Second, the tight integration of the data acquisition unit and the embedded main control unit ensures the real-time performance of the data flow. The end-to-end latency from triggering acquisition to outputting the recognition result can be controlled within hundreds of milliseconds, meeting the needs of rapid on-site screening. Third, the open set classification and incremental learning unit enables the system to have "lifelong learning" capabilities. During long-term use, users can continuously add new varieties to the system without requiring system downtime for retraining, greatly improving practicality and adaptability. Obsolete varieties can also be easily removed, keeping the system clean and efficient. Fourth, the human-computer interaction and display unit presents spectral curves and confidence levels visually, not only outputting results but also allowing users to understand the system's decision-making basis. When the recognition confidence level is low, users can observe whether the spectral curve is abnormal, thereby increasing their trust in the system. Fifth, the data storage and log unit provides valuable data assets for quality traceability, equipment maintenance, and model retraining. Long-term accumulated historical data can be used to analyze the spectral drift patterns of tobacco varieties from different seasons and origins, guiding model update strategies. Sixth, the anomaly protection and ethical compliance unit ensures the system meets the reliability requirements of industrial-grade products. The fault self-recovery function reduces the need for on-site technical support; the data bias verification mechanism ensures algorithmic fairness, avoiding unfair treatment of certain varieties due to training data bias; and access control prevents misoperation and data leakage, complying with data security regulations. In summary, this embodiment provides a fully functional, highly scalable, and practically applicable automatic tobacco variety classification system, truly transforming the technological innovations of the preceding claims into a practical product that can serve frontline work in the tobacco industry.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for automatic tobacco variety classification based on deep learning, characterized in that, include: Visible light images and near-infrared spectral data of multiple known tobacco varieties were collected, and a dataset was formed after preprocessing and data augmentation. The dual-branch feature extraction network includes an image branch and a spectral branch; the image branch uses a convolutional neural network and introduces a hybrid gradient domain attention module to output weighted image features; the spectral branch uses a one-dimensional convolutional neural network or a lightweight multilayer perceptron and introduces a band attention and sparse constraint module to output weighted spectral features. The weighted image features and the weighted spectral features are concatenated and fused to obtain the fused features; For each known variety, a single-class classifier based on deep support vector data description is trained, and all single-class classifiers are integrated in parallel to obtain a variety identification model; the variety is determined as known or unknown based on the comparison results of the distance and radius from the fusion features of the sample to be identified to the center of each hypersphere; when adding or eliminating varieties, the corresponding single-class classifier is added or removed accordingly. Using the variety identification model as the teacher model and the lightweight neural network as the student model, knowledge from the teacher model is transferred to the student model through knowledge distillation. The trained student model is then quantized, compressed, and deployed to an embedded device. Visible light images and near-infrared spectra of tobacco leaves are acquired simultaneously and input into a student model deployed on an embedded device to obtain image recognition results and spectral recognition results. The final classification result is obtained by using a confidence-weighted fusion rule.

2. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The hybrid gradient domain attention module specifically includes: extracting texture gradient maps of the input feature map in parallel using five edge detection operators: Sobel, Scharr, Laplace, Roberts, and Prewitt; concatenating the five gradient maps along the channel dimension; performing feature extraction and channel restoration through convolutional layers to obtain an intermediate feature map; and then encoding the intermediate feature map using a pyramid module to generate an attention weight map; the pyramid module includes multiple convolutional layers, max pooling layers, and deconvolutional layers.

3. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The band attention and sparsity constraint module specifically includes: generating original attention weights for each band using a two-layer fully connected neural network, mapping them to a preset interval via an activation function; sorting the generated attention weights by value and setting a weight threshold, assigning weights smaller than the weight threshold to preset values ​​to obtain sparsified band weights; multiplying the sparsified band weights by the original spectral data to obtain weighted spectral features; wherein, the weight threshold is determined based on a preset number or proportion of retained bands.

4. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The training loss function of a single-class classifier described by deep support vector data includes a first term and a second term. The first term is used to constrain the distance between the fused features of all samples and the center of a preset hypersphere, and the second term is the regularization term of the network weights. The radius of the hypersphere is taken as the preset quantile of the distance from all samples in the training set to the center of the hypersphere.

5. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The loss function of knowledge distillation includes the cross-entropy loss of the student model on the real labels and the soft label cross-entropy loss of the student model and each teacher model under the preset temperature parameters, wherein the loss terms of different teacher models are multiplied by the corresponding balance weights. When a training sample belongs to a variety corresponding to a certain teacher model, the balance weight of that teacher model is greater than the balance weight of other teacher models.

6. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The confidence-weighted fusion rule is as follows: multiply the confidence vectors corresponding to the image recognition result and the spectral recognition result by preset first weight and second weight respectively, and then add them to obtain the fused confidence vector, where the sum of the first weight and the second weight is a set value; when the variety categories corresponding to the image recognition result and the spectral recognition result are consistent, output the category and the corresponding value in the fused confidence vector; when they are inconsistent, output the category corresponding to the modality with higher confidence.

7. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The preprocessing of visible light images includes: image size scaling, pixel normalization, brightness equalization, and noise reduction; the preprocessing of near-infrared spectral data includes: sliding window smoothing, baseline correction, multivariate scattering correction, and derivative processing; the data enhancement includes: random cropping, rotation, brightness adjustment, noise addition, and linear superposition of the image; and translation, addition of random noise, and linear superposition of similar samples of the spectrum.

8. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, It also includes system operation and anomaly management steps: after the device is powered on, it automatically completes hardware initialization, model loading, and spectral sensor connection verification; it has a built-in anomaly capture daemon process that captures anomalies in the camera, spectral sensor, and model inference stages and automatically restarts the system; it automatically saves the image results, spectral curves, fusion results, timestamps, and device numbers for each recognition to local storage; The built-in dataset balancing verification module is used to analyze the distribution of the number of samples in each category and output data bias warnings before model training.

9. The method for automatic tobacco variety classification based on deep learning according to claim 1, characterized in that, The wavelength range of the near-infrared spectrum is the near-infrared band, the embedded device is an embedded development board, and the student model is deployed after quantization and compression.

10. A deep learning-based automatic tobacco variety classification system, used to execute the deep learning-based automatic tobacco variety classification method according to any one of claims 1-9, characterized in that, include: Data acquisition unit: includes a visible light camera and a near-infrared spectral sensor, used to acquire visible light images and near-infrared spectral data of tobacco leaves; Embedded main control unit: It adopts an embedded development board, runs an embedded operating system, and deploys a lightweight classification model after knowledge distillation and quantization compression; Preprocessing unit: used to perform preprocessing and data augmentation on the acquired visible light images and near-infrared spectral data; The dual-modal feature extraction and fusion unit includes a hybrid gradient domain attention module for the image branch, a band attention and sparse constraint module for the spectral branch, and a feature fusion layer, which are used to extract and fuse weighted image features and weighted spectral features respectively. Open set classification and incremental learning unit: Multiple single-class classifiers are constructed and integrated in parallel based on deep support vector data description, which are used to identify known tobacco varieties, reject unknown varieties, and support modular incremental updates of the variety library; Knowledge distillation and model compression unit: used to transfer the knowledge of the integrated complex teacher model to the lightweight student model, and to perform quantitative compression on the student model; Human-computer interaction and display unit: used to display recognition results, spectral curves, confidence levels and system status in real time; Data storage and log unit: used to store identification history, spectral data, operation logs and equipment information; Anomaly Protection and Ethics Compliance Unit: Used to achieve fault self-recovery, data bias verification, and access control.

Images