Method for detecting mildew of redried tobacco leaves based on mixed attention mechanism enhancement and 3D residual CNN (Convolutional Neural Network) network

Abstract

The invention provides a redried tobacco leaf mildewing detection method based on mixed attention mechanism enhancement and a 3D residual CNN network, and the method comprises the steps: collecting redried tobacco leaf samples, and preparing mildewed tobacco leaf samples in different periods; hyper-spectral image data of the mildewed and redried tobacco leaf samples are collected through hyper-spectral image collecting equipment; the collected hyperspectral image data is corrected; establishing a mildewed tobacco leaf data set according to the corrected hyperspectral image data; an MA3D-ResNet model is constructed; carrying out iterative training on the model by utilizing the training set and the verification set, and storing model parameters with the highest accuracy on the verification set within a specified number of iterations; model parameters which are obtained through training and have the highest accuracy rate are imported into the model, test set data are input, evaluation indexes are calculated according to a model output result, the corrected hyperspectral image of the redried tobacco leaves to be detected is input into the tested model, and a mildew detection result is obtained. The method can effectively improve the detection precision of the mildewing degree of tobacco leaves with extremely similar spectral characteristics, especially early mildewing tobacco leaves.

Description

Technical Field

[0001] The present invention relates to the technical field of target classification detection, and in particular to a method for detecting tobacco leaf mildew. Background Art

[0002] Tobacco leaf mold is one of the main problems facing the tobacco industry. It not only affects the quality of tobacco leaves, but may also pose a threat to human health. According to research, the temperature and humidity of the environment and the moisture content of the tobacco leaves themselves are the key factors affecting the occurrence of mold. In addition, due to the geographical location, the diversity of tobacco varieties, and the complexity of the types and numbers of molds, the task of detecting tobacco leaf mold has become extremely difficult. Traditional detection methods such as chemical analysis are accurate but the process is cumbersome and costly, which is not conducive to large-scale application. Multispectral technology is fast but not very accurate. Although hyperspectral imaging technology can provide rich information, traditional machine learning algorithms have difficulty capturing subtle changes in tobacco leaf mold when processing hyperspectral data. Deep learning technologies such as three-dimensional convolutional neural networks (3D-CNN) can simultaneously process the spectral and spatial dimensions of hyperspectral images and comprehensively extract features, but when faced with the degree of tobacco leaf mold with extremely similar spatial spectral features, especially when it comes to the task of detecting early tobacco leaf mold, there are still problems such as low feature discrimination.

[0003] Low feature discrimination: The early stages of tobacco leaf mold may be very similar to the normal state, which makes it difficult to capture subtle visual differences. Existing neural networks may not be able to effectively distinguish these subtle differences, especially when the feature extraction layer is not strong enough to capture key features.

[0004] The invention patent with publication number CN114993993B discloses a method and device for detecting mildew in tobacco leaves. The mildew grade corresponding to each tobacco leaf sample is used as a label to train a neural network mathematical model. The neural network model consists of an input layer, a convolution layer, a Relu activation function layer, and a pooling layer. This application trains a neural network model and inputs the optical data of tobacco leaves into the trained neural network model to determine the mildew grade. This solves the technical problem of low accuracy of tobacco leaf mildew detection in the prior art and achieves the technical effect of improving the accuracy of tobacco leaf mildew detection. However, the neural network model used in this method is difficult to capture subtle changes in tobacco leaf mildew when processing hyperspectral data, and has low feature discrimination when faced with the task of detecting early mildew of tobacco leaves with extremely similar spatial spectral features. Summary of the invention

[0005] In view of the technical problem of low feature differentiation in early mildew detection in existing detection methods, the present invention proposes a method for detecting mildew of redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network, and proposes a multi-attention enhanced three-dimensional residual convolutional neural network (mixed attention 3D-ResNet, MA3D-ResNet) model. The 3D residual CNN network and the hybrid attention mechanism are used to effectively improve the detection accuracy of the early mildew degree of redried tobacco leaves with extremely similar features, especially early mildew.

[0006] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network, comprising the steps of:

[0008] S1: Collect redried tobacco leaf samples and prepare moldy tobacco leaf samples at different stages;

[0009] S2: using a hyperspectral image acquisition device to collect hyperspectral image data of moldy redried tobacco leaf samples; and correcting the collected hyperspectral image data;

[0010] S3: A moldy tobacco leaf dataset is established based on the corrected hyperspectral image data, and the dataset is divided into a training set, a validation set, and a test set;

[0011] S4: Construct the MA3D-ResNet model, including the three-dimensional convolution layer I, the three-dimensional residual unit I, the three-dimensional residual unit II, the spatial and spectral attention mechanism, the three-dimensional convolution layer II, the channel attention mechanism, the Flatten layer, and the classification layer connected in sequence;

[0012] S5: Iteratively train the model using the training set and validation set, and save the model parameters with the highest accuracy on the validation set within the specified number of iterations;

[0013] S6: Import the model parameters with the highest accuracy obtained through training into the model, input the test set data, calculate the evaluation index according to the model output results, input the hyperspectral image of the redried tobacco leaves to be tested that has been corrected in step S2 into the tested model, and obtain the mold detection result.

[0014] The method for correcting the collected hyperspectral image data is as follows: using a standard Teflon white plate and a lens cover to cover and collect a white reference image I under a white background. w and black reference image I on black background b ; Use the correction formula to correct the hyperspectral image data to obtain the corrected hyperspectral image I. The correction formula is:

[0015]

[0016] Among them, I 0 is the unprocessed original hyperspectral image, I w is the white reference image, I b is a black reference image.

[0017] In step S3, some pixel points in the hyperspectral image are selected as sample points in the moldy tobacco leaf dataset, and then an n*n area is selected with each sample point as the center, containing m bands as input samples of the model described in step S4; based on expert judgment and average spectral reflectance, the sample points are classified into non-moldy, slightly moldy and severely moldy, and the sample points are labeled according to the classification type.

[0018] The three-dimensional residual unit I includes a three-dimensional convolution block a and a three-dimensional convolution block b connected in sequence, the output of the three-dimensional convolution block b is residually connected to the input of the three-dimensional convolution block a, and then output after a ReLU activation function, and the input of the three-dimensional convolution block a is connected to the output of the three-dimensional convolution layer I;

[0019] The three-dimensional residual unit II includes a three-dimensional convolution block c and a three-dimensional convolution block d connected in sequence, the output of the three-dimensional convolution block d is residually connected to the input of the three-dimensional convolution block c, and then output after a ReLU activation function, and the input of the three-dimensional convolution block c is connected to the output of the three-dimensional residual unit I;

[0020] The three-dimensional convolution block a, the three-dimensional convolution block b, the three-dimensional convolution block c, and the three-dimensional convolution block d all include three-dimensional convolution layers, ReLU activation functions, and batch normalization layers connected in sequence; the classification layer includes three fully connected networks and a classifier connected in sequence.

[0021] The process of iteratively training the model using the training set and the validation set in step S5 is as follows:

[0022] The training set and validation set data are imported into the constructed MA3D-ResNet model. Each input sample is passed through the three-dimensional convolution layer I to extract the joint features of space and spectrum, and the output feature map I is obtained. 1 ; Output feature map I 1 The features are enhanced by the calculation of the three-dimensional residual unit I and the three-dimensional residual unit II in turn to obtain the output feature map Y; the output feature map Y is enhanced from two dimensions by the spatial attention and spectral attention calculations to obtain the fusion feature map I 2 ; Output feature map I 2 After the three-dimensional convolution layer II, the features are compressed and refined to obtain the output feature map F C ; Output feature map F CAfter the channel attention calculation, the output feature map F is obtained; after the output feature map F is flattened, the one-dimensional vector I is obtained. 3 ; One-dimensional vector I 3 After passing through the fully connected layer of the classification layer, the output vector I is obtained 4 , the output vector I 4 After the classifier, the output vector I is converted by the SoftMax function 4 Convert to a probability distribution; make each element between (0,1), indicating the probability of the input sample being non-moldy, slightly moldy, and severely moldy;

[0023] In each iteration, the cross entropy loss is calculated according to the output probability distribution, and the parameters are updated using the Adam optimizer based on the cross entropy loss. The model trained in each iteration is verified using the validation set, and the classification accuracy of the validation set is monitored. After reaching the specified number of iterations, the model parameters with the highest accuracy on the validation set are saved.

[0024] The method for calculating the spatial attention and spectral attention is: H×W×D Reshape into feature maps Y a ∈R N×D and feature map Y b ∈R D×N , H, W and D are the height, width and number of bands of feature map Y respectively, N = H × W; then the feature map Y a Its transposed feature map Y aT And the feature map Y b Its transposed feature map Y bT Perform matrix multiplication and use the Softmax function to obtain the spatial attention weight matrix M a ∈R N×N and the spectral attention weight matrix M b ∈R D×D , then, respectively in the spatial attention weight matrix M a and feature map Y a And in the spatial spectral attention weight matrix M b and feature map Y b Perform matrix multiplication between them, reshape the multiplication results, and add the reshaped results to the input feature map Y to obtain the feature map E after spatial attention calculation a ∈R H×W×D And the feature map E calculated by spectral attention b , and finally the feature map E calculated by spatial attention a And the feature map E calculated by spectral attention b Splice to get the fusion feature map I 2 .

[0025] The channel attention calculation process is as follows: Input feature map F C ∈R H×W×C First, the global information FC1 of the fusion feature is extracted through average pooling and maximum pooling operations respectively. avg and highlight information FC1 max , then enter the fully connected layer I, then introduce the ReLU activation function, and then enter the fully connected layer II, and use the SoftMax function to get the channel attention weight matrix M C ∈R 1×1×C , the weight matrix M C Multiplying the input feature to get the output feature map F, the formula is:

[0026] FC1 avg =ReLU(FC1(AvgPool(F C )))

[0027] FC1 max =ReLU(FC1(MaxPool(F C )))

[0028]

[0029] F=M C (F C )⊙F C

[0030] in, Represents the accumulation of corresponding pixel values, FC1 represents the fully connected layer I operation function, FC1 represents the fully connected layer II operation function, and ⊙ represents the element multiplication operation.

[0031] The spatial attention weight matrix M a Or the spectral attention weight matrix M b The calculation formula is:

[0032]

[0033] M ji is the spatial attention weight matrix M a or M b An element in i Represents the feature map Y a The i-th element or feature map Y b The i-th element in Y j T Represents the feature map Y aT The jth element or feature map Y bT The jth element in Represents matrix multiplication operation. L takes N when calculating the spatial attention weight matrix, and takes D when calculating the spectral attention weight matrix.

[0034] The feature map E calculated by spatial attention a Or the feature map E calculated by spectral attention b The calculation formula is:

[0035]

[0036] Among them, reshape is the reshaping operation, and E represents the feature map E after spatial attention calculation a Or the feature map E calculated by spectral attention b , M represents the spatial attention weight matrix M a Or the spectral attention weight matrix M b , Y a,b Represents the feature map Y a Or feature map Y b .

[0037] The evaluation indicators are classification accuracy, overall accuracy, average accuracy and Kappa statistic for each category.

[0038] The method for calculating the cross entropy loss is:

[0039]

[0040] Among them, m is the batch size, y s is the real sample label, is the predicted probability value corresponding to the true sample label in the probability distribution.

[0041] Beneficial effects of the present invention:

[0042] (1) An experimental method for detecting mildew in redried tobacco leaves was designed. The differences in pixel points of hyperspectral images with different mildew degrees were analyzed and the mildew grades were divided.

[0043] (2) To solve the problem of cross-infection between healthy tobacco leaves and moldy tobacco leaves caused by centralized storage of re-roasted tobacco leaves, an intelligent, non-destructive, and rapid method for detecting moldy tobacco leaves is provided.

[0044] (3) The designed three-dimensional residual unit can increase the depth of the three-dimensional convolutional neural network model, improve the discrimination performance and avoid the overfitting problem of the model. Since the deep network can learn richer features, the three-dimensional convolution using the three-dimensional residual unit has stronger generalization ability and can perform better on unseen data.

[0045] (4) The spatial and spectral attention mechanisms can enhance the features of data from both spatial and spectral dimensions, effectively improving the model’s discriminative performance. In tobacco leaf mold detection, the spatial attention mechanism can help locate the area where mold occurs, while the spectral attention mechanism can help identify which spectral features best reflect the degree of mold. This combination not only improves the accuracy of the model, but also enhances its adaptability and robustness to complex changes, especially in the case of early mold detection of redried tobacco leaves with extremely similar features.

[0046] (5) The channel attention mechanism can dynamically adjust the importance of different spatial and spectral features, thereby emphasizing key information and suppressing redundant or irrelevant information. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is the overall workflow diagram of the present invention.

[0049] Figure 2 Diagram of the 3D residual convolutional neural network model enhanced by multiple attentions.

[0050] Figure 3 It is a three-dimensional residual structure diagram of the present invention.

[0051] Figure 4 : This is a diagram of the mildew detection result of the present invention, (a) is the original image, and (b) is the mildew recognition result. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] A method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network. Figure 1 As shown, the steps include:

[0054] S1: Collect redried tobacco leaf samples and prepare moldy tobacco leaf samples at different stages;

[0055] S1.1 Collect redried tobacco leaf samples, and select tobacco leaves of the same grade that are judged by experts in advance based on morphological characteristics, sensory evaluation, physical and chemical analysis, etc., and are free of disease spots and insect spots;

[0056] S1.2 Use a constant temperature and humidity chamber to balance the tobacco leaves for 48 hours at 22±2°C and 60±5% humidity to simulate natural growth conditions.

[0057] S1.3 The temperature of the constant temperature and humidity chamber is set to 30°C and the humidity is set to 85%, and a mildew experiment is conducted with a period of 28 days to prepare mildewed tobacco leaf samples of different periods, wherein the 0th day is one period, the 1st to 4th days are one period, the 5th to 10th days are one period, the 11th to 25th days are one period, and the 26th to 28th days are one period.

[0058] S2: using a hyperspectral image acquisition device to collect hyperspectral image data of moldy redried tobacco leaf samples; and correcting the collected hyperspectral image data;

[0059] like Figure 1 As shown, a hyperspectral image acquisition device is selected, including an imaging lens (OLES30, Finland), a visible near-infrared hyperspectral imager (Shuanglihe Spectrum, GalaFeld-V10E-AZ4), an electric stage, 4 50W halogen lamps and a computer, etc., and the hyperspectral image acquisition device is calibrated; the parameters are set: the exposure time is 4.5s, the spectral acquisition range is 390.80-1029.23nm, and 250 spectral bands are selected in this embodiment; the spectral resolution reaches 2.4 nanometers, and the image resolution is 1900×800. Positioning and orientation: determine the target area, and adjust the direction and height of the imager to cover the entire area of ​​interest, start the hyperspectral image acquisition device, and focus the lens and debug the platform movement speed after preheating to avoid distortion of the collected spectral image. Preheating means that before using the device or conducting an experiment, by letting the device run for a period of time, the preheating process can ensure that the light source, sensor and other components of the hyperspectral device work in a stable state, thereby improving the accuracy, stability and reliability of the spectral data.

[0060] Hyperspectral images contain spatial information (i.e., the shape and texture of objects) just like ordinary images. However, ordinary RGB images only have three different bands: red, green, and blue. Hyperspectral images can cover hundreds or even thousands of bands, which makes the hyperspectral image data contain detailed spectral information. For ordinary images and hyperspectral images with a length and width of 1900 and 800 respectively, the size of ordinary images is (1900, 800, 3), and the size of hyperspectral images is (1900, 800, 250), where 250 corresponds to the 250 spectral bands of the device used in the present invention. Each point in this three-dimensional cube data corresponds to a digital value, which together constitute the entire image data.

[0061] The first sampling was carried out on day 0 at the beginning of the experiment, followed by daily sampling from days 1 to 4, every 12 hours from days 5 to 10, every 8 hours from days 11 to 25, and finally every 12 hours from days 26 to 28.

[0062] The method for correcting the collected hyperspectral image data is as follows: while collecting the hyperspectral image data, a white reference image under a white background is collected by using a standard Teflon white board and a lens cover to cover the white reference image (I w ) and a black reference image against a black background (I b ); This step is black and white calibration, including dark current calibration and white board calibration

[0063] Utilizing a standard Teflon white board: Teflon white board has very high diffuse reflectivity and can almost evenly reflect all wavelengths of light. This makes it an ideal reference to standardize sensor response. By photographing the Teflon white board, a reference signal close to 100% reflectivity can be obtained to calculate the true reflectivity of the target object.

[0064] The steps are:

[0065] Place a Teflon white plate under the same lighting conditions.

[0066] Make sure the whiteboard is clean and not covered with shadows.

[0067] Use the hyperspectral image acquisition device to align and photograph the Teflon white plate with the same setting parameters.

[0068] Record this reference image, which represents the maximum response of the system under the current lighting conditions.

[0069] Shielding with lens cap: When the lens is completely blocked, theoretically zero incident light should be received. Therefore, this step is used to measure the dark current noise level of the system. Dark current refers to some electrical signal generated by the sensor even in the absence of external light, which needs to be subtracted from the actual measurement to get an accurate result.

[0070] The steps are:

[0071] Close the shutter securely or completely cover the lens with an opaque material such as a lens cap.

[0072] Make sure no light can enter the camera.

[0073] Use the hyperspectral image acquisition device to capture a “dark frame” image with the same setting parameters.

[0074] This dark frame image is a direct reflection of the background noise and will be used to correct the noise component in the original hyperspectral image during post-processing.

[0075] According to formula (1), the original hyperspectral image I of the collected sample is 0 Perform black and white correction to obtain a corrected hyperspectral image I, where each data point in the hyperspectral image I with a size of (1900, 800, 250) represents a digital value of spectral reflectance;

[0076]

[0077] I 0 is the unprocessed original hyperspectral image, I w is the white reference image, I b is a black reference image.

[0078] S3: A moldy tobacco leaf dataset is established based on the corrected hyperspectral image data, and the dataset is divided into a training set, a validation set, and a test set;

[0079] In this embodiment, two front images of tobacco leaf hyperspectral image samples sampled for the third time on the 15th day are selected to construct a data set; when training the model, some pixels in the hyperspectral image are selected as sample points, and a 7*7 area is selected with each sample point as the center, including 250 bands as the input sample of the model in step S4, and the input size is (7, 7, 250). The information of the entire area is the supplementary information for the classification of each pixel point; in order to balance the classification samples, 115126 pixels are evenly selected from the two hyperspectral images for experiments. According to expert judgment and combined with the average spectral reflectance, the sample points are classified into non-moldy (reflectance peak> 0.5), mildly moldy (0.4<reflectance peak≤0.5) and severely moldy (reflectance peak≤0.4), and the pixels are labeled according to the classification type. The training set, validation set and test set are established in a ratio of 3:1:1 by random sampling.

[0080] S4: Construct the MA3D-ResNet model; including the three-dimensional convolution layer I, three-dimensional residual unit I, three-dimensional residual unit II, spatial and spectral attention mechanism, three-dimensional convolution layer II, channel attention mechanism, Flatten layer, and classification layer connected in sequence.

[0081] The MA3D-ResNet model includes a 3D-ResNet network as a framework and adds a multi-attention mechanism, which greatly improves the detection accuracy of mildew, especially early mildew. In this embodiment, a total of 6 three-dimensional convolutional layers are used, each with a different structure.

[0082] The three-dimensional convolution layer I has 32 convolution kernels of size 7×3×3, stride 2×1×1 and padding (1, 1, 1), which are used to extract joint features of space and spectrum.

[0083] The three-dimensional convolution uses a three-dimensional convolution kernel (filter) with a size of (k D ,k H ,k W ), where k D is the size of the convolution kernel in the depth direction, k H is the size of the convolution kernel in the height direction, k W is the size of the convolution kernel in the width direction. The shape of the input data is (D in ,H in ,W in ,C in ), where D in is the spectral depth, H in , W in are height and width, respectively, reflecting the spatial dimension, C in is the number of channels; usually, the number of channels of the convolution kernel is the same as the number of channels of the input data, and the number of convolution kernels determines the number of channels of the output. The output of a three-dimensional convolution is usually a shape of (D out ,H out ,W out ,C out ) of a four-dimensional tensor, where D out is the output depth (usually smaller than the input depth, depending on the convolution kernel size and stride), H out is the output height (similarly, usually smaller than the input height), W out is the output width, C out The number of output channels, that is, the number of convolution kernels determines the number of output feature map channels.

[0084] If the size of the input data is (D in ,H in ,W in ,C in), the size of the convolution kernel is (k D ,k H ,k W ), the step length is (s D ,s H ,s W ), filled with (p D ,p H ,p W ), the size of the output data is:

[0085]

[0086] like Figure 3 As shown, the three-dimensional residual unit I includes a three-dimensional convolution block a and a three-dimensional convolution block b connected in sequence, the output of the three-dimensional convolution block b is residually connected to the input of the three-dimensional convolution block a, and then output after a ReLU activation function, and the input of the three-dimensional convolution block a is connected to the output of the three-dimensional convolution layer I;

[0087] The three-dimensional residual unit II includes a three-dimensional convolution block c and a three-dimensional convolution block d connected in sequence, the output of the three-dimensional convolution block d is residually connected to the input of the three-dimensional convolution block c, and then output after a ReLU activation function, and the input of the three-dimensional convolution block c is connected to the output of the three-dimensional residual unit I;

[0088] The three-dimensional convolution block a, the three-dimensional convolution block b, the three-dimensional convolution block c, and the three-dimensional convolution block d all include three-dimensional convolution layers connected in sequence (including 32 convolution kernels with a size of 7×3×3, a step size of 1×1×1, and padding of (3, 1, 1)), a ReLU activation function, and a batch normalization layer (BatchNorm).

[0089] The addition of the ReLU activation function aims to enhance the model's ability to capture nonlinear features, alleviate the problem of gradient vanishing, and suppress overfitting. The introduction of the batch normalization layer significantly speeds up the convergence of the model. After processing the two convolutional layers and the corresponding activation functions and batch normalization layers, the result is added to the original input. This residual connection enables the network to maintain the flow of gradients during the learning process, further alleviating the problem of gradient vanishing. Finally, the final output is obtained through ReLU mapping. This design not only retains the important information of the original input, but also effectively enhances the features through the learned residual function.

[0090] The spatial and spectral attention mechanism includes a spatial attention mechanism and a spectral attention mechanism, and both the spatial attention mechanism and the spectral attention mechanism are connected to the output of the three-dimensional residual unit II;

[0091] 1. In hyperspectral images, mold may be confined to specific areas of the image, which are particularly critical for mold detection. If a unified processing strategy is used for the entire image, the information in the key areas may be masked by the information in other areas, thereby reducing the recognition ability of mold features. The spatial attention mechanism dynamically adjusts the model's sensitivity to different spatial locations by analyzing the spatial distribution in the feature map to highlight those areas that are closely related to mold detection.

[0092] 2. Each band in a hyperspectral image contains unique spectral information, which contributes to different degrees to mold detection. Some bands may more clearly reflect the spectral characteristics of mold, while other bands may mainly contain noise or irrelevant information. The spectral attention mechanism dynamically adjusts the model's attention to different bands by analyzing the spectral distribution in the feature map to strengthen the spectral features that are most critical to mold detection.

[0093] The three-dimensional convolution layer II has 32 convolution kernels with a size of 123×3×3, a step size of 1×1×1 and padding of (0, 1, 1). The three-dimensional convolution layer II is connected to the fusion output of the spatial attention mechanism and the spectral attention mechanism; the three-dimensional convolution layer II is used to compress and refine features.

[0094] The input of the channel attention mechanism is connected to the output of the three-dimensional convolutional layer II, the output of the channel attention mechanism is connected to the input of the Flatten layer, and the output of the Flatten layer is connected to the input of the fully connected layer I.

[0095] The Flatten layer is used to change the data structure, flattening the three-dimensional data of size (64,7,7) obtained by channel attention into one-dimensional data (3136).

[0096] In the architecture of convolutional neural networks, the channels of each convolutional layer independently capture different feature dimensions of the image. For the specific task of mold detection, the features extracted by different channels have significant differences in their importance for identifying mold. Some channels may be rich in key information closely related to mold, while other channels may only contain marginal or irrelevant features. If these channels are not treated differently, the key features may be weakened in the deep propagation of the network, thereby affecting the overall recognition performance of the model.

[0097] The classification layer includes three fully connected networks connected in sequence and a classifier. The classifier (SoftMax) uses the SoftMax function to output a probability for the result of the fully connected layer. This probability represents the probability of each pixel being non-moldy, slightly moldy, or severely moldy.

[0098] S5: Iteratively train the model using the training set and validation set, and save the model parameters with the highest accuracy on the validation set within the specified number of iterations;

[0099] The training set and validation set data are imported into the constructed MA3D-ResNet model. Each input sample (a three-dimensional cube image block of size (250,7,7)) is passed through the three-dimensional convolution layer I to extract the joint features of space and spectrum, and the output feature map I is obtained. 1 ; Output feature map I 1 The features are enhanced by the calculation of the three-dimensional residual unit I and the three-dimensional residual unit II in turn to obtain the output feature map Y; the output feature map Y is enhanced from two dimensions by the spatial attention and spectral attention calculations to obtain the fusion feature map I 2 ; Output feature map I 2 After the three-dimensional convolution layer II, the features are compressed and refined to obtain the output feature map F C ; Output feature map F C After the channel attention calculation, the output feature map F is obtained; after the output feature map F is flattened, the one-dimensional vector I is obtained. 3 ; One-dimensional vector I 3 After passing through the fully connected layer in the classification layer, the output vector I is obtained 4 , the output vector I 4 After the classifier, the output vector I is converted by the SoftMax function 4 Convert to a probability distribution; make each element between (0,1), indicating the probability of the input sample being non-moldy, slightly moldy, and severely moldy;

[0100] The calculation process of the three-dimensional residual unit is as follows: Taking the calculation process of the three-dimensional residual unit I as an example, the output feature map I 1 After the three-dimensional convolution operation, ReLU activation function, batch normalization operation in the three-dimensional convolution block a, the three-dimensional convolution operation, ReLU activation function, and batch normalization operation in the three-dimensional convolution block b, the output result of the three-dimensional convolution block b and the output feature map I 1 A residual connection is performed, and the residual connection result is activated by the ReLU function to obtain the output of the three-dimensional residual unit I.

[0101] Spatial attention calculation:

[0102] With feature map Y∈R H×W×D As input, H, W, and D are the height, width, and number of bands of the feature map, respectively. First, the feature map Y is reshaped into feature map Y a ∈R N×D , where N = H × W. Then, in the feature map Y a Its transposed feature map Y aTPerform matrix multiplication between them and use the Softmax function to obtain the spatial attention weight matrix M a ∈R N×N , the specific calculation process is as follows:

[0103]

[0104] in, Represents the relationship between the i-th position and the j-th position of the feature map, which is the spatial attention weight matrix M a An element of Y i a Represents the feature map Y a The i-th element in Y j aT Represents the feature map Y aT For the j-th element in , similar feature representations help to generate greater correlation between them. Represents a matrix multiplication operation.

[0105] Afterwards, in the spatial attention weight matrix M a and feature map Y a Perform matrix multiplication between them and reshape the result into feature map R H×W×D Then, add the result to the input feature map Y to get the final output feature map E after spatial attention calculation. a ∈R H×W×D , which contains the spatial features of all positions and highlights the important spatial position information. The feature map E calculated by spatial attention a The calculation formula is:

[0106]

[0107] Spectral attention calculation:

[0108] First, reshape the input feature map Y into feature map Y b ∈R D×N Then in the feature map Y b Instead of transposing the feature map Y bT Finally, the Softmax function is used to generate the spectral attention weight matrix M b ∈R D×D , the calculation formula is as follows:

[0109]

[0110] in, Represents the spectral relationship between band i and band j, which is the spectral attention weight matrix M b Then, the spectral attention weight matrix Mb and Y b The product is reshaped into the feature map R H×W×D , and add the result to the input feature map Y. Output the feature map E after spectral attention calculation b ∈R H×W×D Contains the spectral relationship of all bands and emphasizes important information bands. Feature map E calculated by spectral attention b The calculation formula is:

[0111]

[0112] Finally, the feature map E calculated by spatial attention a And the feature map E calculated by spectral attention b Splice to get the fusion feature map I 2 .

[0113] Channel attention calculation:

[0114] Input feature map F C ∈R H×W×C It is the new fusion feature obtained after the dimensionality reduction by the three-dimensional convolution layer II, where C represents the number of channels after feature reconstruction. The global information of the fusion feature is extracted through average pooling (AvgPooling) and maximum pooling (MaxPooling) FC1 avg and highlight information FC1 max , and then enters the fully connected layer I, and enhances its nonlinear expression ability through the ReLU activation function. Further global information FC1 avg and highlight information FC1 max The output of is fused, and then enters the fully connected layer II, and finally the SoftMax function is used to generate the channel attention weight matrix M C ∈R 1×1×C .

[0115] Specifically, the global information FC1 avg and highlight information FC1 max The calculation formula is as follows:

[0116] FC1 avg =ReLU(FC1(AvgPool(F C )))

[0117] FC1 max =ReLU(FC1(MaxPool(F C )))

[0118] Channel attention weight matrix M C The calculation formula is:

[0119]

[0120] in, represents the accumulation of corresponding pixel values, FC1 represents the fully connected layer I operation function, and FC1 represents the fully connected layer II operation function.

[0121] In the channel attention model, the final generated feature map F is obtained by adding the input feature F C and the channel attention weight matrix M C The calculation formula of the feature map F is as follows:

[0122] F=M C (F C )⊙F C

[0123] Among them, F∈R H×W×C represents the final feature map, and ⊙ represents the element-wise multiplication operation.

[0124] The cross entropy loss is calculated based on the output probability distribution, and the formula is:

[0125]

[0126] Where m is the batch size, which is 100 in this example, and y s is the real sample label, is the predicted probability value corresponding to the real sample label in the probability distribution. The Adam optimizer is used to update the parameters according to the cross entropy loss, the learning rate is 0.0001, the Dropout rate is set to 0.5, and the trained model is verified using the validation set. The classification accuracy of the validation set is monitored. After reaching the specified number of iterations (the number of iterations is set to 100 in this embodiment), the model parameters with the highest accuracy on the validation set are saved.

[0127] S6: Import the model parameters with the highest accuracy obtained through training into the model, input the test set data, calculate the evaluation index according to the model output results, input the hyperspectral image of the redried tobacco leaves to be tested that has been corrected in step S2 into the tested model, and obtain the mold detection result.

[0128] The redried tobacco leaves to be tested are two tobacco leaf hyperspectral images sampled for the third time on the 15th day selected in step S3. During training, only a portion of the pixels corresponding to a three-dimensional cubic image block of size (250, 7, 7) are selected as samples. This input also includes a large number of unseen samples in the two tobacco leaf hyperspectral images.

[0129] The evaluation indicators are classification accuracy of each category, overall accuracy (OA), average accuracy (AA) and Kappa Coefficient.

[0130] The classification accuracy of each category refers to the proportion of the model's correct predictions for a certain category to the total number of that category, usually called the accuracy of the category. The calculation formula is:

[0131]

[0132] Among them, T i F is the number of samples correctly classified for category i (i.e., true positives, True Positives for class). i is the number of samples that are misclassified for category i (i.e., the total number of misclassifications in this category, including false negatives and false positives).

[0133] Overall Accuracy (OA) measures the overall classification accuracy of the model for all categories, indicating the proportion of correctly classified samples in the total number of samples. The calculation formula is:

[0134]

[0135] Where: k is the total number of categories. i is the number of samples correctly classified for category i. N is the total number of samples (the total number of samples of all categories).

[0136] Average Accuracy (AA) is the average of the classification accuracy of each category. It represents the average of the accuracy of each category and is an average measure of the performance of the model in each category. The calculation formula is:

[0137]

[0138] Where: k is the total number of categories. Accuracy of Classi is the classification accuracy of each category i.

[0139] Kappa Coefficient is used to measure the degree of deviation between the classification performance of the model and the random classification results. It is an indicator that reflects the classification consistency (or classification accuracy). It takes into account the influence of accidental consistency and is defined as follows:

[0140]

[0141] Among them: OA is the overall accuracy. E is the accuracy of accidental classification, that is, the accuracy expected by random guessing, which is defined as:

[0142] Where T i +F i is the total number of true samples of category i, T i +F i ′ is the total number of samples of category i that are predicted to be of this category.

[0143] Table 1 Sample size of hyperspectral dataset

[0144]

[0145] Table 2 Recognition accuracy of different models

[0146]

[0147] The experimental results in Table 2 show the performance of the three models in the classification task of moldy redried tobacco leaves, including the recognition accuracy, overall accuracy (OA), average accuracy (AA), and Kappa statistics (K×100) of the three categories of non-moldy, mildly moldy, and severely moldy. From the analysis results, the proposed method has the best classification performance in all categories. In the non-moldy category, the recognition accuracy of the method reached 99.51%, which is almost perfect; in the mildly moldy and severely moldy categories, its accuracy rates were 98.21% and 98.75%, respectively, showing excellent classification capabilities for different degrees of mold. Compared with 1D-CNN, the model also performed well in the non-moldy category with an accuracy of 97.75%, but the recognition rates of mildly moldy and severely moldy were 91.23% and 92.89%, respectively, which were relatively lower. The classification performance of SVM was the most general, with an accuracy of 91.27% for the non-mold category, 83.16% for mild mold, and 86.02% for severe mold, indicating that this method has limited effectiveness in identifying complex mold conditions.

[0148] In terms of overall accuracy (OA) and average accuracy (AA), the proposed method reached 99.12% and 99.09% respectively, further proving that it has extremely high classification accuracy and balance in all categories. In contrast, the OA and AA of 1D-CNN are 95.43% and 95.27% respectively. Although it performs well, it fails to reach the accuracy of the proposed method. The OA of SVM is 90.09% and AA is 89.82%, which is significantly lower than the other two models, especially in the mild mold category. In addition, the Kappa statistic further verifies the advantages of the proposed method. Its Kappa coefficient reaches 98.82, far exceeding 93.90 of 1D-CNN and 86.78 of SVM, indicating that the method not only has high classification accuracy, but also has extremely high consistency in classification results compared with random classification.

[0149] In general, the proposed method is significantly better than 1D-CNN and SVM in all evaluation indicators, especially when dealing with complex features such as mild and severe mildew, it can better capture the subtle differences between different categories and achieve higher classification accuracy and consistency. Therefore, the proposed method shows the best performance in the task of identifying mildewed redried tobacco leaves, proving that it is suitable for fine classification tasks of different degrees of mildew.

[0150] In view of the current problems of difficulty in detecting the degree of mildew in redried tobacco leaves and low accuracy, the accuracy of mildew detection in redried tobacco leaves is effectively improved. In view of the problem that the distribution of mildew in redried tobacco leaves is random, the MA3D-ResNet model of the present invention can automatically pay attention to the local mildew characteristics and output pixel-level discrimination results, which has an excellent effect on early mildew detection.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network, characterized in that: Includes steps: S1: Collect redried tobacco leaf samples and prepare moldy tobacco leaf samples at different stages; S2: using a hyperspectral image acquisition device to collect hyperspectral image data of moldy redried tobacco leaf samples; and correcting the collected hyperspectral image data; S3: A moldy tobacco leaf dataset is established based on the corrected hyperspectral image data, and the dataset is divided into a training set, a validation set, and a test set; S4: Construct the MA3D-ResNet model, including the three-dimensional convolution layer I, the three-dimensional residual unit I, the three-dimensional residual unit II, the spatial and spectral attention mechanism, the three-dimensional convolution layer II, the channel attention mechanism, the Flatten layer, and the classification layer connected in sequence; S5: Iteratively train the model using the training set and validation set, and save the model parameters with the highest accuracy on the validation set within the specified number of iterations; S6: Import the model parameters with the highest accuracy obtained through training into the model, input the test set data, calculate the evaluation index according to the model output results, input the hyperspectral image of the redried tobacco leaves to be tested that has been corrected in step S2 into the tested model, and obtain the mold detection result.

2. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 1, characterized in that: The method for correcting the collected hyperspectral image data is as follows: using a standard Teflon white plate and a lens cover to cover and collect a white reference image I under a white background. w and black reference image I on black background b ; Use the correction formula to correct the hyperspectral image data to obtain the corrected hyperspectral image I. The correction formula is: Among them, I0 is the unprocessed original hyperspectral image, I w is the white reference image, I b is a black reference image.

3. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 2, characterized in that: In step S3, some pixel points in the hyperspectral image are selected as sample points in the moldy tobacco leaf dataset, and then an n*n area is selected with each sample point as the center, containing m bands as input samples of the model described in step S4; based on expert judgment and average spectral reflectance, the sample points are classified into non-moldy, slightly moldy and severely moldy, and the sample points are labeled according to the classification type.

4. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to any one of claims 1 to 3, characterized in that: The three-dimensional residual unit I includes a three-dimensional convolution block a and a three-dimensional convolution block b connected in sequence, the output of the three-dimensional convolution block b is residually connected to the input of the three-dimensional convolution block a, and then output after a ReLU activation function, and the input of the three-dimensional convolution block a is connected to the output of the three-dimensional convolution layer I; The three-dimensional residual unit II includes a three-dimensional convolution block c and a three-dimensional convolution block d connected in sequence, the output of the three-dimensional convolution block d is residually connected to the input of the three-dimensional convolution block c, and then output after a ReLU activation function, and the input of the three-dimensional convolution block c is connected to the output of the three-dimensional residual unit I; The three-dimensional convolution block a, the three-dimensional convolution block b, the three-dimensional convolution block c, and the three-dimensional convolution block d all include three-dimensional convolution layers, ReLU activation functions, and batch normalization layers connected in sequence; the classification layer includes three fully connected networks and a classifier connected in sequence.

5. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 4, characterized in that: The process of iteratively training the model using the training set and the validation set described in step S5 is as follows: the training set and the validation set data are imported into the constructed MA3D-ResNet model, and each input sample is subjected to the three-dimensional convolution layer I to extract the joint features of space and spectrum to obtain the output feature map I1; the output feature map I1 is sequentially enhanced by the calculation of the three-dimensional residual unit I and the three-dimensional residual unit II to obtain the output feature map Y; The output feature map Y is enhanced from two dimensions by spatial attention and spectral attention calculations, and the fused feature map I2 is obtained. The output feature map I2 is compressed and refined by the three-dimensional convolution layer II to obtain the output feature map F. C ; Output feature map F C After the channel attention calculation, the output feature map F is obtained; after the output feature map F is flattened, a one-dimensional vector I3 is obtained; after the one-dimensional vector I3 passes through the fully connected layer of the classification layer, the output vector I4 is obtained, and the output vector I4 passes through the classifier, and the SoftMax function converts the output vector I4 into a probability distribution; so that each element is between (0,1), indicating the probability that the input sample is not moldy, slightly moldy, and severely moldy. In each iteration, the cross entropy loss is calculated according to the output probability distribution, and the parameters are updated using the Adam optimizer based on the cross entropy loss. The model trained in each iteration is verified using the validation set, and the classification accuracy of the validation set is monitored. After reaching the specified number of iterations, the model parameters with the highest accuracy on the validation set are saved.

6. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 5, characterized in that: The method for calculating the spatial attention and spectral attention is: H×W×D Reshape into feature maps Y a ∈R N×D and feature map Y b ∈R D×N , H, W and D are the height, width and number of bands of feature map Y respectively, N = H × W; then the feature map Y a Its transposed feature map Y aT And the feature map Y b Its transposed feature map Y bT Perform matrix multiplication and use the Softmax function to obtain the spatial attention weight matrix M a ∈R N×N and the spectral attention weight matrix M b ∈R D×D , then, respectively in the spatial attention weight matrix M a and feature map Y a And in the spatial spectral attention weight matrix M b and feature map Y b Perform matrix multiplication between them, reshape the multiplication results, and add the reshaped results to the input feature map Y to obtain the feature map E after spatial attention calculation a ∈R H×W×D And the feature map E calculated by spectral attention b , and finally the feature map E calculated by spatial attention a And the feature map E calculated by spectral attention b The fused feature map I2 is obtained by splicing.

7. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 6, characterized in that: The channel attention calculation process is as follows: Input feature map F C ∈R H×W×C First, the global information FC1 of the fusion feature is extracted through average pooling and maximum pooling operations respectively. avg and highlight information FC1 max , then enter the fully connected layer I, then introduce the ReLU activation function, and then enter the fully connected layer II, and use the SoftMax function to get the channel attention weight matrix M C ∈R 1×1×C , the weight matrix M C Multiplying the input feature to get the output feature map F, the formula is: FC1 avg =ReLU(FC1(AvgPool(F C ))) FC1 max =ReLU(FC1(MaxPool(F C ))) F=M C (F C )⊙F C in, Represents the accumulation of corresponding pixel values, FC1 represents the fully connected layer I operation function, FC1 represents the fully connected layer II operation function, and ⊙ represents the element multiplication operation.

8. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 7, characterized in that: The spatial attention weight matrix M a Or the spectral attention weight matrix M b The calculation formula is: M ji is the spatial attention weight matrix M a or M b An element in i Represents the feature map Y a The i-th element or feature map Y b The i-th element in Y j T Represents the feature map Y aT The jth element or feature map Y bT The jth element in represents matrix multiplication operation, L takes N when calculating the spatial attention weight matrix, and takes D when calculating the spectral attention weight matrix; the feature map E calculated by spatial attention a Or the feature map E calculated by spectral attention b The calculation formula is: Among them, reshape is the reshaping operation, and E represents the feature map E after spatial attention calculation a Or the feature map E calculated by spectral attention b , M represents the spatial attention weight matrix M a Or the spectral attention weight matrix M b , Y a,b Represents the feature map Y a Or feature map Y b .

9. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 8, characterized in that: The evaluation indicators are classification accuracy, overall accuracy, average accuracy and Kappa statistic for each category.

10. The method for detecting moldy redried tobacco leaves based on hybrid attention mechanism enhancement and 3D residual CNN network according to claim 9, characterized in that: The method for calculating the cross entropy loss is: Among them, m is the batch size, y s is the real sample label, is the predicted probability value corresponding to the true sample label in the probability distribution.

Images