Tobacco automatic grading method based on multispectral feature enhancement network

Abstract

The automated tobacco grading algorithm based on a multispectral feature enhancement network relates to the field of image processing technology. It addresses the problem of low visual feature discrimination capability for tobacco leaves from different origins. This invention constructs an automated tobacco grading system by combining multispectral information with deep learning methods from computer vision. This effectively compensates for the limitations of single-spectral visible light information. Through multi-source complementarity and the designed multispectral feature enhancement module, it further enhances the feature representation of visible and near-infrared spectral features, extracting spatial feature information more conducive to grading, thereby improving the accuracy of tobacco grading. This provides a quantifiable grading reference for current manual grading based on subjective experience, saving manpower and material resources, reducing experience-based errors in manual grading, improving the technical level of the tobacco industry, and promoting the improvement of classification standards in the tobacco industry, thus generating significant socio-economic benefits in tobacco processing.

Description

Technical Field

[0001] This invention relates to the field of analytical chemistry, and more specifically, to an automated tobacco leaf grading method based on a multispectral feature enhancement network. Background Technology

[0002] Tobacco leaf grading is a fundamental task in the tobacco industry when purchasing tobacco. For the commercial production and sale of tobacco products, only through proper grading and the application of targeted formulas to different grades of tobacco can cigarettes be produced to meet the demands of different consumers. This avoids the mixing of superior and inferior grades of tobacco and promotes economic growth in the tobacco market.

[0003] For a long time, tobacco leaf grading has mainly relied on manual grading. Due to complex factors such as differences in sunlight, geographical environment, and soil chemical composition, there are significant quality differences between different tobacco leaves, and their visual appearance lacks obvious distinguishing features. These differences make tobacco leaf grading difficult. Manual grading mainly relies on the experience of practitioners and their subjective judgment, which easily leads to unstable grading results and is also time-consuming and lacks standardized criteria. With the development of computer vision, some studies have attempted to use visual methods for automated tobacco leaf grading. Improving the accuracy, objectivity, and reliability of tobacco leaf grading through new computer vision technologies has become an urgent need for tobacco production enterprises.

[0004] Invention patent CN110893399A discloses an intelligent tobacco leaf grading and sorting device and method based on visual recognition. The system includes image acquisition results, a conveying structure, a sorting structure, and a controller. Its image acquisition structure is only equipped with a visible light camera, lacking multispectral image information. Although the device has a high degree of automation, its reliance solely on visible light information for tobacco leaf grading results in low ability to distinguish visual characteristics of different origins, leaving room for improvement in the grading results.

[0005] Invention patent CN201610684806 discloses a tobacco leaf grading method based on hyperspectral images and deep learning algorithms. This method simultaneously acquires visible light and hyperspectral image data and employs an end-to-end deep belief network for feature extraction and tobacco leaf grading. While this method utilizes information from both visible and near-infrared spectra, the designed end-to-end deep network may miss the spatial dependencies of visible light and hyperspectral image features during network propagation, leading to a decrease in grading accuracy. This invention does not delve into the impact of different network module designs on grading; therefore, the grading effect can be further improved. Summary of the Invention

[0006] The purpose of this invention is to provide an automated tobacco leaf grading method based on a multispectral feature enhancement network.

[0007] The technical solution of the present invention:

[0008] An automated tobacco leaf grading method based on a multispectral feature enhancement network includes the following steps:

[0009] Acquire visible light and near-infrared spectral images of tobacco leaves;

[0010] The near-infrared spectral image is preprocessed to obtain a low-distortion near-infrared spectral image;

[0011] Spatial principal component analysis was used to perform feature dimensionality reduction on low-distortion near-infrared spectral images to obtain visible light image-near-infrared principal component spatial spectral pairs.

[0012] The visible light image-near-infrared principal component spatial spectrum is input into each module of the dual-path multispectral feature enhancement network for processing;

[0013] The multispectral feature enhancement network is used to obtain and display the tobacco leaf grade.

[0014] Preferably, the preprocessing step includes:

[0015] While acquiring visible light and near-infrared spectral images of tobacco leaves, a white board and background were also acquired; the brightness of the white board was recorded as... The brightness of dark current is denoted as ;

[0016] Each channel of the near-infrared spectral image is subjected to black-and-white correction, and low-distortion near-infrared spectral image data is calculated based on the black-and-white correction formula.

[0017]

[0018] in This represents the corrected near-infrared spectral image. This represents the original near-infrared spectral image captured by the camera.

[0019] Preferably, the feature dimensionality reduction step includes:

[0020] Select the sliding window size Slide the raw near-infrared spectral image data;

[0021] If the background area is greater than 90%, skip this window; if the background area is less than or equal to 10% and represents tobacco leaf area, then close this window. Dimensionality reduction was achieved using principal component analysis. ;

[0022] By stitching all windows sequentially, the dimension-reduced near-infrared principal component spatial spectrum is obtained, which, together with the visible light image, forms a visible-near-infrared principal component spatial spectrum image pair. .

[0023] Preferably, the .

[0024] Preferably, the step of processing the visible light image-near-infrared principal component spatial spectrum pair input to each module of the dual-path multispectral feature enhancement network includes the following steps:

[0025] Semantic features of the visible light image-near-infrared principal component spatial spectrum data were extracted using a symmetric feature extractor.

[0026] The common-mode and complementary features of the visible light image-near-infrared principal component spatial spectral pair are enhanced by a multispectral difference sensing module;

[0027] Multispectral features are deeply aggregated through a meta-self-attention feature fusion module.

[0028] Preferably, the step of extracting the semantic features of the visible light image-near-infrared principal component spatial spectral pair data using a symmetric feature extractor includes the following steps:

[0029] Acquiring visible light images and near-infrared principal component spatial spectrum Scaled down to a uniform 428×286 pixels;

[0030] Using ResNet-34 as the backbone network for symmetric feature extraction, and passing through the dual-path symmetric feature extraction layers conv1, conv2_x, conv3_x, conv4_x, and conv5, multi-layer visible light and near-infrared spectral features are obtained. The visible and near-infrared spectral features extracted from the layer are denoted as follows: and ;

[0031] The output of the conv5 layer is used as the output of the symmetric feature extractor, and the visible light and near-infrared spectral features are output as follows: and .

[0032] Preferably, enhancing the common-mode and complementary features of the visible light image-near-infrared principal component spatial spectral pair through the multispectral difference sensing module includes the following steps:

[0033] The first Intermediate layer features extracted from residual blocks of layered backbone network and The complementary feature vectors of the visible light image and the near-infrared spectrum are calculated using the following formula.

[0034]

[0035] in Indicates global average pooling. This represents the Sigmoid activation function. It is a weighted vector along the channel dimension;

[0036] The channel feature differences of multispectral imaging are enhanced by using a weighted vector based on convolution and channel dimensions. The calculation process can be represented as follows:

[0037]

[0038] in This represents element-wise addition. This indicates element-wise multiplication. Represents convolution. One-dimensional convolution kernel parameters representing the visible light feature extraction path;

[0039] Simultaneously, the near-infrared spectral features are processed in the same way, and the calculation process can be expressed as follows:

[0040]

[0041] in One-dimensional convolution kernel parameters representing the near-infrared feature extraction path;

[0042] Complementary parts of the original visible light and near-infrared spectral features Each feature was enhanced, and the enhanced features are: and This feature serves as the next level ( The input to the symmetric feature extractor of the layer.

[0043] Preferably, the deep aggregation of multispectral features through the meta-self-attention feature fusion module includes the following steps:

[0044] Output of the dual-channel symmetric feature extractor and By splicing the data, the fused features are obtained. ;

[0045] For the fusion features Calculate the multi-head attention matrix The specific calculation formula is as follows:

[0046]

[0047] in, This represents the scaling factor. , and Indicates the first The learnable parameter matrix of the size, This represents the softmax activation function;

[0048] Next, the multiple heads are spliced ​​together to form an attention output matrix. The specific calculation formula is as follows:

[0049]

[0050] in, For the output linear layer, Indicates a concatenation character;

[0051] The fully connected layer is used as the output of the meta-self-attention module. This includes the similarity probability of different grades of tobacco leaves; the specific calculation formula is as follows:

[0052] ,

[0053] in This indicates a fully connected layer.

[0054] Beneficial Effects: This invention constructs an automated tobacco leaf grading system by combining multispectral information with deep learning methods from computer vision. This effectively compensates for the limitations of single-spectral visible light information. Through multi-source complementarity and a designed multispectral feature enhancement module, it further enhances the feature representation of visible and near-infrared spectral characteristics, extracting spatial feature information more conducive to grading, thereby improving the accuracy of tobacco leaf grading. It helps provide quantifiable grading references for current manual grading based on subjective experience, saving manpower and material resources for manual grading, reducing experience-based misjudgments, improving the technical level of the tobacco industry, and promoting the improvement of classification standards in the tobacco industry, thus generating significant socio-economic benefits in tobacco leaf processing. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0056] Figure 1 This is a flowchart of the automated tobacco leaf grading method based on a multispectral feature enhancement network according to the present invention.

[0057] Figure 2 This is a flowchart of the spectral data preprocessing process of the present invention.

[0058] Figure 3 This is a schematic diagram of the multimodal feature enhancement network of the present invention. Detailed Implementation

[0059] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0060] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0061] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0062] An automated tobacco leaf grading method based on multispectral feature enhancement networks, such as Figure 1 As shown, the method includes the following steps:

[0063] Step 1: As Figure 2 As shown, original visible light and near-infrared spectral images of tobacco leaves were acquired; the resulting image was shaped as follows. The image pairs are typically RGB three-channel images, while near-infrared spectral images are multi-channel spectra corresponding to multiple wavelength bands. Here, we take a near-infrared spectral image with 250 channels covering wavelengths from 1000 to 2500 nm as an example.

[0064] Step 2: Preprocess the near-infrared spectral image to obtain a low-distortion near-infrared spectral image; while capturing visible light and near-infrared spectral images, the camera simultaneously captures images of a white board with uniform reflectivity. and dark background The brightness of the white board and the brightness of the dark background are obtained, and the influence of stray light caused by the spot characteristics of the optical system is compensated by black and white correction.

[0065] Each channel of the near-infrared spectral image is subjected to black-and-white correction, and low-distortion near-infrared spectral image data is calculated based on the black-and-white correction formula.

[0066]

[0067] in This represents the corrected near-infrared spectral image. This represents the original near-infrared spectral image captured by the camera.

[0068] Step 3: Use spatial principal component analysis to perform feature dimensionality reduction on the low-distortion near-infrared spectral image to obtain the visible light image-near-infrared principal component spatial spectral pair;

[0069] Select the sliding window size Slide the raw near-infrared spectral image data;

[0070] If the background area is greater than 90%, skip this window; if the background area is less than or equal to 10% and represents tobacco leaf area, then close this window. Dimensionality reduction using PCA (Principal Component Analysis) ;

[0071] By stitching all windows sequentially, the dimension-reduced near-infrared principal component spatial spectrum is obtained, which, together with the visible light image, forms a visible-near-infrared principal component spatial spectrum image pair. In this embodiment, .

[0072] Step 4: Input the visible light image-near infrared principal component spatial spectrum into each module of the dual-path multispectral feature enhancement network for processing;

[0073] Step 4.1: Extract the semantic features of the visible light image-near-infrared principal component spatial spectral data using a symmetric feature extractor;

[0074] Step 4.1.1: As Figure 3 As shown, acquiring visible light images and near-infrared principal component spatial spectrum Scaled down to a uniform 428×286 pixels;

[0075] Step 4.1.2: Using ResNet-34 as the backbone network for the symmetric feature extractor, the data passes through the conv1, conv2_x, conv3_x, conv4_x, and conv5 layers of the dual-path symmetric feature extractor to obtain multi-layer visible light and near-infrared spectral features. The visible and near-infrared spectral features extracted from the layer are denoted as follows: and ;

[0076] Step 4.1.3: Use the output of the conv5 layer as the output of the symmetric feature extractor. The visible light and near-infrared spectral features are output as follows: and .

[0077] Step 4.2: Enhance the common-mode and complementary features of the visible light image-near-infrared principal component spatial spectral pair using a multispectral difference sensing module;

[0078] Step 4.2.1: Place the first Intermediate layer features extracted from residual blocks of layered backbone network and The complementary feature vectors of the visible light image and the near-infrared spectrum are calculated using the following formula. ,

[0079]

[0080] in Indicates global average pooling. This represents the Sigmoid activation function. It is a weighted vector along the channel dimension;

[0081] Step 4.2.2: Enhance the channel feature differences of multispectral imaging through convolution and channel-dimensional weighted vectors. The calculation process can be expressed as follows:

[0082]

[0083] in This represents element-wise addition. This indicates element-wise multiplication. Represents convolution. One-dimensional convolution kernel parameters representing the visible light feature extraction path;

[0084] Step 4.2.3: Simultaneously, the near-infrared spectral features are also processed in the same way; the calculation process can be expressed as follows:

[0085]

[0086] in One-dimensional convolution kernel parameters representing the near-infrared feature extraction path;

[0087] Step 4.2.4: Complementary components of the original visible light and near-infrared spectral features Each feature was enhanced, and the enhanced features are: and This feature serves as the next level ( The input to the symmetric feature extractor of the layer.

[0088] Step 4.3: Deeply aggregate multispectral features through the meta-self-attention feature fusion module.

[0089] Step 4.3.1: Output of the dual-channel symmetric feature extractor and By splicing the data, the fused features are obtained. ;

[0090] Step 4.3.2: For the fusion features Calculate the multi-head attention matrix The specific calculation formula is as follows:

[0091]

[0092] in, This represents the scaling factor. , and Indicates the first The learnable parameter matrix of the size, This represents the softmax activation function;

[0093] Step 4.3.3: Next, the multiple heads are concatenated to form an attention output matrix. The specific calculation formula is as follows:

[0094]

[0095] in, For the output linear layer, Indicates a concatenation character;

[0096] Step 4.3.4: Use a fully connected layer as the output of the meta-self-attention module. This includes the similarity probability of different grades of tobacco leaves; the specific calculation formula is as follows: ,in Indicates a fully connected layer;

[0097] Step 4.3.5: Calculate the maximum grade probability index using Softmax and output it as the tobacco grade, then return it to the computer for display.

[0098] The method of this invention is also compared with previous mainstream computer vision-based methods, as shown in Table 1. AlexNet, VGG-16, ResNet-34, and InceptionNet take RGB three-channel visible light images of tobacco leaves as input, while the method of this invention takes visible light images of tobacco leaves and near-infrared principal component spatial spectra as input. The final classification accuracy of the method of this invention is 90.47%, surpassing existing classification methods.

[0099] Table 1. Comparison of classification accuracy between multimodal feature enhancement networks and other mainstream classification networks.

[0100] Model Accuracy (%) AlexNet 81.35 VGG-16 83.05 ResNet-34 84.74 InceptionNet 79.66 Method of the present invention 90.47

[0101] Regarding the inference speed of model deployment, a comparison was made with mainstream Transformers or CNN models, as shown in Table 2. For example, Swin Transformer, InceptionV3, and MobileViT. The method of this invention achieves a single tobacco leaf grading result inference time of 10.58ms on an NVIDIA RTX 3090 device, which is better than other methods, demonstrating the advantages of high grading accuracy and fast computation speed.

[0102] Table 2 Comparison of inference speed between multimodal feature enhancement network and other mainstream classification networks

[0103] Model Delay (ms) GFLOPs Swin Transformer 16.26 8.54 InceptionV3 11.91 2.85 MobileViT 11.47 1.42 Method of the present invention 10.58 7.37

[0104] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. An automated tobacco leaf grading method based on a multispectral feature enhancement network, characterized in that, The method includes the following steps: Acquire visible light and near-infrared spectral images of tobacco leaves; The near-infrared spectral image is preprocessed to obtain a low-distortion near-infrared spectral image; Spatial principal component analysis was used to perform feature dimensionality reduction on low-distortion near-infrared spectral images to obtain visible light image-near-infrared principal component spatial spectral pairs. The visible light image-near-infrared principal component spatial spectrum is input into each module of the dual-path multispectral feature enhancement network for processing, including the following steps: Semantic features of the visible light image-near-infrared principal component spatial spectral pair data are extracted using a symmetric feature extractor; common-mode and complementary features of the visible light image-near-infrared principal component spatial spectral pair are enhanced using a multispectral difference perception module; and multispectral features are deeply aggregated using a meta-self-attention feature fusion module. The step of extracting semantic features from the visible light image-near-infrared principal component spatial spectrum data using a symmetric feature extractor includes the following steps: acquiring the visible light image. and near-infrared principal component spatial spectrum The image is scaled to a uniform 428×286 pixels. Using ResNet-34 as the backbone network for symmetric feature extraction, the image passes through the dual-path symmetric feature extractor layers conv1, conv2_x, conv3_x, conv4_x, and conv5 to obtain multiple layers of visible light and near-infrared spectral features. The visible and near-infrared spectral features extracted from the layer are denoted as follows: and The output of the conv5 layer is used as the output of the symmetric feature extractor, and the visible light and near-infrared spectral features are output as follows: and ; The step of enhancing the common-mode and complementary features of the visible light image-near-infrared principal component spatial spectral pair through the multispectral difference sensing module includes the following steps: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Intermediate layer features extracted from residual blocks of layered backbone network and The complementary feature vectors of the visible light image and the near-infrared spectrum are calculated using the following formula. , ， in Indicates global average pooling. This represents the Sigmoid activation function. It is a channel-dimensional weighted weight vector; by using convolution and the channel-dimensional weighted weight vector, the channel feature differences of multispectral imaging are enhanced. Its calculation process can be expressed as follows: ， in This represents element-wise addition. This indicates element-wise multiplication. Represents convolution. The parameter represents the one-dimensional convolution kernel for the visible light feature extraction path; similarly, the near-infrared spectral features are processed in the same way, and their calculation process can be expressed as follows: ， in One-dimensional convolution kernel parameters representing the near-infrared feature extraction path; complementary parts of the original visible light features and near-infrared spectral features. Each feature was enhanced, and the enhanced features are: and This feature serves as the next level ( The input to the symmetric feature extractor of the layer; The multispectral feature enhancement network is used to obtain and display the tobacco leaf grade.

2. The automated tobacco leaf grading method based on multispectral feature enhancement network according to claim 1, characterized in that, The preprocessing is as follows: While acquiring visible light and near-infrared spectral images of tobacco leaves, a white board and background were also acquired; the brightness of the white board was recorded as... Dark current brightness is denoted as ; Each channel of the near-infrared spectral image is subjected to black-and-white correction, and low-distortion near-infrared spectral image data is calculated based on the black-and-white correction formula. ， in This represents the corrected near-infrared spectral image. This represents the original near-infrared spectral image captured by the camera.

3. The automated tobacco leaf grading method based on multispectral feature enhancement network according to claim 1, characterized in that, The feature dimensionality reduction is as follows: Select the sliding window size Slide the raw near-infrared spectral image data; If the background area is greater than 90%, skip this window; if the background area is less than or equal to 10% and represents tobacco leaf area, then close this window. Dimensionality reduction was achieved using principal component analysis. ; By stitching all windows sequentially, the dimension-reduced near-infrared principal component spatial spectrum is obtained, which, together with the visible light image, forms a visible-near-infrared principal component spatial spectrum image pair. .

4. The automated tobacco leaf grading method based on multispectral feature enhancement network according to claim 3, characterized in that, The .

5. The automated tobacco leaf grading method based on multispectral feature enhancement network according to claim 1, characterized in that, The deep aggregation of multispectral features through the meta-self-attention feature fusion module includes the following steps: Output of the dual-channel symmetric feature extractor and By splicing the data, the fused features are obtained. ; For the fusion features Calculate the multi-head attention matrix The specific calculation formula is as follows: ， in, This represents the scaling factor. , and Indicates the first The learnable parameter matrix of the size, This represents the softmax activation function; Next, the multiple heads are spliced ​​together to form an attention output matrix. The specific calculation formula is as follows: ， in, For the output linear layer, Indicates a concatenation character; The fully connected layer is used as the output of the meta-self-attention module. This includes the similarity probability of different grades of tobacco leaves; the specific calculation formula is as follows: ， in This indicates a fully connected layer.

Images