Curing barn tobacco leaf state identification method based on multi-feature fusion

Abstract

The invention discloses a curing barn tobacco leaf state identification method based on multi-feature fusion. The method is mainly used for identifying the yellowing degree, the browning degree and the drying degree of the tobacco leaves in the baking process. The method comprises three feature extraction modules and a classification network, a pre-training backbone network is used for extracting tobacco leaf image features, a high-frequency color filtering module is used for extracting several color features with the highest occurrence frequency in an image, and a Fourier filtering module is used for extracting tobacco leaf surface wrinkle features. And then splicing and fusing the three features and inputting the three features into a classification network to identify three states. According to the method, in addition to the characteristics of the image, the high-frequency color characteristics of the tobacco leaves in the baking process and wrinkle information generated due to drying are also concerned, various state changes of the tobacco leaves in the baking process are comprehensively considered, and complementary information of different characteristics is utilized, so that the accuracy and robustness of a tobacco leaf state recognition model are improved.

Description

Technical Field

[0001] The invention belongs to the field of intelligent tobacco leaf baking, and provides a method for identifying tobacco leaf states in a baking room based on multi-feature fusion. Background Art

[0002] Flue-cured tobacco is an important part of tobacco agricultural production, which determines the transformation of tobacco leaves from agricultural products with no commercial value to raw materials for the cigarette industry. During the flue-cured tobacco process, personnel are required to be on duty for a long time and judge whether the tobacco leaves meet the requirements at the key temperature stabilization points. At the same time, it is necessary to consider the optimization strategy of process parameters such as temperature stabilization time and humidity control at each stage, and finally complete the curing of a single batch of tobacco leaves. Therefore, accurate judgment of the state of tobacco leaves during the flue-cured tobacco process is a key factor in determining the success or failure of flue-cured tobacco.

[0003] There are three main states of flue-cured tobacco, namely yellowing, dryness and browning, and each state corresponds to multiple levels. Among them, the yellowing degree is divided into 7 levels, the dryness degree is divided into 9 levels, and the browning degree is divided into 6 levels. Such a detailed classification of tobacco leaf states is not only time-consuming and laborious for manual identification, but also easily affected by human subjective consciousness, resulting in inaccurate judgment and recognition errors. Deep neural networks can learn feature details from a large amount of data and model the mapping relationship between labels and data features. Using deep neural networks can not only save labor costs, but also speed up the recognition of tobacco leaf states and improve the accuracy of recognition. Existing research on tobacco leaf state recognition mainly focuses on the time dimension of leaves, and morphological characteristics are often ignored. In addition, these studies usually use a limited number of non-industrial images for training, which is significantly different from the images encountered in real applications. Summary of the invention

[0004] The purpose of the present invention is to propose a real-time identification system and method for field tobacco leaf baking characteristics based on the above-mentioned existing technical conditions, so as to solve the problem that the existing technical indicators of field fresh tobacco leaf baking characteristics are unclear and real-time in-situ identification is difficult.

[0005] In order to overcome the shortcomings of previous technical conditions and judge the flue-cured tobacco state from the tobacco leaf morphology in a real flue-curing barn environment, the present invention proposes a flue-curing barn tobacco leaf state recognition method based on multi-feature fusion.

[0006] The technical solution adopted by the present invention comprises the following steps:

[0007] S1: Acquire tobacco leaf images at different times in an actual flue-curing room environment;

[0008] S2: Extract tobacco leaf image features F img, use the backbone feature extraction network Swin-Transformer V2 to extract image features. The network is pre-trained on the imageNet dataset. Then, the dataset is made from the images collected in step S1. The dataset is divided into a training set, a validation set, and a test set according to the ratio of 8:1:1. The data is applied to the Swin-Transformer V2 network for training to extract image features. The main modules of Swin-Transformer V2 include patch segmentation, patch merging, and attention mechanism;

[0009] S3: Extract high-frequency color features F of tobacco leaves color , a high-frequency color filter module is used for tobacco leaf images to filter out the colors with the highest frequency in tobacco leaf images; the high-frequency color filter module can quantize the color space, and quantization is a process of reducing redundant color noise in an image without changing the overall visual structure of the image;

[0010] S4: Extract tobacco leaf wrinkle information F spect , use the Fourier filter module to extract the surface wrinkle information of the tobacco leaf image; S5: fuse the three features, the three feature maps have the same size, and use concat to splice the three feature maps;

[0011] S6: input the fused feature map into the network to identify the tobacco leaf state;

[0012] In step S1, an image acquisition system is used to collect data (this image acquisition system is a self-developed product, which has been commercialized and is named: BTH100 type flue-curing room image and temperature and humidity integrated acquisition equipment; currently known image acquisition systems can be used as long as the image quality is met (the image is complete, there is no obvious distortion, the color temperature is normal, and the color difference is normal)). The system automatically samples every 10 minutes, and the expert group marks the three states of tobacco leaves: yellowing degree, browning degree and dryness degree. A total of 17,420 images of tobacco leaves during the baking process were collected from 10 major production areas including Henan, Fujian, Yunnan, and Guizhou.

[0013] In step S2, in Swin-Transformer V2, the input tobacco leaf image is segmented into a series of non-overlapping patches of fixed size, and each patch is flattened and mapped to a high-dimensional space as the input of the subsequent attention module. Patch merging refers to splicing and linearly transforming the flattened patches to reduce the spatial size of the feature map while increasing the number of channels. The attention mechanism is the most important module in the network. It is applied separately to each patch. The patch is obtained by three different weight matrices, Q, K, and V, and then feature extraction is performed to obtain the image feature Fimg. Its main formula is:

[0014] Attention(Q,K,V)=SoftMax(cos(Q,K) / γ+B)V,

[0015] Among them, γ is a learnable scalar, Q (Query) is a query vector, which is used to indicate how the currently processed element matches other elements. Each input vector generates a Query vector through a linear transformation. K (Key) is a key vector, which represents the characteristics or content of all input elements. Key is used to perform dot product calculations with Query to measure the correlation between elements. V (Value) is a value vector, which represents the actual information of each element. cos (Q, K) is used to calculate the correlation scores of Q and K, and these correlation scores are normalized by the Softmax function to obtain the attention weight. The attention mechanism performs a weighted average of the Value through the correlation weight to obtain the final attention output. B is a relative position code, which is used to identify the relative position of the current patch after segmentation.

[0016] In step S3, a high-frequency color filter module is used on the tobacco leaf image to filter out the colors with the highest frequency in the tobacco leaf image. The high-frequency color filter module can quantize the color space. Quantization is a process that reduces the redundant color noise in the image without changing the overall visual structure of the image. The high-frequency color filter module can calculate the color distribution histogram of the image, thereby selecting the colors with the highest frequency, which can reduce the influence of color noise on the discrimination of the degree of yellowing and browning. A single fully connected layer will then be used to extract the color feature F color ; In step S4, during the baking process, as the dryness of the tobacco leaves increases, the number of wrinkles on the tobacco leaves will also increase, which is related to the high-frequency information in the Fourier spectrum of the image. Therefore, the tobacco leaf image is converted into a black and white image, and then the two-dimensional Fourier spectrum of the black and white image is calculated. The calculation formula is: Where F(u,v) is the frequency component of the image at the frequency coordinate (u,v), H and W are the height and width respectively, and f(m,n) represents the pixel value of the image at the coordinate (m,n); then the combination F of the real and imaginary components of each frequency position in the Fourier spectrum is used as the input of the neural network.

[0017] The neural network in the above S4 contains four convolutional layers, which can extract high-frequency information and surface wrinkle features related to the dryness of tobacco leaves from the Fourier spectrum, providing more specific features for the subsequent tobacco leaf state classification network, thereby improving the accuracy of discrimination.

[0018] In steps S5 and S6, after extracting the image features in the previous steps S2-S4, the features of the image itself, the wrinkle features related to the degree of dryness, and the color features related to the degree of yellowing and browning are obtained. The three features are fused and input into the final classification network to predict the degree of dryness, browning and yellowing.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] This method introduces the Swin-Transformer V2 pre-trained backbone network for feature extraction, fully utilizes the prior knowledge of a large amount of image data, and combines high-frequency color features and surface wrinkle features with image features. It comprehensively considers the various state changes of tobacco leaves during the baking process, utilizes the complementary information of different features, and further improves the accuracy of the recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a structural diagram of the tobacco leaf status identification system in the present invention.

[0022] Figure 2 Photos of different degrees of wrinkles.

[0023] Figure 3 Statistics of high-frequency and low-frequency intensities at different degrees of wrinkles in the data set (the spectrum at the center of the spectrum 60x60 is set to low frequency, and the remaining positions are high frequency).

[0024] Figure 4 Visualization of the color filtering process: Processing process diagram: (a) original image, (b) quantized cropped image, (c) the color retained after filtering (in pink);

[0025] Figure 4 In the example, the image bit depth is reduced from 8 to 6, the color space is compressed (i.e., the color space is 256→64), and the 20 highest frequency colors are retained after filtering. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention.

[0027] The present invention uses a Swin-Transformer V2 feature extraction network, a high-frequency color filtering module and a Fourier filtering module to process tobacco leaf images obtained from a flue-curing barn. After processing, three types of information of the image are fused and finally input into a classification network to identify the three states of the tobacco leaves.

[0028] like Figure 1As shown, a method for identifying tobacco leaf status in a flue-curing room based on multi-feature fusion comprises the following steps:

[0029] 1) Use the image acquisition system (the image acquisition device has been patented "China National Tobacco Corporation Zhengzhou Tobacco Research Institute. A tobacco leaf baking process information image acquisition device and method: CN202311731784.0[P].

[0030] 2024-04-19.", now publicly available; the image acquisition device has an embedded color optimization algorithm to ensure that images that can reflect the true color state of tobacco leaves are collected in the high temperature and high humidity environment of the curing room. The algorithm has been patented "Zhengzhou Tobacco Research Institute of China National Tobacco Corporation. A method for color tuning of tobacco leaf baking process images: CN202311242506.9[P]. 2023-12-29.", now publicly available) for data collection. The system automatically samples every 10 minutes, and an expert team marks the three states of tobacco leaves: yellowing degree, browning degree and dryness degree. A total of 17,420 images of tobacco leaves during baking were collected from 10 major producing areas including Henan, Fujian, Yunnan and Guizhou.

[0031] 2) Further, the backbone feature extraction network Swin-Transformer V2 is used to extract image features, such as Figure 1 As shown in the "Pre-trained Backbone Network" module in the figure. The network is pre-trained on the imageNet dataset (pre-training: the network is pre-trained for image recognition on ImageNet22K, which contains more than 14 million labeled images and covers more than 20,000 image categories. The specific method is to add a fully connected layer after the backbone network to make it predict the corresponding image category and compare it with the labeled label for training. After training, the fully connected layer is removed and only the weight parameters trained by the backbone network are used), and then the dataset of 17,420 images of tobacco leaves during baking collected by the image acquisition system is fine-tuned to extract image features. The main modules of Swin-Transformer V2 are patch segmentation, patch merging and attention mechanism.

[0032] 2.1) In Swin-Transformer V2, the input tobacco leaf image is segmented into a series of non-overlapping patches of fixed size, and each patch is flattened and mapped into a high-dimensional space as the input of the subsequent attention module.

[0033] 2.2) Patch merging refers to concatenating and linearly transforming the flattened patches to reduce the spatial size of the feature map while increasing the number of channels.

[0034] 2.3) The attention mechanism is the most important module in the network. It is applied to each patch separately. The patch is obtained by three different weight matrices, Q, K, and V, and then feature extraction is performed to obtain the image feature Fimg. Its main formula is:

[0035] Attention(Q,K,V)=SoftMax(cos(Q,K) / γ+B)V,

[0036] Among them, γ is a learnable scalar, Q (Query) is a query vector, which is used to indicate how the currently processed element matches other elements. Each input vector generates a Query vector through a linear transformation. K (Key) is a key vector, which represents the characteristics or content of all input elements. Key is used to perform dot product calculations with Query to measure the correlation between elements. V (Value) is a value vector, which represents the actual information of each element. cos (Q, K) is used to calculate the correlation scores of Q and K, and these correlation scores are normalized by the Softmax function to obtain the attention weight. The attention mechanism performs a weighted average of the Value through the correlation weight to obtain the final attention output. B is a relative position code, which is used to identify the relative position of the current patch after segmentation.

[0037] 3) Use the high-frequency color filter module to filter out the colors with the highest frequency in the tobacco leaf image. The high-frequency color filter module can quantize the color space. Quantization is a process that reduces the redundant color noise in the image without changing the overall visual structure of the image (see Figure 4 ). The high-frequency color filter module can calculate the color distribution histogram of the image, thereby selecting several colors with the highest frequency of occurrence, which can reduce the impact of color noise on the degree of yellowing and browning. A single fully connected layer will then be used to extract the color feature F color ;

[0038] 4) Use the Fourier filter module to extract the surface wrinkle information of the tobacco leaf image. During the baking process, as the dryness of the tobacco leaves increases, the number of wrinkles on the tobacco leaves will also increase (see Figure 2 ), which is related to the high-frequency information in the Fourier spectrum of the image (see Figure 4 ). Therefore, the tobacco leaf image is converted into a black and white image, and then the two-dimensional Fourier spectrum of the black and white image is calculated. The calculation formula is: Where F(u,v) is the frequency component of the image at the frequency coordinate (u,v), H and W are the height and width respectively, and f(m,n) represents the pixel value of the image at the coordinate (m,n); then the combination F of the real and imaginary components of each frequency position in the Fourier spectrum is used as the input of the neural network.

[0039] The above neural network contains four convolutional layers, which can extract high-frequency information and surface wrinkle features related to the dryness of tobacco leaves from the Fourier spectrum, providing more specific features for the subsequent tobacco leaf state classifier, thereby improving the accuracy of discrimination.

[0040] 5) After the above extraction of image features, the features of the image itself, the wrinkle features related to the degree of dryness, and the color features related to the degree of yellowing and browning are obtained. The sizes of the three feature maps are adjusted to be consistent, and the three feature maps are superimposed and spliced ​​using concat.

[0041] 6) The spliced ​​and fused features are input into the final classification network to predict the degree of dryness, browning and yellowing.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion, characterized in that: The following steps are involved: S1: Acquire tobacco leaf images at different times in an actual flue-curing room environment; S2: Extract tobacco leaf image features F img , use the backbone feature extraction network Swin-Transformer V2 to extract image features. The network is pre-trained on the imageNet dataset. Then, the dataset is made from the images collected in step S1. The dataset is divided into a training set, a validation set, and a test set according to the ratio of 8:1:

1. The data is applied to the Swin-Transformer V2 network for training to extract image features. The main modules of Swin-Transformer V2 include patch segmentation, patch merging and attention mechanism; S3: Extract high-frequency color features F of tobacco leaves color , a high-frequency color filter module is used for tobacco leaf images to filter out the colors with the highest frequency in tobacco leaf images; the high-frequency color filter module can quantize the color space, and quantization is a process of reducing redundant color noise in an image without changing the overall visual structure of the image; S4: Extract tobacco leaf wrinkle information F spect , use Fourier filter module to extract surface wrinkle information from tobacco leaf images; S5: Fusion of three features. The three feature maps have the same size. Use concat to concatenate the three feature maps. S6: Input the fused feature map into the network for tobacco leaf state recognition.

2. The method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion according to claim 1, characterized in that: In step S1, an image acquisition system is used to collect data. The system automatically samples every 10 minutes, and a team of experts marks the three states of tobacco leaves: yellowing degree, browning degree, and dryness degree. A total of 17,420 images of tobacco leaves during the baking process were collected from 10 major producing areas including Henan, Fujian, Yunnan, and Guizhou.

3. The method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion according to claim 1, characterized in that: In Swin-Transformer V2 of step S2, patch segmentation means that the input tobacco leaf image will be segmented into a series of non-overlapping patches of fixed size, and each patch is flattened and mapped to a high-dimensional space as the input of the subsequent attention module; patch merging means concatenating and linearly transforming the flattened patches, thereby reducing the spatial size of the feature map and increasing the number of channels; The attention mechanism is the most important module in the network. It is applied to each patch separately. The patch is obtained by three different weight matrices Q, K, V, and then feature extraction is performed to obtain the image feature Fimg. Its main formula is: Attention(Q,K,V)=SoftMax(cos(Q,K) / γ+B)V, Among them, γ is a learnable scalar, Q (Query) is a query vector, which is used to indicate how the currently processed element matches other elements; each input vector generates a Query vector through a linear transformation; K (Key) is a key vector, which represents the characteristics or content of all input elements; Key is used to perform dot product calculations with Query to measure the correlation between elements; V (Value) is a value vector, which represents the actual information of each element; cos (Q, K) is used to calculate the correlation scores of Q and K, and these correlation scores are normalized by the Softmax function to obtain the attention weight; the attention mechanism performs a weighted average of the Value through the correlation weight to obtain the final attention output; B is a relative position code, which is used to identify the relative position of the current patch after segmentation.

4. The method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion according to claim 1, characterized in that: In step S3, the high-frequency color filter module can calculate the color distribution histogram of the image, thereby selecting several colors with the highest frequency of occurrence, which can reduce the influence of color noise on the discrimination of yellowing and browning degrees; then a single fully connected layer will be used to extract the color feature F color .

5. The method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion according to claim 1, characterized in that: In step S4, during the baking process, as the dryness of the tobacco leaves increases, the number of wrinkles of the tobacco leaves will also increase, which is related to the high-frequency information in the Fourier spectrum of the image; therefore, the tobacco leaf image is converted into a black and white image, and then the two-dimensional Fourier spectrum of the black and white image is calculated, and the calculation formula is: Where F(u,v) is the frequency component of the image at the frequency coordinate (u,v), H and W are the height and width respectively, and f(m,n) represents the pixel value of the image at the coordinate (m,n); then the combination F of the real and imaginary components of each frequency position in the Fourier spectrum is used as the input of the neural network.

6. The method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion according to claim 5, characterized in that: The neural network contains four convolutional layers, which can extract high-frequency information and surface wrinkle features related to the dryness of tobacco leaves from the Fourier spectrum, and provide more specific features for the subsequent tobacco leaf state classification network, thereby improving the accuracy of discrimination.

7. The method for identifying tobacco leaf status in a flue-curing barn based on multi-feature fusion according to claim 1, characterized in that: In steps S5 and S6, after the image features are extracted in the previous steps S2-S4, the characteristics of the image itself, the wrinkle characteristics related to the degree of dryness, and the color characteristics related to the degree of yellowing and browning are obtained; the three features are fused and input into the final classification network to predict the degree of dryness, browning and yellowing.

Images