A Flue-cured Tobacco Leaf Image Grading Method Combining CNN and Transformer

Abstract

A flue-cured tobacco leaf image classification method combining CNN and Transformer. The method first collects flue-cured tobacco leaf images by photographing them to establish a flue-cured tobacco leaf dataset. The flue-cured tobacco leaf dataset is then preprocessed. The preprocessed flue-cured tobacco leaf dataset is then input into a flue-cured tobacco leaf image classification model for classification training. A positioning module obtains target locations and key local regions of the tobacco leaf images, calculates an original loss by the positioning module, and updates the parameters of the flue-cured tobacco leaf image classification model based on the original loss. A classification module then obtains classification results based on the key local regions, calculates a classification loss by the classification module, and updates the parameters of the flue-cured tobacco leaf image classification model based on the classification loss to obtain an optimized flue-cured tobacco leaf image classification model. Finally, the flue-cured tobacco leaf images to be classified are input into the trained flue-cured tobacco leaf image classification model, and the classification results of the tobacco leaves are output. The method is reliable, effective, and has good classification effects.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a flue-cured tobacco leaf image grading method combining CNN and Transformer. Background Art

[0002] Tobacco is a key agricultural pillar in my country. Grading, the process of grading tobacco leaves based on their quality, characteristics, and degree of excellence, is a crucial step in tobacco production. The effectiveness of tobacco grading significantly impacts the ultimate economic benefits for tobacco farmers. Although my country has established relatively comprehensive tobacco grading standards, the current tobacco grading and procurement process still primarily relies on manual grading, relying on the sensory input and experience of graders. This traditional manual grading method not only suffers from high labor intensity, low efficiency, and high labor costs, but is also influenced by factors such as the experience, sensory perception, and subjective evaluation of graders, making it difficult to ensure objective grading and affecting grading accuracy. Therefore, research on tobacco grading methods holds significant scientific significance and practical application value.

[0003] With the gradual development and maturity of image-based science and technology, digital image processing techniques are increasingly being applied to tobacco leaf grading. Traditional tobacco leaf image grading algorithms extract multiple features from images to grade tobacco leaves. This approach not only requires a deep understanding of tobacco leaf characteristics through experience, but also extensive testing to verify the accuracy of the extracted features and their relevance to tobacco leaf grading. Furthermore, for features that are difficult to quantify from images, such as leaf identity and oil content, tobacco leaf grading using traditional image algorithms alone is extremely challenging. In recent years, deep learning has made significant progress, finding widespread applications in fields such as face recognition, object detection, and semantic segmentation. Computer vision has also achieved significant breakthroughs in tobacco leaf image processing. Convolutional neural networks (CNNs) can extract higher-level abstract features from tobacco leaf images, addressing the parameter extraction challenges of traditional image processing algorithms. Therefore, the use of computer technology to replace traditional manual tobacco leaf grading methods will become a mainstream trend in tobacco grading.

[0004] Image classification tasks in computer vision are categorized into coarse-grained and fine-grained classification. Coarse-grained classification involves large differences between classes, while fine-grained classification involves a more detailed subdivision of categories, such as bird species or car models. Fine-grained classification is characterized by the close similarity of images across different categories; the differences between images of different categories are almost as small as, or even smaller than, those between images of the same category. This presents a significant challenge for fine-grained classification. Methods for identifying strong discriminative regions to enhance fine-grained visual classification can be categorized into three types. The first method uses a Region Proposal Network (RPN), the second method enhances feature maps through an attention mechanism, and the third method uses a self-attention mechanism to enhance fine-grained visual classification. The first two methods are primarily based on CNNs, while the third method leverages the Vision Transformer to achieve fine-grained vision. The Vision Transformer is a fully Transformer-based image recognition model without any standard convolutional layers. However, this pure Transformer requires pre-training with a particularly large dataset. Otherwise, due to the lack of CNN's inductive bias characteristics and strong constraints such as dropout, the model will not generalize well and will easily overfit. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a flue-cured tobacco leaf image grading method combining CNN and Transformer to improve the tobacco leaf classification ability.

[0006] In order to solve the above technical problems, the present invention adopts the following technical method: a flue-cured tobacco leaf image grading method combining CNN and Transformer, comprising the following steps:

[0007] Step S1, taking photos of flue-cured tobacco leaves on site to establish a flue-cured tobacco leaf dataset;

[0008] Step S2, preprocessing the flue-cured tobacco leaf dataset;

[0009] Step S3, constructing a flue-cured tobacco leaf image classification model including a CNN-based positioning module and a CNN and Transformer-based classification module;

[0010] Step S4: inputting the preprocessed flue-cured tobacco leaf dataset into the flue-cured tobacco leaf image classification model for classification training. During training, the tobacco leaf images in the flue-cured tobacco leaf dataset are first positioned by the positioning module to obtain the target position, and then the key local area of the tobacco leaf image is determined. The positioning module calculates the original loss of the tobacco leaf image, and then the gradient is back-propagated according to the original loss to update the parameters of the flue-cured tobacco leaf image classification model. Then, the key local area is processed by the classification module to obtain a classification result, and the classification module calculates the classification loss of the key local area. The gradient is back-propagated according to the classification loss to update the parameters of the flue-cured tobacco leaf image classification model again, thereby obtaining an optimized flue-cured tobacco leaf image classification model.

[0011] Step S5: input the flue-cured tobacco leaf image to be classified into the optimized and trained flue-cured tobacco leaf image classification model, and output the classification result of the tobacco leaf.

[0012] Furthermore, in step S2, when preprocessing the flue-cured tobacco leaf dataset, the tobacco leaf image is first randomly inverted and then cropped to a size of 224x224.

[0013] Furthermore, in step S4, the positioning module performs the following positioning processing on the tobacco leaf image in the flue-cured tobacco leaf dataset to obtain the target position of the tobacco leaf image:

[0014] The positioning module first extracts features from tobacco leaf images in the flue-cured tobacco leaf dataset through its CNN to obtain a feature map, and then aggregates the feature map in the channel dimension to obtain an activation map. F , as shown in expression (1):

[0015] (1)

[0016] in, For the i The feature map of each channel, C is the number of image channels;

[0017] Then according to the activation map F Find the average , as shown in expression (2):

[0018] (2)

[0019] Where W is the image width, H is the image height, x, y are the horizontal and vertical coordinate values of a certain position on the feature map respectively;

[0020] Then the average As a threshold to judge the activation map F Whether the element at each position in belongs to the target object;

[0021] Finally, a rough mask image is obtained according to the following expression (3): ,The minimum bounding box containing the largest connected area in the mask image is the target position of the tobacco leaf image;

[0022] (3).

[0023] Furthermore, in step S4, after obtaining the target position of the tobacco leaf image, the key local area of the tobacco leaf image is first determined according to the coordinates of the target position, and then the image of the key local area is cropped to a specified size using a sampling function.

[0024] Furthermore, the CNN of the classification module adopts an efficient net network.

[0025] Furthermore, in step S4, the classification module processes the key local areas sequentially through the Efficient Net network and the Transformer to obtain the classification results of the tobacco leaves, wherein:

[0026] Efficient Net: extracts features from images sampled to key local areas of a specified size.

[0027] Transformer: First, the feature map obtained by the Efficient Net network is upgraded through a standard convolution of Patch Embedding. Then, the upgraded feature map is converted into two-dimensional data through Flatten. Then, the trainable parameter class token and position information Position Embedding are added. Finally, after passing through multiple stacked Transformer Encoders, the MLP head is used to extract the class token to obtain the tobacco leaf classification result.

[0028] Furthermore, the number of the Transformer Encoders is six, and the Transformer Encoder includes a feedforward neural network FFN and a multi-head self-attention layer MSA.

[0029] Furthermore, in step S4, the classification loss calculated by the classification module includes the loss calculated by the efficient net network and the loss calculated by the transformer. The positioning module uses the cross entropy loss function when calculating the original loss, the efficient net network loss, and the transformer loss. The cross entropy loss function is as follows (4):

[0030] (4)

[0031] in, x is the predicted value of tobacco leaf classification, class is the true value of tobacco leaf classification, j is the number of tobacco leaf categories.

[0032] Preferably, in step S4, the preprocessed flue-cured tobacco leaf dataset is input into the flue-cured tobacco leaf image classification model for 200 rounds of classification training.

[0033] Preferably, in step S4, when backpropagating the gradient to update the parameters of the flue-cured tobacco leaf image classification model according to the original loss, and when backpropagating the gradient to update the parameters of the flue-cured tobacco leaf image classification model again according to the classification loss, the SGD optimization algorithm is adopted.

[0034] The present invention provides a flue-cured tobacco leaf image grading method that combines CNN and Transformer. It primarily utilizes a novel flue-cured tobacco leaf image classification model that combines CNN and Transformer to classify tobacco leaves. The flue-cured tobacco leaf image classification model includes two parts: a positioning module and a classification module. The positioning module extracts tobacco leaf image features through CNN to obtain a feature map, which then sequentially forms an activation map and a mask map to locate key local regions of the image, namely, the tobacco leaves. These key local regions are fine-grained regions. The classification module utilizes the inherent inductive bias characteristics of CNN and the advantages of Transformer in processing high-level visual information to extract features of fine-grained region information, further extracting tobacco leaf image features to predict tobacco leaf classification results. These two modules work in synergy, fully utilizing both local and global information in tobacco leaf images, significantly improving the model's classification performance. Therefore, the present invention provides an effective, reliable, and efficient method for grading flue-cured tobacco leaf images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of the flue-cured tobacco leaf image grading method combining CNN and Transformer according to the present invention;

[0036] Figure 2 This is a schematic diagram of the training of a flue-cured tobacco leaf image classification model in the flue-cured tobacco leaf image grading method combining CNN and Transformer according to the present invention;

[0037] Figure 3 This is a working diagram of the positioning module in the flue-cured tobacco leaf image grading method combining CNN and Transformer according to the present invention;

[0038] Figure 4This is a schematic diagram of the working of the classification module in the flue-cured tobacco leaf image grading method combining CNN and Transformer involved in the present invention. DETAILED DESCRIPTION

[0039] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0040] Tobacco leaf classification is a type of fine-grained classification. The national standard for flue-cured tobacco classifies tobacco leaves based on their location on the bead, their color, and other appearance factors. Because the background of the collected tobacco leaf images is relatively simple, many patches that do not contain the target object are obtained when the feature map is embedded in the Transformer and segmented in the classification model. Therefore, it is considered to combine a positioning module to first locate the tobacco leaves in the image. The classification module then uses the information of the fine-grained regions of the tobacco leaf image obtained by the positioning module to further enhance the classification capability. In view of this, the present invention provides a flue-cured tobacco leaf image grading method that combines CNN and Transformer, as follows.

[0041] like Figure 1 As shown in the figure, a flue-cured tobacco leaf image grading method combining CNN and Transformer mainly includes two stages: training and prediction.

[0042] 1. Training Phase

[0043] 1) Collect tobacco leaf data on-site. Take photos of tobacco leaf images under natural lighting conditions, establish a flue-cured tobacco leaf dataset, and classify the collected tobacco leaf images.

[0044] 2) Data preprocessing: The tobacco leaf images in the flue-cured tobacco leaf dataset were randomly inverted and then cropped to a size of 224x224.

[0045] 3) Construct a flue-cured tobacco leaf image classification model that includes a CNN-based positioning module and a CNN and Transformer-based classification module.

[0046] 4) The preprocessed flue-cured tobacco leaf dataset is input into the flue-cured tobacco leaf image classification model for classification training. During training, the working processes of the positioning module and the classification module are as follows.

[0047] 1. Positioning module

[0048] The positioning module first extracts features from tobacco leaf images in the flue-cured tobacco leaf dataset through its CNN to obtain a feature map, and then aggregates the feature map in the channel dimension to obtain an activation map. F , as shown in expression (1):

[0049] (1)

[0050] in, For the i The feature map of each channel, C is the number of image channels.

[0051] Then according to the activation map F Find the average , as shown in expression (2):

[0052] (2)

[0053] Where W is the image width, H is the image height, x, y are the horizontal and vertical coordinate values of a certain position on the feature map respectively.

[0054] Then the average As a threshold to judge the activation map F Whether the element at each position in belongs to the target object.

[0055] Finally, a rough mask image is obtained according to the following expression (3): The minimum bounding box containing the largest connected area in the mask image is the target position of the tobacco leaf image. The key local area of the tobacco leaf image can be determined according to the coordinates of the target position. Then, the sampling function is used to crop the image of the key local area to the specified size. Specifically, the sampling function here refers to the torch.nn.functional.interpolate() function.

[0056] (3)

[0057] like Figure 2 and Figure 3 As shown, while the positioning module obtains the target position of the tobacco leaf image, the positioning module calculates the original loss of the tobacco leaf image and back-propagates the gradient according to the original loss to update the parameters of the flue-cured tobacco leaf image classification model.

[0058] 2. Classification module

[0059] In this embodiment, the CNN of the classification module adopts an efficient net network, such as Figure 4As shown in the figure, the EfficientNet network is mainly composed of the inverted residual structure MBConv. The inverted residual structure MBConv uses depthwise separable convolution to reduce the amount of computation and parameters. The inverted residual structure MBConv includes the Squeeze and Excitation module, namely the SE module. The SE module is a channel attention module. Through this module, the network can learn to use global information to selectively emphasize information features and suppress features that are not very useful for the current task.

[0060] like Figure 4 As shown in the figure, the classification module first uses its Efficient Net network to extract features from the image of the key local area of the specified size sampled by the aforementioned positioning module. Then, the feature map obtained by the Efficient Net network is converted into two-dimensional data through Patch Embedding in the Transformer (specifically, the feature map obtained by the Efficient Net network is first upgraded in dimension through a normal convolution of Patch Embedding, and then the upgraded feature map is converted into two-dimensional data through Flatten). The trainable parameter class token is then spliced in, and the position information Position Embedding is also spliced in. Finally, after passing through six stacked Transformer Encoders (each Transformer Encoder has the same structure, mainly consisting of a feed-forward neural network FFN and a multi-head self-attention layer MSA), the MLP head is used to extract the class token to obtain the tobacco leaf classification result.

[0061] like Figure 2 and Figure 4 As shown in the figure, while the classification module obtains the classification result, the Efficient Net network in the classification module calculates its own loss, and the Transformer also calculates its own loss. The two losses are combined to form the classification loss of the key local area. Based on the classification loss, the gradient is backpropagated to update the parameters of the flue-cured tobacco leaf image classification model again, thereby obtaining the optimized flue-cured tobacco leaf image classification model.

[0062] It is worth mentioning that the nn.CrossEntropyLoss() cross entropy loss function is called when the positioning module calculates the original loss, the efficient net network calculates the loss, and the Transformer calculates the loss. The cross entropy loss function is as follows (4):

[0063] (4)

[0064] in, xis the predicted value of tobacco leaf classification, class is the true value of tobacco leaf classification, j is the number of tobacco leaf categories.

[0065] In addition, in order to enhance the optimization effect of the flue-cured tobacco leaf image classification model, the number of classification training for the flue-cured tobacco leaf image classification model in this embodiment is 200 rounds, and the SGD optimization algorithm is used when the gradient is back-propagated to update the parameters of the flue-cured tobacco leaf image classification model according to the original loss, and when the gradient is back-propagated to update the parameters of the flue-cured tobacco leaf image classification model again according to the classification loss.

[0066] 2. Prediction Stage

[0067] 1) Input the flue-cured tobacco leaf image to be classified into the optimized and trained flue-cured tobacco leaf image classification model, and output the classification result of the tobacco leaf.

[0068] Compared with the traditional flue-cured tobacco leaf image grading method, the advantages of the present invention are obvious.

[0069] First, using Transformer to process computer vision tasks requires converting three-dimensional image data into two-dimensional data, which is achieved through Patch Embedding, that is, slicing. The background of the collected tobacco leaf images is relatively simple. It can be seen that many patches obtained through Patch Embedding are almost pure-color backgrounds that do not contain tobacco leaf objects. Such patches will appear widely in data of different categories. If these patches are input into Transformer for training, it is easy to see that the model effect will not be very good. On the contrary, if as many patches containing distinguishable objects as possible are used for training, the model effect can be further improved. Therefore, the present invention obtains the target position of the tobacco leaf in the image through the positioning module, and then the classification module uses the information of the fine-grained area obtained by the positioning module to further enhance its classification ability.

[0070] Secondly, the present invention adopts a hybrid structure combining CNN and Transformer to perform feature extraction. Transformer has shown strong performance in traditional classification tasks. Its advantage is that it uses attention to capture global contextual information to establish a long-range dependence on the target, thereby extracting more powerful features and paying more attention to the global information of the image. Since Transformer has relatively high requirements for the size of the data set, it needs to be trained with a particularly large data set to obtain better results. Therefore, the present invention combines CNN and Transformer, and uses the inductive bias characteristics of CNN and the advantage of Transformer in capturing high-level visual information to classify tobacco leaves, so as to enhance the generalization performance of the model and thus improve the classification effect of the model.

[0071] The above embodiments are preferred implementation schemes of the present invention. In addition, the present invention can also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the scope of protection of the present invention.

[0072] In order to make it easier for ordinary technicians in this field to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified, and for the sake of clarity, some other elements are omitted in this application document. Ordinary technicians in this field should realize that these omitted elements may also constitute the content of the present invention.

Claims

1. A flue-cured tobacco leaf image grading method combining CNN and Transformer, characterized in that: The following steps are involved: Step S1, taking photos of flue-cured tobacco leaves on site to establish a flue-cured tobacco leaf dataset; Step S2, preprocessing the flue-cured tobacco leaf dataset; Step S3, constructing a flue-cured tobacco leaf image classification model including a CNN-based positioning module and a CNN and Transformer-based classification module; Step S4: inputting the preprocessed flue-cured tobacco leaf dataset into the flue-cured tobacco leaf image classification model for classification training. During training, the tobacco leaf images in the flue-cured tobacco leaf dataset are first positioned by the positioning module to obtain the target position, and then the key local area of the tobacco leaf image is determined. The positioning module calculates the original loss of the tobacco leaf image, and then the gradient is back-propagated according to the original loss to update the parameters of the flue-cured tobacco leaf image classification model. Then, the key local area is processed by the classification module to obtain a classification result, and the classification module calculates the classification loss of the key local area. The gradient is back-propagated according to the classification loss to update the parameters of the flue-cured tobacco leaf image classification model again, thereby obtaining an optimized flue-cured tobacco leaf image classification model. The positioning module performs the following positioning processing on the tobacco leaf images in the flue-cured tobacco leaf dataset to obtain the target position of the tobacco leaf images: The positioning module first extracts features from tobacco leaf images in the flue-cured tobacco leaf dataset through its CNN to obtain a feature map, and then aggregates the feature map in the channel dimension to obtain an activation map. F , as shown in expression (1): （1） in, For the i The feature map of each channel, C is the number of image channels; Then according to the activation map F Find the average , as shown in expression (2): （2） Where W is the image width, H is the image height, x, y are the horizontal and vertical coordinate values of a certain position on the feature map respectively; Then the average As a threshold to judge the activation map F Whether the element at each position in belongs to the target object; Finally, a rough mask image is obtained according to the following expression (3): ,The minimum bounding box containing the largest connected area in the mask image is the target position of the tobacco leaf image; （3） The CNN of the classification module adopts the Efficient Net network; the classification module processes the key local areas through the Efficient Net network and the Transformer in sequence to obtain the classification results of the tobacco leaves, wherein: Efficient Net: extracts features from images sampled to key local areas of a specified size. Transformer: First, the feature map obtained by the Efficient Net is upgraded through a standard convolution of Patch Embedding. Then, the upgraded feature map is converted into two-dimensional data through Flatten. Then, the trainable parameter class token and position information Position Embedding are added. Finally, after passing through multiple stacked Transformer Encoders, the MLP head is used to extract the class token to obtain the tobacco leaf classification result. Step S5: input the flue-cured tobacco leaf image to be classified into the optimized and trained flue-cured tobacco leaf image classification model, and output the classification result of the tobacco leaf.

2. The flue-cured tobacco leaf image grading method combining CNN and Transformer according to claim 1, characterized in that: In step S2, when preprocessing the flue-cured tobacco leaf dataset, the tobacco leaf images are first randomly inverted and then cropped to a size of 224x224.

3. The flue-cured tobacco leaf image grading method combining CNN and Transformer according to claim 2, characterized in that: In step S4, after obtaining the target position of the tobacco leaf image, the key local area of the tobacco leaf image is first determined according to the coordinates of the target position, and then the image of the key local area is cropped to a specified size using a sampling function.

4. The flue-cured tobacco leaf image grading method combining CNN and Transformer according to claim 3, characterized in that: The number of the Transformer Encoders is six, and the Transformer Encoder includes a feedforward neural network FFN and a multi-head self-attention layer MSA.

5. The flue-cured tobacco leaf image grading method combining CNN and Transformer according to claim 4, characterized in that: In step S4, the classification loss calculated by the classification module includes the loss calculated by the efficient net network and the loss calculated by the transformer. The positioning module uses the cross entropy loss function when calculating the original loss, the efficient net network loss, and the transformer loss. The cross entropy loss function is as follows (4): （4） in, x is the predicted value of tobacco leaf classification, class is the true value of tobacco leaf classification, j is the number of tobacco leaf categories.

6. The flue-cured tobacco leaf image grading method combining CNN and Transformer according to claim 5, characterized in that: In step S4, the preprocessed flue-cured tobacco leaf dataset is input into the flue-cured tobacco leaf image classification model for 200 rounds of classification training.

7. The flue-cured tobacco leaf image grading method combining CNN and Transformer according to claim 6, characterized in that: In step S4, when backpropagating the gradient to update the parameters of the flue-cured tobacco leaf image classification model according to the original loss, and when backpropagating the gradient to update the parameters of the flue-cured tobacco leaf image classification model again according to the classification loss, the SGD optimization algorithm is adopted.

Images