A crop disease recognition method based on local selection and feature interaction

Abstract

The application discloses a crop disease identification method based on local selection and feature interaction, which adopts a deep learning image recognition technology to identify and classify crop disease images. The application firstly performs data preprocessing on the disease images, and then extracts leaf disease features through an LSFI-ViT model constructed by the application. The LSFI-ViT takes a ViT as a backbone network to extract global features of the whole image, a local selection module LSM and a local-global feature interaction attention encoder LGFIA-Former are connected in series to form an extended branch, the LSM is responsible for selecting a local area with distinguishing features in the image, and the LGFIA-Former simultaneously processes local and global feature sequence inputs through the LGFIA to obtain interactive features of the local-global. Two groups of features are respectively sent to different fully connected layers to obtain two groups of prediction results, and finally, effective identification of different crop diseases and severity is realized in an integrated voting manner. The application effectively solves the problem of unsatisfactory crop disease and severity identification.

Description

A Crop Disease Identification Method Based on Local Selection and Feature Interaction Technical Field

[0001] The technical solution of this invention relates to image recognition technology for plant leaf diseases, specifically a crop disease recognition method based on Local Selection and Feature Interaction (ViT, LSFI-ViT). Background Technology

[0002] Currently, crop disease identification in my country mainly relies on manual observation and experience-based judgment, which is highly subjective and has low accuracy. With the rapid development of artificial intelligence technology, deep learning-based image recognition technology has demonstrated high efficiency and accuracy in disease identification tasks, effectively replacing time-consuming and labor-intensive manual identification methods.

[0003] Most deep learning-based disease identification methods employ Convolutional Neural Networks (CNNs). CN111951227A discloses a disease identification method based on residual convolutional neural networks, but this method only targets common rice diseases and lacks robustness and generalization when identifying multiple crop diseases. CN115587296A discloses a crop disease identification method based on transfer learning and step-by-step identification. This method trains VGG and ResNet separately after transfer learning to identify crop varieties and disease types, but it lacks the ability to perform end-to-end training. CN112699896A discloses a fine-grained crop disease identification method based on MDFC-ResNet. This method integrates three ResNet networks to extract features at different granularities, leading to feature redundancy and high computational costs while acquiring multi-granular features. Furthermore, although CNNs can effectively extract feature representations of local lesions, they are limited by local receptive fields, and the dependencies between lesions in different locations are not fully established, resulting in poor performance in disease identification tasks with high similarity. Transformer is a deep learning architecture based entirely on attention mechanisms. Compared to CNNs, Transformer utilizes self-attention to achieve a large or even global receptive field, enabling the establishment of long-distance correlations between different locations in an image. It is gradually replacing CNNs and becoming the preferred architecture for various image recognition tasks. In crop disease recognition tasks, CN113723312A integrates two Transformer architectures to identify rice diseases, improving recognition accuracy. However, this method only constructs an overall image feature representation based on Transformer, neglecting the importance of local discriminative features for fine-grained disease identification.

[0004] Disease identification tasks are often categorized according to different crop varieties, disease types, and severity. Generally, the features relevant to identifying crop varieties come from the entire leaf surface and its edges, relying heavily on the global feature representation of the image. Discriminating features for identifying disease types and severity, however, are relatively concentrated, mainly existing in the local areas where lesions are located. Existing work mostly focuses on identification from the overall image perspective, lacking attention to the features of local high-response areas, resulting in less than ideal performance in identifying specific diseases and severity levels. Therefore, how to integrate local information to learn the detailed features of disease images should be a major research direction. Summary of the Invention

[0005] To address the lack of local detailed features in existing methods, this invention aims to propose a crop disease identification method based on LSFI-ViT. This method uses ViT as the backbone network of the identification model, leveraging the global modeling advantages of Transformer to overcome the shortcomings of CNN architecture in modeling long-range correlations. By designing reasonable functional modules to extend the ViT network, it selects local regions containing detailed features and effectively integrates these features, thus solving the problem of difficulty in distinguishing between disease types and severity.

[0006] The technical solution adopted by this invention to solve this technical problem is: a crop disease identification method based on LSFI-ViT, which includes two core modules: LSM (Local Selection Module) and LGFIA (Local Global Feature Interaction Attention). LSM utilizes the attention matrix generated by the ViT self-attention module to obtain the position indices corresponding to high-response image tokens, thereby achieving the selection of local high-response regions. LGFIA is an improvement on the self-attention module; replacing the self-attention module in the Transformer with LGFIA yields LGFIA-Former (Local Global Feature Interaction Attention Transformer). LGFIA-Former takes both the local token representing the high-response region and the global token of the image as input, and achieves information interaction through attention calculation, embedding local detailed features for disease identification. Using LSM and LGFIA-Former, this method constructs an extended model, LSFI-ViT. The backbone network of this model is responsible for learning the global features of the image, while the extended branches acquire the interaction features of local and global features during the learning process. The representation vectors of the two features are integrated and voted through a fully connected layer to output the final prediction result.

[0007] A crop disease identification method based on local selection and feature interaction LSFI-ViT, the specific steps of which are as follows:

[0008] Step 1: Disease data acquisition and preprocessing:

[0009] Step 1.1, Obtaining the Crop Disease Dataset: This invention uses the open-source dataset AI Challenger 2018 for training and testing the disease identification model. This dataset involves 26 types of diseases in 10 crops. In addition to healthy samples, diseased samples have two different severity levels: general and severe. All samples are divided into 61 categories according to crop variety-disease type-severity. Samples of different categories are placed in different datasets for data labeling. The category labels are 0-60. Among them, the samples in categories 44 and 45 have only 3 images in total. Since the amount of data in these two categories is insufficient to support the model to learn the relevant features, these two categories of samples are removed. The category labels of the remaining samples are 0-58. The dataset is divided into training set and test set.

[0010] Step 1.2: In order to further enrich the dataset samples and help the model training, the training set data is preprocessed, including: resizing images, random cropping, random flipping, and standardization.

[0011] The second step is to construct the LSFI-ViT model:

[0012] Step 2.1: Input the disease images obtained in Step 1.2 into the recognition model. The dimensions of a single image are 224×224×3. Use a convolutional layer with a kernel size of 16×16, a stride of 16, and 768 output channels to achieve non-overlapping partitioning of image blocks. Each image block is represented as a token, i.e., a vector with dimensions of 1×1×768. All image blocks form the image feature map F. 2.1 The dimensions are 14×14×768;

[0013] Step 2.2, F 2.1 Flattening the image along the first two dimensions outputs the sequence feature representation of the entire image [T]. global ] 2.2 The dimensions are 196×768;

[0014] Step 2.3: Generate a trainable classification head T cls The dimension is 1×768, T cls T is responsible for continuously updating and aggregating global interaction information between different image patches in different layers of the subsequent network. cls With [T] global ] 2.2 The feature sequence [T] is obtained by splicing. cls ,T global] 2.3 The dimensions are 197×768.

[0015] Step 2.4, generate a [T] global ] 2.3 A tensor with the same dimensions and size is initialized, and this tensor is related to [T]. cls ,T global ] 2.3 The purpose of performing the addition operation is to encode the absolute position information of each token in the feature sequence, resulting in the feature sequence [T]. cls ,T global ] 2.4 The dimensions are 197×768;

[0016] Step 2.5, for the feature sequence [T] cls ,T global ] 2.4 Seven consecutive Transformer encoding operations are performed, each using a self-attention module with 768 channels and 12 heads. The input and output channels are both 768, the hidden layer expansion ratio is 4, and it is a multilayer perceptron containing two fully connected layers and one GeLU activation layer. Layer normalization is applied along the channel direction, along with dropout layers and residual connections, resulting in an output feature sequence [T]. cls ,T global ] 2.5 The dimensions are 197×768, and the multi-head attention matrices A1-A7 generated by each Transformer encoding operation are all 12×197×197 in dimension;

[0017] The specific steps of the Transformer encoding operation in step 2.5 above are as follows:

[0018] Input feature sequence [T] cls ,T global ] in The output [T] passes through a normalization layer, a multi-head self-attention module, and then a residual connection to obtain an intermediate output. cls ,T global ] mid As shown in formula (1):

[0019] (1)

[0020] in: The representation layer is a normalization layer, and MSA stands for Multi-Head Self-Attention Module.

[0021] Next, [T] cls ,T global ] midThen, after passing through a normalization layer, a multilayer perceptron containing two fully connected layers and one GeLU activation layer, and again using residual connections, [T] is obtained. cls ,T global ] out As shown in formula (2):

[0022] (2)

[0023] in: The representation layer is a normalization layer, and MLP stands for Multilayer Perceptron.

[0024] Specifically, the MSA method in formula (1) is as follows:

[0025] Let's take standard self-attention (SA) as an example: First, input the feature sequence T. in After a linear mapping, three matrices are generated for attention calculation: query Q, key K, and value V, as shown in formula (3):

[0026] (3)

[0027] in: It is a linear layer with 768×3 channels;

[0028] Then, Q is multiplied by the transpose of K and divided by the scaling factor. The softmax function is used, and the result is multiplied by the value matrix V to obtain the output feature sequence T. out As shown in formula (4):

[0029] (4)

[0030] Where: Attention represents attention calculation, and softmax is the softmax function. Indicates matrix transpose. It is a scaling factor. The size equals and vector dimension, The attention matrix generated during the computation process has a size equal to ;

[0031] Multi-head self-attention MSA divides the data into multiple heads in the channel direction with head=12, i.e., it groups the data in the channel direction. First, self-attention is calculated in the data of each head, as shown in formula (5):

[0032] (5)

[0033] Where: Attention represents the attention calculation shown in formula (4), , , This is a mapping matrix that maps Q, K, V to different heads, where i represents a different number of the head and takes a value from 1 to 12.

[0034] Then, the results from all the heads are concatenated using concat, along with an output transformation matrix W. 0 Multiplying them together yields the final output, as shown in formula (6):

[0035] (6)

[0036] Where: concat represents the concat concatenation operation, head i The results obtained by attention calculation for each head. To output the transformation matrix;

[0037] In formula (2), the MLP is defined as a multilayer perceptron, which contains two fully connected layers, one activation layer, and a dropout layer. For intermediate features [T] cls ,T global ] mid The processing procedure is shown in formula (7):

[0038] (7)

[0039] in, Indicates the dropout layer. Indicates a fully connected layer. This represents the activation layer using GeLU as the activation function;

[0040] Step 2.6: Input the obtained attention matrix A1-A7 into the LSM module to obtain the location indexes corresponding to the local high response image patches, with a dimension of 39×1;

[0041] Specifically, the steps of the LSM module are as follows:

[0042] First, the multi-head attention matrices A1-A7 are flattened along the head=12 dimension, resulting in the flattened weight matrix W. i As shown in formula (8):

[0043] (8)

[0044] Where head represents the number of heads. Represents the identity matrix. This represents the flattened weight matrix corresponding to the i-th Transformer encoding operation;

[0045] Then, the flattened W1-W7 are integrated into a weight matrix W with dimensions of 197×197 through a rollout operation, as shown in formula (9):

[0046] (9)

[0047] Where: rollout represents the rollout operation. Indicates the first The flattened weight matrix corresponding to the Transformer encoding operation;

[0048] Finally, obtain The first row and first column vectors are divided, and the first element of each vector is removed. The two vectors are then multiplied after being subjected to softmax operation. Then, the top-k function is used to obtain the position indices corresponding to a certain number of high-response image blocks, as shown in formula (10).

[0049] (10)

[0050] in, express The first row vector after removing the first element. express The first column vector after removing the first element; softmax is the softmax function. For the top-k function with K=39;

[0051] Step 2.7, based on Indices and the output sequence [T] cls ,T global ] 2.5 From T global Select the image patch corresponding to the corresponding index to obtain the local sequence T. local T cls With T local By concatenating the sequences, we obtain the local feature sequence [T]. cls ,T local ] 2.7 The dimensions are 40×768;

[0052] Step 2.8, for [T] cls ,T global ] 2.5 T in global and [T] cls ,T local ] 2.7Perform an LGFIA-Former encoding operation using an LGFIA module with 768 channels and 12 heads. The input and output channels are both 768, the hidden layer expansion rate is 4, and the activation function is GeLU. Layer normalization is applied along the channel direction, followed by dropout layers and residual connections, to obtain the output sequence [T]. cls ,T local ] 2.8 The dimensions are 40×768;

[0053] Specifically, the LGFIA module's specific method is as follows:

[0054] [T cls ,T local The local query Q required for attention computation is generated through linear mapping. l [T] cls ,T global [China T] global Similarly, the global key K required for attention calculation is generated through linear mapping. g Sum V g As shown in formula (11):

[0055] (11)

[0056] in: For a linear layer with 768×2 channels, A linear layer with 768 channels;

[0057] Using the generated Q l K g V g Computational interactive attention The formula is shown in (12):

[0058] (12)

[0059] Where: softmax is the softmax function. Indicates matrix transpose. It is a scaling factor. The size is equal to Q l and K g The vector dimension;

[0060] Furthermore, LGFIA-Former replaces the self-attention module in Transformer with the LGFIA module. The execution process of LGFIA-Former is similar to that of formulas (1) and (2), as shown in formulas (13) and (14):

[0061] (13)

[0062] (14)

[0063] in: This indicates LGFIA interactive attention calculation. Representative layer normalization layer, For a linear layer with 768×2 channels, For a linear layer with 768 channels, MLP stands for Multilayer Perceptron;

[0064] Step 2.9, for [T] cls ,T global ] 2.5 Perform a Transformer encoding operation using a self-attention layer with 768 channels and 12 heads, a multilayer perceptron with 768 input and output channels, a hidden layer expansion rate of 4, and a GeLU activation function. Perform layer normalization along the channel direction, dropout layers, and residual connections to obtain the output sequence [T]. cls ,T global ] 2.9 The dimensions are 197×768;

[0065] Step 2.10, because LGFIA requires [T] for attention calculation... cls ,T local Generate queries through linear mapping [T] cls ,T global [T before Transformer encoding operation] global Generate key K and value V; therefore, steps 2.8 and 2.9 above are executed in parallel. Similar operations are executed five times consecutively in the subsequent five-layer branch structure, resulting in two output sequences [T]. cls ,T global ] 2.10 and [T] cls ,T local ] 2.10 The dimensions are 197×768 and 40×768 respectively;

[0066] Step 2.11, extract [T] respectively cls ,T global ] 2.10 and [T] cls ,T local ] 2.10 The classification head T in cls Redefining it as T cls1 and T cls2 The two classification heads are each passed through a normalization layer, a fully connected layer, and a softmax operation to obtain two output prediction results P.global and P global&local As shown in formula (15):

[0067] (15)

[0068] in: It is a fully connected layer with an output dimension of 59. For layer normalization in the channel direction;

[0069] Step 2.12, P global and P global&local The final prediction result P is obtained by integrated voting, as shown in formula (16):

[0070] (16)

[0071] The LSFI-ViT model is now complete.

[0072] The third step is to initialize the weight parameters of the ViT model in the backbone network using the parameters of the ViT model pre-trained on the ImageNet-21K dataset.

[0073] The fourth step involves setting the relevant parameters for model training and determining the optimal loss function. The input image size is 224×224, the batch size is 32, the number of epochs is 100, the initial learning rate is 0.001, the optimizer uses the SGD algorithm with momentum of 0.9 and weight decay of 5×10⁻⁵, and the learning rate decay uses a cosine annealing strategy. Both the backbone network and the extended branches of the model use the cross-entropy loss function. As shown in formula (17):

[0074] (17)

[0075] in, y represents the predicted value of the sample, and y represents the true value of the sample.

[0076] Final loss value The sum of the losses of the two branches is shown in formula (18):

[0077] (18)

[0078] in: The predicted categories of the backbone network in the model. To expand the predicted categories of the branches, y represents the true value of the sample;

[0079] Fifth, train the LSFI-ViT model based on the settings in step 4. During this process, save the model parameters with the highest recognition accuracy on the validation set.

[0080] The sixth step is to input the image of the disease to be identified into the trained LSFI-ViT model to obtain information such as crop variety, disease type and severity under the corresponding category;

[0081] The above-mentioned crop disease identification method based on LSFI-ViT uses ViT as its backbone network, which is a well-known algorithm in this technical field.

[0082] Compared with existing methods, the beneficial effects of the present invention are:

[0083] It exhibits excellent recognition accuracy. This invention proposes a crop disease recognition method based on LSFI-ViT. Compared to other methods that only rely on global image features for disease recognition, this invention can accurately select the local areas containing subtle distinguishing features to achieve more effective feature construction, which is crucial for fine-grained recognition of crop diseases and their severity. Specifically, this invention selects local regions using LSM and utilizes LGFIA to embed subtle local features, effectively solving the problem of unsatisfactory recognition of crop diseases and their severity.

[0084] It exhibits good recognition stability. This invention is an extended branch model. The backbone network extracts global features based on the long-distance modeling advantage of the self-attention mechanism in ViT, which is beneficial for constructing the dependency relationship between the overall leaf outline and lesions at different locations. The extended branch utilizes a multi-layer LGFIA-Former to obtain local and global interactive features. While ensuring contextual semantic information, it can learn subtle feature representations in the image, aiding in the classification of disease and its severity. Thus, the representation vectors of two different features are integrated and voted on by a fully connected layer to determine the final recognition result, which can achieve error correction and make the recognition effect more stable. Attached Figure Description

[0085] Figure 1 is an overall flowchart of the crop disease identification method based on LSFI-ViT of the present invention;

[0086] Figure 2 is a diagram of the Local Selection Module (LSM) constructed in the second step of this invention;

[0087] Figure 3 is a diagram of the LGFIA module in the LGFIA-Former constructed in the second step of this invention;

[0088] Figure 4 is a diagram of the LGFIA-Former encoder constructed in the second step of this invention. Detailed Implementation

[0089] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0090] As shown in Figure 1, the overall process of the crop disease identification method based on LSFI-ViT of the present invention is as follows:

[0091] Input disease images → Perform online data preprocessing → Load pre-trained parameters of the backbone network ViT → Train and optimize LSFI-ViT → Evaluate model performance → Save optimal model parameters → Input the sample to be identified into the trained LSFI-ViT model for prediction → Obtain classification results.

[0092] As shown in Figure 2, the LSM module flow in the LSFI-ViT constructed in this invention is as follows:

[0093] Input the multi-head attention matrix A1-A7 → Average A1-A7 along the head dimension → Obtain the flattened attention weight matrix W1-W7 → Perform a rollout operation on W1-W7 → Obtain a unique weight matrix W → Extract the first row vector V1 and the first column vector V2 from W → Remove the first element of V1 and V2 → Apply the softmax function to V1 and V2 respectively → Obtain the scaled vectors → Multiply the corresponding elements of V1 and V2 respectively → Obtain a unique weight vector V → Apply the top-k function → Obtain the indices corresponding to the top K images with the highest attention weights.

[0094] As shown in Figure 3, the LGFIA module flow in the LGFIA-Former encoder constructed in this invention is as follows:

[0095] Input local feature sequence [T] with classification head cls ,T local Global feature sequence T global → [T cls ,T local ]、T global Using linear mapping, we obtain the local query matrix Q. l Global key matrix K g Value matrix V g → Divide the channel into 12 heads, i.e., 12 groups → Perform attention calculation within each head → Concatenate the output sequences of the 12 heads → Obtain the concatenated output sequence → The output sequence and the local input sequence [T] cls ,T local Perform residual connection → obtain the feature sequence [T] after interaction of local and global information. cls ,T local ] LGFIA ;

[0096] As shown in Figure 4, the LGFIA-Former encoding module constructed in this invention has the following flow:

[0097] Input a local sequence [T] with a classification header cls,T local Global sequence T global → Apply LayerNorm normalization in the channel direction → Use LGFIA to achieve local and global information exchange → Obtain the output sequence [T] cls ,T local ] LGFIA → Apply layer normalization again in the channel direction → Use a fully connected layer to expand the channel dimension to 4 times its original size → Apply the GeLU activation function → Use a fully connected layer to restore the channel dimension to its original size → Obtain the output sequence [T] cls ,T local ] MLP →[T cls ,T local ] LGFIA With [T] cls ,T local ] MLP Use residual join → to obtain the output sequence [T] cls ,T local ] LGFIA-Former .

[0098] Example 1

[0099] The specific steps of the crop disease image recognition method based on LSFI-ViT in this embodiment are as follows:

[0100] Step 1: Disease data acquisition and preprocessing:

[0101] Step 1.1, Obtaining the Crop Disease Dataset: This invention uses the open-source dataset AI Challenger 2018 for training and testing the disease identification model. This dataset involves 26 types of diseases in 10 crops. In addition to healthy samples, diseased samples have two different severity levels: general and severe. All samples are divided into 61 categories according to crop variety-disease type-severity. Samples of different categories are placed in different datasets for data labeling. The category labels are 0-60. Among them, the samples in categories 44 and 45 have only 3 images in total. Since the amount of data in these two categories is insufficient to support the model to learn the relevant features, these two categories of samples are removed. The category labels of the remaining samples are 0-58. The dataset is divided into training set and test set.

[0102] Step 1.2: In order to further enrich the dataset samples and help the model training, the training set data is preprocessed, including: resizing images, random cropping, random flipping, and standardization.

[0103] The second step is to construct the LSFI-ViT model:

[0104] Step 2.1: Input the disease images obtained in Step 1.2 into the recognition model. The dimensions of a single image are 224×224×3. Use a convolutional layer with a kernel size of 16×16, a stride of 16, and 768 output channels to achieve non-overlapping partitioning of image blocks. Each image block is represented as a token, i.e., a vector with dimensions of 1×1×768. All image blocks form the image feature map F. 2.1 The dimensions are 14×14×768;

[0105] Step 2.2, F 2.1 Flattening the image along the first two dimensions outputs the sequence feature representation of the entire image [T]. global ] 2.2 The dimensions are 196×768;

[0106] Step 2.3: Generate a trainable classification head T cls The dimension is 1×768, T cls T is responsible for continuously updating and aggregating global interaction information between different image patches in different layers of the subsequent network. cls With [T] global ] 2.2 The feature sequence [T] is obtained by splicing. cls ,T global ] 2.3 The dimensions are 197×768.

[0107] Step 2.4, generate a [T] global ] 2.3 A tensor with the same dimensions and size is initialized, and this tensor is related to [T]. cls ,T global ] 2.3 The purpose of performing the addition operation is to encode the absolute position information of each token in the feature sequence, resulting in the feature sequence [T]. cls ,T global ] 2.4 The dimensions are 197×768;

[0108] Step 2.5, for the feature sequence [T] cls ,T global ] 2.4 Seven consecutive Transformer encoding operations are performed, each using a self-attention module with 768 channels and 12 heads. The input and output channels are both 768, the hidden layer expansion ratio is 4, and it is a multilayer perceptron containing two fully connected layers and one GeLU activation layer. Layer normalization is applied along the channel direction, along with dropout layers and residual connections, resulting in an output feature sequence [T]. cls ,T global ] 2.5The dimensions are 197×768, and the multi-head attention matrices A1-A7 generated by each Transformer encoding operation are all 12×197×197 in dimension;

[0109] The specific steps of the Transformer encoding operation in step 2.5 above are as follows:

[0110] Input feature sequence [T] cls ,T global ] in The output [T] passes through a normalization layer, a multi-head self-attention module, and then a residual connection to obtain an intermediate output. cls ,T global ] mid The process is shown in formula (1):

[0111] (1)

[0112] in: The representation layer is a normalization layer, and MSA stands for Multi-Head Self-Attention Module.

[0113] Next, [T] cls ,T global ] mid Then, after passing through a normalization layer, a multilayer perceptron containing two fully connected layers and one GeLU activation layer, and again using residual connections, [T] is obtained. cls ,T global ] out As shown in formula (2):

[0114] (2)

[0115] in: Presentation layer normalization layer, This represents a multilayer perceptron;

[0116] Specifically, the MSA method in formula (1) is as follows:

[0117] Let's take standard self-attention (SA) as an example: First, input the feature sequence T. in After a linear mapping, three matrices are generated for attention calculation: query Q, key K, and value V, as shown in formula (3):

[0118] (3)

[0119] in: It is a linear layer with 768×3 channels;

[0120] Then, Q is multiplied by the transpose of K and divided by the scaling factor. The softmax function is used, and the result is multiplied by the value matrix V to obtain the output feature sequence T. out As shown in formula (4):

[0121] (4)

[0122] Where: Attention represents attention calculation, softmax is the softmax function, and matrix transpose is also present. It is a scaling factor. The size equals and vector dimension, The attention matrix generated during the computation process has a size equal to ;

[0123] Multi-head self-attention MSA divides the data into multiple heads in the channel direction with head=12, i.e., it groups the data in the channel direction. First, self-attention is calculated in the data of each head, as shown in formula (5):

[0124] (5)

[0125] Where: Attention represents the attention calculation shown in formula (4), , , This is a mapping matrix that maps Q, K, V to different heads, where i represents a different number of the head and takes a value from 1 to 12.

[0126] Then, the results from all the heads are concatenated using concat, along with an output transformation matrix W. 0 Multiplying them together yields the final output, as shown in formula (6):

[0127] (6)

[0128] Where: concat represents the concat concatenation operation, head i The results obtained by attention calculation for each head. To output the transformation matrix;

[0129] In formula (2), the MLP is defined as a multilayer perceptron, which contains two fully connected layers, one activation layer, and a dropout layer. For intermediate features [T] cls ,T global ] mid The processing procedure is shown in formula (7):

[0130] (7)

[0131] in, Indicates the dropout layer. Indicates a fully connected layer. This represents the activation layer using GeLU as the activation function;

[0132] Step 2.6: Input the obtained attention matrix A1-A7 into the LSM module to obtain the location indexes corresponding to the local high response image patches, with a dimension of 39×1;

[0133] Specifically, the steps of the LSM module are as follows:

[0134] First, the multi-head attention matrices A1-A7 are flattened along the head=12 dimension, resulting in the flattened weight matrix W. i As shown in formula (8):

[0135] (8)

[0136] Where head represents the number of heads. Represents the identity matrix. This represents the flattened weight matrix corresponding to the i-th Transformer encoding operation;

[0137] Then, the flattened W1-W7 are integrated into a weight matrix W with dimensions of 197×197 through a rollout operation, as shown in formula (9):

[0138] (9)

[0139] Where: rollout represents the rollout operation. Indicates the first The flattened weight matrix corresponding to the Transformer encoding operation;

[0140] Finally, obtain The first row and first column vectors are divided, and the first element of each vector is removed. The two vectors are then multiplied after being subjected to softmax operation. Then, the top-k function is used to obtain the position indices corresponding to a certain number of high-response image blocks, as shown in formula (10).

[0141] (10)

[0142] in, express The first row vector after removing the first element. express The first column vector after removing the first element; softmax is the softmax function. For the top-k function with K=39;

[0143] Step 2.7, based on Indices and the output sequence [T] cls ,T global ] 2.5 From T global Select the image patch corresponding to the corresponding index to obtain the local sequence T. local T cls With T local By concatenating the sequences, we obtain the local feature sequence [T]. cls ,T local ] 2.7 The dimensions are 40×768;

[0144] Step 2.8, for [T] cls ,T global ] 2.5 T in global and [T] cls ,T local ] 2.7 Perform an LGFIA-Former encoding operation using an LGFIA module with 768 channels and 12 heads. The input and output channels are both 768, the hidden layer expansion rate is 4, and the activation function is GeLU. Layer normalization is applied along the channel direction, followed by dropout layers and residual connections, to obtain the output sequence [T]. cls ,T local ] 2.8 The dimensions are 40×768;

[0145] Specifically, the LGFIA module's specific method is as follows:

[0146] [T cls ,T local The local query Q required for attention computation is generated through linear mapping. l [T] cls ,T global [China T] global Similarly, the global key K required for attention calculation is generated through linear mapping. g Sum V g As shown in formula (11):

[0147] (11)

[0148] in: For a linear layer with 768×2 channels, A linear layer with 768 channels;

[0149] Using the generated Q l K g V g Computational interactive attention The formula is shown in (12):

[0150] (12)

[0151] Where: softmax is the softmax function. Indicates matrix transpose. It is a scaling factor. The size is equal to Q l and K g The vector dimension;

[0152] Furthermore, LGFIA-Former replaces the self-attention module in Transformer with the LGFIA module. The execution process of LGFIA-Former is similar to that of formulas (1) and (2), as shown in formulas (13) and (14):

[0153] (13)

[0154] (14)

[0155] in: This indicates LGFIA interactive attention calculation. Representative layer normalization layer, For a linear layer with 768×2 channels, For a linear layer with 768 channels, MLP stands for Multilayer Perceptron;

[0156] Step 2.9, for [T] cls ,T global ] 2.5 Perform a Transformer encoding operation using a self-attention layer with 768 channels and 12 heads, a multilayer perceptron with 768 input and output channels, a hidden layer expansion rate of 4, and a GeLU activation function. Perform layer normalization along the channel direction, dropout layers, and residual connections to obtain the output sequence [T]. cls ,T global ] 2.9 The dimensions are 197×768;

[0157] Step 2.10, because LGFIA requires [T] for attention calculation... cls ,T local Generate queries through linear mapping [T] cls ,T global [T before Transformer encoding operation] globalGenerate key K and value V; therefore, steps 2.8 and 2.9 above are executed in parallel. Similar operations are executed five times consecutively in the subsequent five-layer branch structure, resulting in two output sequences [T]. cls ,T global ] 2.10 and [T] cls ,T local ] 2.10 The dimensions are 197×768 and 40×768 respectively;

[0158] Step 2.11, extract [T] respectively cls ,T global ] 2.10 and [T] cls ,T local ] 2.10 The classification head T in cls Redefining it as T cls1 and T cls2 The two classification heads are each passed through a normalization layer, a fully connected layer, and a softmax operation to obtain two output prediction results P. global and P global&local As shown in formula (15):

[0159] (15)

[0160] in: It is a fully connected layer with an output dimension of 59. For layer normalization in the channel direction;

[0161] Step 2.12, P global and P global&local The final prediction result P is obtained by integrated voting, as shown in formula (16):

[0162] (16)

[0163] The LSFI-ViT model is now complete.

[0164] The third step is to initialize the weight parameters of the ViT model in the backbone network using the parameters of the ViT model pre-trained on the ImageNet-21K dataset.

[0165] The fourth step involves setting the relevant parameters for model training and determining the optimal loss function. The input image size is 224×224, the batch size is 32, the number of epochs is 100, the initial learning rate is 0.001, the optimizer uses the SGD algorithm with momentum of 0.9 and weight decay of 5×10⁻⁵, and the learning rate decay uses a cosine annealing strategy. Both the backbone network and the extended branches of the model use the cross-entropy loss function. As shown in formula (17):

[0166] (17)

[0167] in, y represents the predicted value of the sample, and y represents the true value of the sample.

[0168] Final loss value The sum of the losses of the two branches is shown in formula (18):

[0169] (18)

[0170] in: The predicted categories of the backbone network in the model. To expand the predicted categories of the branches, y represents the true value of the sample;

[0171] Fifth, train the LSFI-ViT model based on the settings in the fourth step. During this process, save the model parameters with the highest recognition accuracy on the validation set.

[0172] The sixth step is to input the image of the disease to be identified into the trained LSFI-ViT model to obtain information such as crop variety, disease type and severity under the corresponding category;

[0173] The above-mentioned crop disease identification method based on LSFI-ViT uses ViT as its backbone network, which is a well-known algorithm in this technical field.

[0174] This embodiment's identification method targets image recognition tasks involving various plant diseases and their severity, compared to commonly used structures such as VGG, ResNet, Mobile-Net, and ViT. While maintaining a focus on global model features, this method significantly improves disease identification accuracy by extending the model structure to select discriminative local regions and learning subtle feature embeddings. Furthermore, the ViT backbone and extended branches of LSFI-ViT integrate voting to determine the final predicted category, enabling error correction; therefore, the method exhibits both good stability and generalization ability.

[0175] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for identifying crop diseases based on local selection and feature interaction, characterized in that, This method comprises two core modules: a Local Selection Module (LSM) and a Local-Global Feature Interaction Attention (LGFIA) module. LSM utilizes the attention matrix generated by the Vit self-attention module of the visual encoder to obtain the position indices corresponding to high-response image tokens, thus enabling the selection of local high-response regions. LGFIA is an improvement on the self-attention module, replacing the self-attention module in the Transformer with LGFIA to obtain the LGFIA-Former local-global feature interaction attention encoder. LGFIA-Former takes both the local token representing the high-response region and the global token of the image as input, achieving information interaction through attention calculation to embed local detailed features for disease identification. Using LSM and LGFIA-Former, this method constructs... An extension branch based on local selection and feature interaction is proposed. The backbone network of the model constructed by this method, namely the ViT model, is responsible for learning the global features of the image. The extension branch constructed by LSM and LGFIA-Former acquires the interaction features of local and global features during the learning process. The representation vectors of the two features are integrated and voted through a fully connected layer to output the final prediction result. The overall process of the method is as follows: First, the crop disease image is preprocessed. Then, the global features and local-global interaction features of the image are extracted simultaneously through the constructed model based on local selection and feature interaction for identification. Finally, the model parameters are updated and saved using the cross-entropy loss function. The process of the LGFIA module is as follows: Input local feature sequence and global feature sequence with classification head → Apply linear mapping respectively → Obtain local query matrix, global key matrix, and value matrix → The LGFIA-Former encoder works as follows: It divides the input into 12 heads (12 groups) along the channel direction; attention is calculated within each head; the output sequences of the 12 heads are concatenated to obtain the output sequence; residual connections are made between the output sequence and the input local sequence to obtain the feature sequence after interaction of local and global information. The LGFIA-Former encoder's process is as follows: Input local and global feature sequences with classification heads → normalize using LayerNorm layers along the channel direction → use LGFIA to achieve interaction of local and global information → obtain intermediate output sequences → normalize again along the channel direction → expand the channel dimension to 4 times its original size using fully connected layers → use the GeLU activation function → restore the channel dimension to its original size using fully connected layers → use residual connections with the intermediate output sequence to obtain the final output sequence of the module.

2. The crop disease identification method based on local selection and feature interaction according to claim 1, characterized in that, The LSM module's process is as follows: Input 7 multi-head attention matrices → Average the multi-head attention matrices along the head dimension → Obtain the flattened attention weight matrix → Perform a rollout operation on all weight matrices → Obtain a unique weight matrix → Extract the first row and first column vectors of the weight matrix → Remove the first element of the vector → Use the softmax function → Obtain the scaled vector → Multiply the corresponding elements of the two vectors → Obtain a unique weight vector → Use the top-k function → Obtain the indices of the top K=39 image tokens with the highest attention weights.

3. The crop disease identification method based on local selection and feature interaction according to claim 1, characterized in that, LSFI-ViT is an extension branch. The backbone network is the classic ViT network, which contains 12 Transformer encoder modules and a fully connected layer. The extension branch consists of an LSM module, 5 LGFIA-Former encoders and a fully connected layer.

4. The crop disease identification method based on local selection and feature interaction according to claim 1, characterized in that, The specific steps of this method are as follows: First step, disease data acquisition and preprocessing: Step 1.1, acquiring crop disease dataset: This invention uses the open-source dataset AI Challenger 2018 for training and testing the disease identification model. This dataset involves 26 types of diseases in 10 crops. In addition to healthy samples, diseased samples have two different degrees of disease severity: general and severe. All samples are divided into 61 categories according to crop variety-disease type-severity. Samples of different categories are placed in different datasets to achieve data labeling. The category labels are 0-60. Among them, the samples of categories 44 and 45 have only 3 images in total. Since the amount of data in these two categories is insufficient to support the model to learn relevant features, these two categories of samples are removed. The category labels of the remaining samples are 0-58. The dataset is divided into training set and test set. Step 1.2: To further enrich the dataset samples and aid model training, preprocessing operations are performed on the training set data, including: image resizing, random cropping, random flipping, and standardization. Step 2: Construct the LSFI-ViT model: Step 2.1: Input the disease images obtained in Step 1.2 into the recognition model. Each image has a dimension of 224×224×3. A convolutional layer with a kernel size of 16×16, a stride of 16, and 768 output channels is used to achieve non-overlapping partitioning of image blocks. Each image block is represented as a token, i.e., a vector, with a dimension of 1×1×768. All image blocks form the image feature map F. 2.1 The dimensions are 14×14×768; Step 2.2, F 2.1 Flattening the image along the first two dimensions outputs the sequence feature representation of the entire image [T]. global ] 2.2 The dimensions are 196×768; Step 2.3, generate a trainable classification head T. cls The dimension is 1×768, T cls T is responsible for continuously updating and aggregating global interaction information between different image patches in different layers of the subsequent network. cls With [T] global ] 2.2 The feature sequence [T] is obtained by splicing. cls ,T global ] 2.3 The dimension is 197×768. Step 2.4 generates a [T] global ] 2.3 A tensor with the same dimensions and size is initialized, and this tensor is related to [T]. cls ,T global ] 2.3 The purpose of performing the addition operation is to encode the absolute position information of each token in the feature sequence, resulting in the feature sequence [T]. cls ,T global ] 2.4 The dimension is 197×768; Step 2.5, process the feature sequence [T] cls ,T global ] 2.4 Seven consecutive Transformer encoding operations are performed, each using a self-attention module with 768 channels and 12 heads. The input and output channels are both 768, the hidden layer expansion ratio is 4, and it is a multilayer perceptron containing two fully connected layers and one GeLU activation layer. Layer normalization is applied along the channel direction, along with dropout layers and residual connections, resulting in an output feature sequence [T]. cls ,T global ] 2.5 The dimension is 197×768, and the multi-head attention matrices A1-A7 generated by each Transformer encoding operation are all 12×197×197 in dimension; the specific steps of the above 2.5 Transformer encoding operation are as follows: input feature sequence [T cls ,T global ] in The output [T] passes through a normalization layer, a multi-head self-attention module, and then a residual connection to obtain an intermediate output. cls ,T global ] mid As shown in formula (1): (1) Wherein: The representation layer is a normalization layer, where MSA represents a multi-head self-attention module; then, [T cls ,T global ] mid Then, after passing through a normalization layer, a multilayer perceptron containing two fully connected layers and one GeLU activation layer, and again using residual connections, [T] is obtained. cls ,T global ] out As shown in formula (2): (2) Wherein: The representation layer is the normalization layer, and MLP stands for Multilayer Perceptron; specifically, the specific method of MSA in formula (1) is as follows: First, take the standard self-attention SA as an example: First, input the feature sequence T in After a linear mapping, three matrices are generated for attention calculation: query Q, key K, and value V, as shown in formula (3): (3) Among them: For a linear layer with 768×3 channels; then, Q is multiplied by the transpose of K and divided by the scaling factor. The softmax function is used, and the result is multiplied by the value matrix V to obtain the output feature sequence T. out As shown in formula (4): (4) Where: Attention represents attention calculation, and softmax is the softmax function. Indicates matrix transpose. It is a scaling factor. The size equals and vector dimension, The attention matrix generated during the computation process has a size equal to Multi-head self-attention MSA divides the data into multiple heads in the channel direction with head=12, i.e., grouping in the channel direction. First, self-attention is calculated in the data of each head, as shown in formula (5): (5) Where: Attention represents the attention calculation shown in formula (4), , , To map Q, K, V to different heads, a mapping matrix is ​​used, where i represents a different head number, ranging from 1 to 12; then the results from all heads are concatenated using concat, and combined with an output transformation matrix W. 0 Multiplying them together yields the final output, as shown in formula (6): (6) Where: concat represents the concat concatenation operation, head i The results obtained by attention calculation for each head. The output transformation matrix; in formula (2), the MLP is defined as a multilayer perceptron, which contains two fully connected layers, one activation layer and a dropout layer, for the intermediate feature [T cls ,T global ] mid The processing procedure is shown in formula (7): (7) Among them, Indicates the dropout layer. Indicates a fully connected layer. This indicates the activation layer using GeLU as the activation function; in step 2.6, the obtained attention matrices A1-A7 are input into the LSM module to obtain the position indices corresponding to the local high-response image patches, with a dimension of 39×1; specifically, the steps of the LSM module are as follows: first, the multi-head attention matrices A1-A7 are flattened on the head=12 dimension to obtain the flattened weight matrix W. i As shown in formula (8): (8) Where head represents the number of heads, Represents the identity matrix. Let W1 represent the flattened weight matrix corresponding to the i-th Transformer encoding operation; then, the flattened W1-W7 are integrated into a single weight matrix W with dimensions of 197×197 through a rollout operation, as shown in formula (9): (9) Where: rollout represents the rollout operation, Indicates the first The flattened weight matrix corresponding to the Transformer encoding operation; finally, obtain... The first row and first column vectors are divided, and the first element of each vector is removed. The two vectors are then multiplied after being subjected to softmax operation. Then, the top-k function is used to obtain the position indices corresponding to a certain number of high-response image blocks, as shown in formula (10). (10) Among them, express The first row vector after removing the first element. express The first column vector after removing the first element; softmax is the softmax function. For the top-k function with K=39; Step 2.7, based on Indices and the output sequence [T cls ,T global ] 2.5 From T global Select the image patch corresponding to the corresponding index to obtain the local sequence T. local T cls With T local By concatenating the sequences, we obtain the local feature sequence [T]. cls ,T local ] 2.7 The dimension is 40×768; Step 2.8, for [T cls ,T global ] 2.5 T in global and [T] cls ,T local ] 2.7 Perform an LGFIA-Former encoding operation using an LGFIA module with 768 channels and 12 heads. The input and output channels are both 768, the hidden layer expansion rate is 4, and the activation function is GeLU. Layer normalization is applied along the channel direction, followed by dropout layers and residual connections, to obtain the output sequence [T]. cls ,T local ] 2.8 The dimension is 40×768; specifically, the LGFIA module's method is: [T cls ,T local The local query Q required for attention computation is generated through linear mapping. l [T] cls ,T global [China T] global Similarly, the global key K required for attention calculation is generated through linear mapping. g Sum V g As shown in formula (11): (11) Among them: For a linear layer with 768×2 channels, For a linear layer with 768 channels; using the generated Q... l K g V g Computational interactive attention The formula is shown in (12): (12) Where: softmax is the softmax function, Indicates matrix transpose. It is a scaling factor. The size is equal to Q l and K g The vector dimension; furthermore, LGFIA-Former replaces the self-attention module in Transformer with the LGFIA module. The execution process of LGFIA-Former is similar to that of formulas (1) and (2), as shown in formulas (13) and (14): （13） (14) Among them: This indicates LGFIA interactive attention calculation. Representative layer normalization layer, For a linear layer with 768×2 channels, For a linear layer with 768 channels, MLP represents a multilayer perceptron; in step 2.9, for [T cls ,T global ] 2.5 Perform a Transformer encoding operation using a self-attention layer with 768 channels and 12 heads, a multilayer perceptron with 768 input and output channels, a hidden layer expansion rate of 4, and a GeLU activation function. Perform layer normalization along the channel direction, dropout layers, and residual connections to obtain the output sequence [T]. cls ,T global ] 2.9 The dimension is 197×768; in step 2.10, since LGFIA requires [T] for attention calculation... cls ,T local Generate queries through linear mapping [T] cls ,T global [T before Transformer encoding operation] global Generate key K and value V; therefore, steps 2.8 and 2.9 above are executed in parallel. Similar operations are executed five times consecutively in the subsequent five-layer branch structure, resulting in two output sequences [T]. cls ,T global ] 2.10 and [T] cls ,T local ] 2.10 The dimensions are 197×768 and 40×768 respectively; in step 2.11, [T] is extracted respectively. cls ,T global ] 2.10 and [T] cls ,T local ] 2.10 The classification head T in cls Redefining it as T cls1 and T cls2 The two classification heads are each passed through a normalization layer, a fully connected layer, and a softmax operation to obtain two output prediction results P. global and P global&local As shown in formula (15): (15) Among them: It is a fully connected layer with an output dimension of 59. For the layer normalization layer in the channel direction; Step 2.12, P global and P global&local The final prediction result P is obtained by integrated voting, as shown in formula (16): (16) The LSFI-ViT model is now complete. The third step is to initialize the weight parameters of the ViT model in the backbone network using the ViT model parameters pre-trained on the ImageNet-21K dataset. The fourth step is to set the relevant parameters for model training and optimize the loss function. The fifth step is to train the LSFI-ViT model based on the relevant settings in the fourth step. During this process, the model parameters with the highest recognition accuracy on the validation set are saved. The sixth step is to input the disease images to be identified into the trained LSFI-ViT model to obtain information on crop varieties, disease types and severity under the corresponding categories.

5. The crop disease identification method based on local selection and feature interaction according to claim 4, characterized in that, During the training of the LSFI-ViT model, the input image size was 224×224, the batch size was 32, the number of epochs was 100, the initial learning rate was 0.001, the optimizer momentum was 0.9, and the weight decay was 5×10⁻⁶. -5 The SGD algorithm uses a cosine annealing strategy for learning rate decay, and both the backbone network and extended branches of the model use the cross-entropy loss function. As shown in formula (17): (17) Among them, Let y be the predicted value of the sample, and y be the true value of the sample; the final loss value. The sum of the losses of the two branches is shown in formula (18): (18) Among them: The predicted categories of the backbone network in the model. To expand the predicted categories of the branches, y represents the true value of the sample.

6. The crop disease identification method based on local selection and feature interaction according to claim 4, characterized in that, In the sixth step, the LSFI-ViT backbone network obtains the global features of the image, and the extended branches obtain the local-global interactive features of the image. The two branches use different fully connected layers to predict the disease category, and the ensemble voting yields the final prediction result.

Images