Apple leaf disease identification method based on multi-modal feature aggregation network

Abstract

The invention discloses an apple leaf disease identification method based on a multi-modal feature aggregation network. The method comprises the following steps: extracting local features and global features of an image through a multi-branch network model CT-CLIP, and fusing the local features and the global features to obtain image output fusion features; text information is input into the CT-CLIP to obtain text global features; performing cross-modal feature fusion and enhancement on the image output fusion features and the text global features to obtain classification fusion features; and the CT-CLIP classifies and identifies the apple leaf diseases according to the classification fusion features. According to the method, feature extraction is carried out on an image from two different directions to obtain local features and global features, the local features and the global features are fused, then text global features of a text are extracted to carry out semantic guidance on a model, so that feature information is prevented from being lost or degraded in the transmission and interaction process, and a self-adaptive feature fusion module is progressively designed; features from different image branch networks can be dynamically weighted, so that the recognition precision is improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for identifying apple leaf diseases based on a multimodal feature aggregation network. Background Art

[0002] Due to the complexity of orchard environments and the diversity of disease backgrounds, traditional single-modal, single-branch image networks struggle to integrate the strengths of different networks and exploit the complementary properties of multimodal data. Disease feature extraction faces the dual dilemma of coarse representation granularity and global information loss. With the advancement of deep learning, single-modal plant leaf disease recognition methods based on convolutional neural networks (CNNs) and image transformers (Vision Transformers) have gained widespread application.

[0003] CLIP, an advanced pre-trained multimodal model, has been widely used in various fields thanks to its extensive pre-training data and contrastive learning-based loss function. However, apple leaves are susceptible to various diseases during their growth, and the field environment is complex and variable. Factors such as lighting conditions and background interference can affect recognition accuracy. Furthermore, occlusion and overlap between apple leaves also complicate recognition.

[0004] Therefore, considering the characteristics of apple leaves and the complexity of the field environment, how to improve the accuracy of disease identification has become a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an apple leaf disease identification method based on a multimodal feature aggregation network, comprising:

[0007] The multi-branch network model CT-CLIP is used to extract local features and global features of the image and fuse them to obtain the image output fusion features;

[0008] Inputting text information into the CT-CLIP to obtain global features of the text;

[0009] Performing cross-modal feature fusion and enhancement on the image output fusion feature and the text global feature to obtain a classification fusion feature;

[0010] The CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features.

[0011] In one possible implementation, the CT-CLIP includes an image processing branch, a text processing branch, and a classification head. The image output fusion features output by the image processing branch and the text global features output by the text processing branch are fused and input into the classification head. The image branch includes a first encoder CLIP-CNN Encoder, a second encoder CLIP-ViT Encoder, a vector conversion module Patch Embedding, and an AFFM module. The text processing branch includes a third encoder Text Encoder and FEB. The classification head includes a fully connected layer.

[0012] In a possible implementation, extracting local features and global features of an image using a multi-branch network model CT-CLIP and fusing them to obtain image output fusion features includes:

[0013] Input the image information into the CLIP-CNN Encoder to obtain the local feature F L ∈R B×C×H×W , where B represents the batch size, C represents the number of feature channels, H and W represent the height and width of the feature respectively;

[0014] After the image information is input into Patch Embedding, vector information I∈R is obtained. B×C×H×W The vector information passes through the CLIP-ViT Encoder and outputs the global feature F G ∈R B×N×C , where N is the number of patches;

[0015] F G Adjust the dimensions back to B, C, H, W and F L After splicing and alignment, they are jointly used as the input of the AFFM and the output fusion feature F∈R B×N×C .

[0016] In a possible implementation, inputting text information into the CT-CLIP to obtain global text features includes:

[0017] Each apple leaf disease category name C i Insert into the predefined prompt template T to generate prompt text;

[0018] The prompt text is encoded by the Text Encoder to obtain T∈R B×L×C , L represents the length of the text feature;

[0019] Take the CLS tag as the text global feature F T ∈R B×1×C .

[0020] In one possible implementation, the AFFM module includes a dynamic attention module DAM and a fusion part;

[0021] The DAM first performs adaptive average pooling on the input features to extract global context information, then performs nonlinear mapping and feature transformation through a fully connected layer and ReLU activation function. After the features are normalized by SoftMax, two sets of dynamic attention weights are dynamically generated based on the input features, and weights are assigned to different features.

[0022] The fusion part introduces the image long short-term memory network V-LSTM to construct implicit sequences in image data, and uses it to process local features to capture long-distance dependency information in features;

[0023] The output features of the dynamic attention module are fused and enhanced with the fusion part output through the convolution layer and ReLU activation function to generate the final output feature X * :

[0024] X * =f conv (att1×x v +att2×x c )

[0025] Where: x v represents the output of the V-LSTM branch, x c Represents the output of the CNN branch, and att is the weight of different branches. These features contain the weighted results of multi-scale information, long-distance dependency information, and dynamic attention.

[0026] In a possible implementation, the cross-modal feature fusion and enhancement of the image output fusion feature and the text global feature to obtain the classification fusion feature includes: combining the fusion feature F and the text feature F T The FEB is passed in for cross-modal feature fusion and enhancement. The formula is as follows:

[0027] T = BERT(Tokenizer([T; C i ]))

[0028] F L =ClipCNN(I);F G =ClipVit(I)

[0029] F=AFFM(F G ,F L )

[0030] Among them: ClipCNN(·) represents the ResNet50 encoder pre-trained by Clip, ClipVit(·) represents the ViT encoder pre-trained by Clip, and AFFM(·) represents feature fusion.

[0031] In one possible implementation, the FEB uses bidirectional multi-head attention. The FEB performs linear transformations on the input image and text features respectively to obtain query, key, and value matrices. The projection process is as follows: Among them: v is the image feature, l is the text feature, N v and N l are the sequence lengths of image and text features, respectively, d v and d l Dimensions representing image and text features respectively and is the learnable parameter matrix, d is the common embedding dimension;

[0032] Then, the embedded features are divided into multiple attention heads, each with a dimension of d h =d / H, where H is the number of heads. The shape after conversion is as follows: Where N = N for image features v , for text features N=N l ;

[0033] In the image-to-text direction, the similarity between the image query and the text key is calculated, and a scaled dot product attention calculation is performed: In the text-to-image direction, calculate the similarity between the text query and the image key: Among them A v and A l Represent the cross-modal attention weight matrix respectively;

[0034] Use the attention weight to perform weighted summation on the Value to get the final attention output:

[0035] The outputs of multiple heads are spliced together and the output features are obtained through the final linear transformation in: is the projection matrix, and the final output shapes are and

[0036] Will get and Perform global average pooling to obtain and

[0037] Finally, weighted fusion is performed to obtain the classification fusion feature Where: W v and W l are learnable weights.

[0038] In a possible implementation, the CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features, including: fusion Entering the classification head, the fully connected layer FC in the classification head performs classification: y = softmax(FC2(ReLU(FC1(F fusion )))),in is the hidden layer projection, is the final classification layer.

[0039] In one possible implementation, CT-CLIP is trained using a public dataset, and the trained model is evaluated, adjusted, and optimized. The optimal weighted model is retained and verified. In the image processing branch, a small number of parameters are fine-tuned by training the Adapter layer to enhance the model's fitting ability. The formula is as follows:

[0040]

[0041] Where N represents the total number of training samples; when i is equal to the true category label When y i =1, otherwise y i =0; represents the predicted probability of category i, are all learnable parameters.

[0042] In one possible implementation, the evaluation metrics used to evaluate the CT-CLIP include Precision, Recall, F1-Score, and Accuracy. The total number of true positives (TP), true negatives (TN), false negatives (FN), and false positives (FP) is calculated globally, and Precision, Recall, F1-Score, and Accuracy are calculated according to the following formulas:

[0043]

[0044] In an embodiment of the present application, feature extraction is performed on the image from two different directions to obtain local features and global features, which are then fused. Then, the global features of the text are extracted to provide semantic guidance for the model, thereby preventing the loss or degradation of feature information during transmission and interaction. An adaptive feature fusion module is progressively designed, which can dynamically weight features from different image branch networks to improve recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of a method for identifying apple leaf diseases based on a multimodal feature aggregation network provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of CT-CLIP provided in the examples of this application;

[0047] Figure 3 A schematic diagram of the structure of the AFFM module provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of the FEB provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The present invention will be described below with reference to the accompanying drawings and specific implementation methods.

[0050] See also Figure 1 The apple leaf disease identification method based on the multimodal feature aggregation network provided in this embodiment includes:

[0051] S101, extracting local features and global features of the image through the multi-branch network model CT-CLIP and fusing them to obtain image output fusion features.

[0052] The multi-branch network model CT-CLIP proposed in this embodiment combines the advantages of convolutional neural network CNN and self-attention mechanism Transformer to enhance representation learning capabilities. Figure 2 As shown, the CT-CLIP includes an image processing branch, a text processing branch and a classification head. The image output fusion features output by the image processing branch and the text global features output by the text processing branch are fused and input into the classification head; the image branch includes a first encoder CLIP-CNNEncoder, a second encoder CLIP-ViT Encoder, a vector conversion module Patch Embedding and an adaptive feature fusion module AFFM; the text processing branch includes a third encoder Text Encoder and a feature enhancement module FEB; the classification head includes a fully connected layer.

[0053] This example uses a public dataset to train CT-CLIP. The trained model is evaluated, adjusted, and optimized, retaining the optimal weighted model after training and performing validation. A large amount of apple leaf image data from different regions, backgrounds, and growth periods is collected. The collected data is preprocessed, augmented through online and offline data augmentation, and classified and annotated to create a dataset. This dataset is further divided into training, validation, and test sets. The images are divided into a 7:2:1 ratio to create the training, validation, and test sets, respectively.

[0054] The image processing branch fine-tunes a small number of parameters by training the Adapter layer, enhancing the model's fitting capabilities. The Adapter module is integrated twice in each Transformer layer: once after the multi-head attention mechanism and again after the normalization layer. The Adapter module uses a bottleneck structure, significantly reducing the number of parameters compared to the attention and feedforward layers in the original model. The specific calculation formula is as follows:

[0055]

[0056] Where N represents the total number of training samples; when i is equal to the true category label When y i =1, otherwise y i =0; represents the predicted probability of category i, are all learnable parameters.

[0057] The evaluation metrics used to evaluate the CT-CLIP include Precision, Recall, F1-Score, and Accuracy. The total number of true positive TPs, true negative TNs, false negative FNs, and false positive FPs is calculated globally. Precision, Recall, F1-Score, and Accuracy are calculated according to the following formulas:

[0058]

[0059] In order to obtain the output fusion features of the image, this embodiment inputs the image information into the CLIP-CNNEncoder to obtain the local features F L ∈R B×C×H×W After inputting the image information into Patch Embedding, vector information I∈R is obtained B×C×H×W The vector information passes through the CLIP-ViT Encoder and outputs the global feature F G ∈R B×N×C , where N is the number of patches. GAdjust the dimensions back to B, C, H, W and F L After splicing and alignment, they are jointly used as the input of the AFFM and the output fusion feature F∈R B×N×C .

[0060] In this embodiment, the AFFM module includes a dynamic attention module DAM and a fusion part. Figure 3 As shown in Figure 1, the DAM part first performs adaptive average pooling on the input feature X to extract global context information. Subsequently, it performs nonlinear mapping and feature transformation through the fully connected layer and ReLU activation function. After the features are normalized by SoftMax, two sets of dynamic attention weights (att 1 and att 2 ) and assign weights to different features, thereby enhancing the model's focus on important features. These dynamic weights are applied to the feature representation through a dot product operation, effectively optimizing the importance of features and enabling the model to focus on key areas or significant features.

[0061] The fusion part introduces the Vision-Long Short Term Memory (V-LSTM) network. V-LSTM can construct implicit sequences in image data. Using it to process local features can capture long-range dependency information in features, thereby obtaining a portion of global features. By adding V-LSTM, the module can more comprehensively model the dynamic associations between features. Finally, the output features of the dynamic attention module are fused and further enhanced through the convolutional layer and ReLU activation function to generate the final output feature X * :

[0062] X * =f conv (att1×x v +att2×x c )

[0063] Among them, x v represents the output of the V-LSTM branch, x c Represents the output of the CNN branch, and att is the weight of different branches. These features contain the weighted results of multi-scale information, long-distance dependency information, and dynamic attention.

[0064] S102: Input text information into the CT-CLIP to obtain global text features.

[0065] In the text processing branch, each apple leaf disease category name C i Insert it into the predefined prompt template T to generate prompt text; encode the prompt text through the Text Encoder to obtain T∈R B×L×C, B is the batch size; take the CLS tag as the global feature F of the text T ∈R B×1×C .

[0066] S103: Perform cross-modal feature fusion and enhancement on the image output fusion feature and the text global feature to obtain a classification fusion feature.

[0067] In this embodiment, the fusion feature F and the text feature F T The FEB is passed in for cross-modal feature fusion and enhancement. The formula is as follows:

[0068] T = BERT(Tokenizer([T; C i ]))

[0069] F L =ClipCNN(I);F G =ClipVit(I)

[0070] F=AFFM(F G ,F L )

[0071] Among them: ClipCNN(·) represents the ResNet50 encoder pre-trained by Clip, ClipVit(·) represents the ViT encoder pre-trained by Clip, and AFFM(·) represents feature fusion.

[0072] like Figure 4 As shown, the FEB adopts bidirectional multi-head attention. The FEB performs linear transformation on the input image and text features respectively to obtain query, key and value matrices. The projection process is as follows: Among them: v is the image feature, l is the text feature, N v and N l are the sequence lengths of image and text features, respectively, d v and d l Dimensions representing image and text features respectively and is the learnable parameter matrix and d is the common embedding dimension.

[0073] Then, the embedded features are divided into multiple attention heads, each with a dimension of d h =d / H, where H is the number of heads. The shape after conversion is as follows: Where N = N for image features v , for text features N=N l .

[0074] In the image-to-text direction, the similarity between the image query and the text key is calculated, and a scaled dot product attention calculation is performed: In the text-to-image direction, calculate the similarity between the text query and the image key: Among them A v and A l Represent the cross-modal attention weight matrices respectively.

[0075] Use the attention weight to perform weighted summation on the Value to get the final attention output: The outputs of multiple heads are spliced together and the output features are obtained through the final linear transformation in: is the projection matrix, and the final output shapes are and

[0076] Will get and Perform global average pooling to obtain and Finally, weighted fusion is performed to obtain the classification fusion feature Where: W v and W l are learnable weights.

[0077] S104, the CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features.

[0078] F fusion Entering the classification head, the fully connected layer FC in the classification head performs classification: t = softmax(FC2(ReLU(FC1(F fusion )))),in is the hidden layer projection, is the final classification layer.

[0079] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0080] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A method for identifying apple leaf diseases based on a multimodal feature aggregation network, characterized in that: include: The multi-branch network model CT-CLIP is used to extract local features and global features of the image and fuse them to obtain the image output fusion features; Inputting text information into the CT-CLIP to obtain global features of the text; Performing cross-modal feature fusion and enhancement on the image output fusion feature and the text global feature to obtain a classification fusion feature; The CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features.

2. The apple leaf disease identification method based on multimodal feature aggregation network according to claim 1 is characterized in that: The CT-CLIP includes an image processing branch, a text processing branch, and a classification head. The image output fusion features output by the image processing branch and the text global features output by the text processing branch are fused and input into the classification head. The image branch includes a first encoder CLIP-CNN Encoder, a second encoder CLIP-ViT Encoder, a vector transformation module Patch Embedding, and an adaptive feature fusion module AFFM. The text processing branch includes a third encoder Text Encoder and a feature enhancement module FEB. The classification head includes a fully connected layer.

3. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 2, characterized in that: The method of extracting local features and global features of an image through a multi-branch network model CT-CLIP and fusing them to obtain image output fusion features includes: Input the image information into the CLIP-CNN Encoder to obtain the local feature F L ∈R B×C×H×W , where B represents the batch size, C represents the number of feature channels, H and W represent the height and width of the feature respectively; After the image information is input into Patch Embedding, vector information I∈R is obtained. B×C×H×W The vector information passes through the CLIP-ViT Encoder and outputs the global feature F G ∈R B×N×C , where N is the number of patches; F G Adjust the dimensions back to B, C, H, W and F L After splicing and alignment, they are jointly used as the input of the AFFM and the output fusion feature F∈R B×N×C .

4. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 3, characterized in that: Inputting text information into the CT-CLIP to obtain global text features includes: Each apple leaf disease category name C i Insert into the predefined prompt template T to generate prompt text; The prompt text is encoded by the Text Encoder to obtain T∈R B×L×C , where L represents the length of the text feature; Take the CLS tag as the text global feature F T ∈R B×1×C .

5. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to any one of claims 2 to 4, characterized in that: The AFFM module includes a dynamic attention module DAM and a fusion part. The DAM first performs adaptive average pooling on the input features to extract global context information, then performs nonlinear mapping and feature transformation through a fully connected layer and ReLU activation function. After the features are normalized by SoftMax, two sets of dynamic attention weights are dynamically generated based on the input features, and weights are assigned to different features. The fusion part introduces the image long short-term memory network V-LSTM to construct implicit sequences in image data, and uses it to process local features to capture long-distance dependency information in features; The output features of the dynamic attention module are fused and enhanced with the fusion part output through the convolution layer and ReLU activation function to generate the final output feature X * : X * =f conv (att1×x v +att2×x c ) Where: f conv Represents the convolution block operation, x v represents the output of the V-LSTM branch, x c Represents the output of the CNN branch, and att is the weight of different branches. These features contain the weighted results of multi-scale information, long-distance dependency information, and dynamic attention.

6. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 5, characterized in that: The cross-modal feature fusion and enhancement of the image output fusion feature and the text global feature to obtain the classification fusion feature includes: combining the fusion feature F and the text feature F T The FEB is passed in for cross-modal feature fusion and enhancement. The formula is as follows: T=BERT(Tokenizer([T;C i ])) F L =ClipCNN(I);F G =ClipVit(I) F=AFFM(F G ,F L ) Among them: ClipCNN(·) represents the ResNet50 encoder pre-trained by Clip, ClipVit(·) represents the ViT encoder pre-trained by Clip, and AFFM(·) represents feature fusion.

7. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 6, characterized in that: The FEB adopts bidirectional multi-head attention. The FEB performs linear transformation on the input image and text features respectively to obtain query, key and value matrices. The projection process is as follows: Among them: v is the image feature, l is the text feature, N v and N l are the sequence lengths of image and text features, respectively, d v and d l Dimensions representing image and text features respectively and is the learnable parameter matrix, d is the common embedding dimension; Then, the embedded features are divided into multiple attention heads, each with a dimension of d h =d / H, where H is the number of heads. The shape after conversion is as follows: Where N = N for image features v , for text features N=N l ; In the image-to-text direction, the similarity between the image query and the text key is calculated, and a scaled dot product attention calculation is performed: In the text-to-image direction, calculate the similarity between the text query and the image key: Among them A v and A l Represent the cross-modal attention weight matrix respectively; Use the attention weight to perform weighted summation on the Value to get the final attention output: v =A v V l , O l =A l V v , The outputs of multiple heads are spliced together and the output features are obtained through the final linear transformation in: is the projection matrix, and the final output shapes are and Will get and Perform global average pooling to obtain and Finally, weighted fusion is performed to obtain the classification fusion feature Where: W v and W l are learnable weights.

8. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 7, characterized in that: The CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features, including: fusion Entering the classification head, the fully connected layer FC in the classification head performs classification: y = softmax(FC2(ReLU(FC1(F fusion )))),in is the hidden layer projection, is the final classification layer.

9. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 1, characterized in that: CT-CLIP is trained using a public dataset, and the trained model is evaluated, adjusted, and optimized. The optimal weighted model is retained and verified. In the image processing branch, a small number of parameters are fine-tuned by training the Adapter layer to enhance the model's fitting ability. The formula is as follows: Where: N represents the total number of training samples; when i is equal to the true category label When y i =1, otherwise y i =0; represents the predicted probability of category i, For all learnable parameters, A I (.) indicates a learnable adapter, Represents the global features obtained, represents the global feature after residual connection, and α represents the residual coefficient.

10. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 9, characterized in that: The evaluation metrics used to evaluate the CT-CLIP include Precision, Recall, F1-Score, and Accuracy. The total number of true positive TPs, true negative TNs, false negative FNs, and false positive FPs is calculated globally. Precision, Recall, F1-Score, and Accuracy are calculated according to the following formulas:

Images