An apple leaf disease identification method based on a multi-modal feature aggregation network
By combining the multi-branch network model CT-CLIP with CNN and Transformer, local and global features of apple leaves are extracted and fused with text information across modalities. This solves the problem of disease identification accuracy in complex environments by traditional single-modal networks and achieves higher identification accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG AGRICULTURAL UNIVERSITY
- Filing Date
- 2025-04-25
- Publication Date
- 2026-07-21
AI Technical Summary
Traditional single-modality, single-branch image networks struggle to integrate the advantages of different networks and fail to deeply explore the complementary characteristics of multimodal data, resulting in limited accuracy in identifying apple leaf diseases, especially increasing the difficulty of identification in complex field environments.
The CT-CLIP multi-branch network model is adopted, which combines the convolutional neural network (CNN) and the self-attention mechanism (Transformer) to extract and fuse local and global features of the image. It also combines text information to perform cross-modal feature fusion and enhancement, and optimizes feature representation through multi-head attention mechanism and long short-term memory network to improve recognition accuracy.
By using a multimodal feature aggregation network, the accuracy of apple leaf disease identification was improved, feature information loss or degradation was prevented, the model's ability to focus on key features was enhanced, and the identification accuracy was improved.
Smart Images
Figure CN120472310B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image recognition technology, specifically to a method for identifying apple leaf diseases based on a multimodal feature aggregation network. Background Technology
[0002] Due to the complexity of orchard environments and the diversity of disease backgrounds, traditional single-modality, single-branch image networks struggle to integrate the advantages of different networks and fail to deeply exploit the complementary characteristics of multimodal data. In the process of disease feature extraction, there is a dual dilemma: coarse representation granularity and loss of global information. With the development of deep learning, single-modality plant leaf disease identification methods based on convolutional neural networks (CNNs) and image transformers (Vision Transformers) have been widely applied.
[0003] CLIP, as an advanced pre-trained multimodal model, has been widely applied in various fields due to its massive pre-training data and contrastive learning-based loss function. However, apple leaves are susceptible to various diseases during their growth, and the field environment is complex and variable, with factors such as light conditions and background interference potentially affecting recognition accuracy. Furthermore, the occlusion and overlap between apple leaves also increase the difficulty of recognition.
[0004] Therefore, given the characteristics of apple leaves and the complexity of the field environment, improving the accuracy of disease identification has become a pressing technical problem to be solved in this field. Summary of the Invention
[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:
[0006] In a first aspect, embodiments of this application provide a method for identifying apple leaf diseases based on a multimodal feature aggregation network, including:
[0007] The CT-CLIP multi-branch network model extracts local and global features of the image and fuses them to obtain the image output fused features;
[0008] Text information is input into the CT-CLIP to obtain global text features;
[0009] The image output fusion features are combined with the text global features through cross-modal feature fusion and enhancement to obtain classification fusion features;
[0010] The CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features.
[0011] In one possible implementation, the CT-CLIP includes an image processing branch, a text processing branch, and a classification head. The image output fusion features from the image processing branch and the text global features from the text processing branch are fused and then input into the classification head. The image branch includes a first encoder CLIP-CNN Encoder, a second encoder CLIP-ViT Encoder, a vector transformation module Patch Embedding, and an AFFM module. The text processing branch includes a third encoder Text Encoder and a FEB. The classification head includes fully connected layers.
[0012] In one possible implementation, the step of extracting local and global features of the image through the multi-branch network model CT-CLIP and fusing them to obtain the image output fused features includes:
[0013] The image information is input into the CLIP-CNN Encoder to obtain local features F. L ∈R B×C×H×W Where B is the batch size, C represents the number of feature channels, and H and W represent the height and width of the feature, respectively.
[0014] After inputting the image information into Patch Embedding, vector information I∈R is obtained. B×C×H×W The vector information is passed through the CLIP-ViT Encoder to output global features F. G ∈R B×N×C Where N is the number of patches;
[0015] F G The dimensions are adjusted back to B, C, H, W and F. L After concatenation and alignment, the components are used as inputs to the AFFM, and the output is a fused feature F∈R. B×N×C .
[0016] In one possible implementation, obtaining global text features by inputting text information into the CT-CLIP includes:
[0017] Each apple leaf disease category name C i Generate prompt text by inserting it into a predefined prompt template T;
[0018] The prompt text is encoded using the Text Encoder to obtain T∈R B×L×C L represents the length of the text feature;
[0019] Take the CLS marker as the global text feature F T ∈R B×1×C .
[0020] In one possible implementation, the AFFM module includes a dynamic attention module (DAM) and a fusion section;
[0021] The DAM first performs adaptive average pooling on the input features to extract global context information. Then, it performs nonlinear mapping and feature transformation through a fully connected layer and ReLU activation function. After the features are normalized by SoftMax, two sets of dynamic attention weights are dynamically generated based on the input features, and weights are assigned to different features.
[0022] The fusion part introduces the Image Long Short-Term Memory Network (V-LSTM) to construct implicit sequences in the image data, and uses it to process local features to capture long-distance dependency information in the features;
[0023] The output features of the dynamic attention module are enhanced by fusing them with the output of the fusion part through convolutional layers and ReLU activation functions to generate the final output feature X. * :
[0024] X * =f conv (att1×x v +att2×x c )
[0025] Where: x v x represents the output of the V-LSTM branch. c This represents the output of the CNN branch, and at represents the weights of different branches. These features include multi-scale information, long-distance dependency information, and the weighted results of dynamic attention.
[0026] In one possible implementation, the step of fusing and enhancing the image output fusion features with the text global features across modalities to obtain classification fusion features includes: fusing the fusion features F and the text features F T The input to the FEB is used for cross-modal feature fusion and enhancement, as shown in the following formula:
[0027] T = BERT(Tokenizer([T; C); i ]))
[0028] F L =ClipCNN(I);F G =ClipVit(I)
[0029] F = AFFM(F G ,F L )
[0030] Where ClipCNN(·) represents the Clip pre-trained ResNet50 encoder, ClipVit(·) represents the Clip pre-trained ViT encoder, and AFFM(·) represents feature fusion.
[0031] In one possible implementation, the FEB employs bidirectional multi-head attention. The FEB performs linear transformations on the input image and text features respectively to obtain query, key, and value matrices. The projection process is as follows: Where: v represents image features, l represents text features, and N represents text features. v and N l The sequence lengths of image and text features, d, are respectively. v and d l Dimensions representing image and text features respectively and Let be a learnable parameter matrix, and d be the common embedding dimension;
[0032] Subsequently, the embedded features are divided into multiple attention heads, each with a dimension of d. h =d / H, where H is the number of heads, and the converted shape is as follows: Where for image features N = N v For text features N=N l ;
[0033] In the image-to-text direction, the similarity between the image query and the text key is calculated, and scaled dot product attention is performed: In the text-to-image direction, calculate the similarity between the text query and the image key: Where A v and A l These represent the attention weight matrices across modalities;
[0034] The attention weights are used to perform a weighted summation of the Values to obtain the final attention output:
[0035] The outputs of multiple sources are concatenated, and the output features are obtained through a final linear transformation. in: The projection matrix has the following final output shapes: and
[0036] The result and Global average pooling is performed to obtain and
[0037] Finally, weighted fusion is performed to obtain the classification fusion features. Among them: W v and W l These are learnable weights.
[0038] In one possible implementation, the CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features, including: F fusion Entering the classification head, the fully connected layer FC in the classification head performs classification: y = softmax(FC2(ReLU(FC1(F fusion )))),in For the projection of the hidden layer, This is the final classification layer.
[0039] In one possible implementation, CT-CLIP is trained using a public dataset, and the trained model is evaluated, adjusted, and optimized. The optimal weight model is retained and validated. In the image processing branch, a small number of parameters are fine-tuned by training the Adapter layer to enhance the model's fitting ability, as shown in the following formula:
[0040]
[0041] Where N represents the total number of training samples; when i equals the true class label At that time, y i =1, otherwise y i =0; This represents the predicted probability of category i. For all learnable parameters.
[0042] In one possible implementation, the evaluation metrics used to assess the CT-CLIP include Precision, Recall, F1-Score, and Accuracy. The total number of true positives (TP), true negatives (TN), false negatives (FN), and false positives (FP) is calculated globally, and Precision, Recall, F1-Score, and Accuracy are calculated according to the following formulas:
[0043]
[0044] In this embodiment, features are extracted from the image from two different directions to obtain local and global features, which are then fused. The global text features are then extracted to provide semantic guidance to the model, thereby preventing the loss or degradation of feature information during transmission and interaction. An adaptive feature fusion module is progressively designed to dynamically weight features from different image branch networks, thereby improving recognition accuracy. Attached Figure Description
[0045] Figure 1 A flowchart illustrating an apple leaf disease identification method based on a multimodal feature aggregation network, provided for an embodiment of this application;
[0046] Figure 2 This is a schematic diagram of the structure of CT-CLIP provided in the embodiments of this application;
[0047] Figure 3 This is a schematic diagram of the AFFM module provided in an embodiment of this application;
[0048] Figure 4 This is a schematic diagram of the structure of the FEB provided in an embodiment of this application. Detailed Implementation
[0049] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.
[0050] See Figure 1 The apple leaf disease identification method based on a multimodal feature aggregation network provided in this embodiment includes:
[0051] S101 extracts local and global features of the image through the multi-branch network model CT-CLIP and fuses them to obtain the image output fused features.
[0052] The multi-branch network model CT-CLIP proposed in this embodiment combines the advantages of convolutional neural networks (CNNs) and the self-attention mechanism (Transformer) to enhance representation learning capabilities. For example... Figure 2 As shown, the CT-CLIP includes an image processing branch, a text processing branch, and a classification head. The image output fusion features from the image processing branch and the text global features from the text processing branch are fused and then input into the classification head. The image branch includes a first encoder CLIP-CNNEncoder, a second encoder CLIP-ViT Encoder, a vector transformation module Patch Embedding, and an adaptive feature fusion module AFFM. The text processing branch includes a third encoder Text Encoder and a feature enhancement module FEB. The classification head includes fully connected layers.
[0053] This embodiment uses a public dataset to train CT-CLIP, and evaluates, adjusts, and optimizes the trained model, retaining the optimal weight model and validating it. A large amount of apple leaf image data from different regions, backgrounds, and growth stages is collected. The collected data is preprocessed, expanded using online and offline data augmentation methods, and classified and labeled to create a dataset, which is further divided into training, validation, and test sets. The images are divided into training, validation, and test sets according to a 7:2:1 ratio.
[0054] The image processing branch enhances the model's fitting ability by fine-tuning a small number of parameters through training the Adapter layer. The Adapter module is integrated twice in each Transformer layer: once after the multi-head attention mechanism and again after the normalization layer. The Adapter module employs a bottleneck structure, significantly reducing the number of parameters compared to the attention and feedforward layers in the original model. The specific calculation formula is as follows:
[0055]
[0056] Where N represents the total number of training samples; when i equals the true class label At that time, y i =1, otherwise y i =0; This represents the predicted probability of category i. For all learnable parameters.
[0057] The evaluation metrics used to assess the CT-CLIP include Precision, Recall, F1-Score, and Accuracy. The total number of true positives (TP), true negatives (TN), false negatives (FN), and false positives (FP) is calculated globally, and Precision, Recall, F1-Score, and Accuracy are calculated using the following formulas:
[0058]
[0059] In this embodiment, to obtain the output fusion features of the image, image information is input into the CLIP-CNNEncoder to obtain local features F. L ∈R B×C×H×W After inputting the image information into Patch Embedding, vector information I∈R is obtained. B×C×H×W The vector information is passed through the CLIP-ViT Encoder to output global features F. G ∈R B×N×C Where N is the number of patches. Let F... GThe dimensions are adjusted back to B, C, H, W and F. L After concatenation and alignment, the components are used as inputs to the AFFM, and the output is a fused feature F∈R. B×N×C .
[0060] In this embodiment, the AFFM module includes a Dynamic Attention Module (DAM) and a fusion component. For example... Figure 3 As shown, the DAM part first performs adaptive average pooling on the input feature X to extract global context information. Then, it performs non-linear mapping and feature transformation through a fully connected layer and the ReLU activation function. After the features are normalized by SoftMax, two sets of dynamic attention weights (att) are dynamically generated based on the input features. 1 and att 2 The model assigns weights to different features, thereby enhancing its focus on important features. These dynamic weights are applied to the feature representation through dot product operations, effectively optimizing the importance of features and enabling the model to focus on key regions or salient features.
[0061] The fusion section introduces a Visual Long Short-Term Memory (V-LSTM) network. V-LSTM can construct implicit sequences in image data, and using it to process local features can capture long-distance dependencies within those features, thereby obtaining a portion of the global features. By incorporating V-LSTM, the module can more comprehensively model the dynamic relationships between features. Finally, the output features of the dynamic attention module are fused and further enhanced through convolutional layers and the ReLU activation function to generate the final output feature X. * :
[0062] X * =f conv (att1×x v +att2×x c )
[0063] Where, x v x represents the output of the V-LSTM branch. c This represents the output of the CNN branch, and at represents the weights of different branches. These features include multi-scale information, long-distance dependency information, and the weighted results of dynamic attention.
[0064] S102, Input text information into the CT-CLIP to obtain global text features.
[0065] In the text processing branch, each apple leaf disease category name C i The prompt text is inserted into a predefined prompt template T to generate prompt text; the prompt text is then encoded using the Text Encoder to obtain T∈R. B×L×CB represents the batch size; the CLS marker is taken as the global text feature F. T ∈R B×1×C .
[0066] S103, perform cross-modal feature fusion and enhancement of the image output fusion features and the text global features to obtain classification fusion features.
[0067] In this embodiment, the fused feature F and the text feature F are combined. T The input to the FEB is used for cross-modal feature fusion and enhancement, as shown in the following formula:
[0068] T = BERT(Tokenizer([T; C); i ]))
[0069] F L =ClipCNN(I);F G =ClipVit(I)
[0070] F = AFFM(F G ,F L )
[0071] Where ClipCNN(·) represents the Clip pre-trained ResNet50 encoder, ClipVit(·) represents the Clip pre-trained ViT encoder, and AFFM(·) represents feature fusion.
[0072] like Figure 4 As shown, the FEB employs bidirectional multi-head attention. The FEB performs linear transformations on the input image and text features respectively to obtain query, key, and value matrices. The projection process is as follows: Where: v represents image features, l represents text features, and N represents text features. v and N l The sequence lengths of image and text features, d, are respectively. v and d l Dimensions representing image and text features respectively and Let be the learnable parameter matrix, and d be the common embedding dimension.
[0073] Subsequently, the embedded features are divided into multiple attention heads, each with a dimension of d. h =d / H, where H is the number of heads, and the converted shape is as follows: Where for image features N = N v For text features N=N l .
[0074] In the image-to-text direction, the similarity between the image query and the text key is calculated, and scaled dot product attention is performed: In the text-to-image direction, calculate the similarity between the text query and the image key: Where A v and A l These represent the attention weight matrices across modalities.
[0075] The attention weights are used to perform a weighted summation of the Values to obtain the final attention output: The outputs of multiple sources are concatenated, and the output features are obtained through a final linear transformation. in: The projection matrix has the following final output shapes: and
[0076] The result and Global average pooling is performed to obtain and Finally, weighted fusion is performed to obtain the classification fusion features. Among them: W v and W l These are learnable weights.
[0077] S104, the CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features.
[0078] F fusion Entering the classification head, the fully connected layer FC in the classification head performs classification: t = softmax(FC2(ReLU(FC1(F fusion )))),in For the projection of the hidden layer, This is the final classification layer.
[0079] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0080] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. A method for identifying apple leaf diseases based on a multimodal feature aggregation network, characterized in that, include: The CT-CLIP multi-branch network model extracts local and global features of the image and fuses them to obtain the image output fused features; Text information is input into the CT-CLIP to obtain global text features; The image output fusion features are combined with the text global features through cross-modal feature fusion and enhancement to obtain classification fusion features; The CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features. The CT-CLIP includes an image processing branch, a text processing branch, and a classification head. The image output fusion features from the image processing branch and the text global features from the text processing branch are fused and then input into the classification head. The image processing branch includes a first encoder CLIP-CNN Encoder, a second encoder CLIP-ViT Encoder, a vector transformation module Patch Embedding, and an adaptive feature fusion module AFFM. The text processing branch includes a third encoder Text Encoder and a feature enhancement module FEB. The classification head includes fully connected layers. The AFFM includes a Dynamic Attention Module (DAM) and a fusion component. The DAM first performs adaptive average pooling on the input features to extract global context information. Then, it performs nonlinear mapping and feature transformation through a fully connected layer and ReLU activation function. After the features are normalized by SoftMax, two sets of dynamic attention weights are dynamically generated based on the input features, and weights are assigned to different features. The fusion part introduces the Image Long Short-Term Memory Network (V-LSTM) to construct implicit sequences in the image data, and uses it to process local features to capture long-distance dependency information in the features; The output features of the dynamic attention module are enhanced by fusing them with the output of the fusion part through convolutional layers and ReLU activation functions to generate the final output features. : in: This represents the convolution block operation. This represents the output of the V-LSTM branch. This represents the output of the CNN branch. and These represent the weights of different branches.
2. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 1, characterized in that, The process of extracting local and global features of an image using the multi-branch network model CT-CLIP and fusing them to obtain the image output fused features includes: Image information is input into the CLIP-CNN Encoder to obtain local features. ; The image information is input into Patch Embedding to obtain vector information. The vector information is then processed by the CLIP-ViT Encoder to output global features. ; Will Dimensional adjustment and The concatenated features are used as inputs to the AFFM and output as fused features. .
3. The apple leaf disease identification method based on a multimodal feature aggregation network according to claim 2, characterized in that, The step of inputting text information into the CT-CLIP to obtain global text features includes: Name of each type of apple leaf disease Insert into a predefined prompt template Generate prompt text; The prompt text is encoded using the Text Encoder. ; Take the CLS tag as a global text feature .
4. The apple leaf disease identification method based on a multimodal feature aggregation network according to claim 3, characterized in that, The step of fusing and enhancing the image output fusion features with the text global features across modalities to obtain classification fusion features includes: fusing the fusion features... and text global features The data is fed into the FEB for cross-modal feature fusion and enhancement.
5. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 4, characterized in that, The FEB employs bidirectional multi-head attention, and performs linear transformations on the input image and text features to obtain query, key, and value matrices. Subsequently, the embedded features are divided into multiple attention heads, each with a dimension of [missing information]. ,in For the number of heads, For public embedded dimensions; In the image-to-text direction, the similarity between the image query and the text key is calculated, and scaling dot product attention is performed. The attention weights are used to perform a weighted summation of Values to obtain the final attention output; The outputs of multiple sources are concatenated, and the output features are obtained through a final linear transformation. , ; The result and Global average pooling is performed to obtain and ; Finally, weighted fusion is performed to obtain the classification fusion features. .
6. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 5, characterized in that, The CT-CLIP classifies and identifies apple leaf diseases based on the classification fusion features, including: The system then proceeds to the classification head, where the fully connected layer (FC) performs the classification.
7. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 1, characterized in that, CT-CLIP was trained using a public dataset, and the trained model was evaluated, adjusted, and optimized. The optimal weight model was retained and validated. In the image processing branch, a small number of parameters were fine-tuned by training the Adapter layer to enhance the model's fitting ability.
8. The method for identifying apple leaf diseases based on a multimodal feature aggregation network according to claim 7, characterized in that, The evaluation metrics used to assess the CT-CLIP include Precision, Recall, F1-Score, and Accuracy. The total number of true positives (TP), true negatives (TN), false negatives (FN), and false positives (FP) is calculated globally, and Precision, Recall, F1-Score, and Accuracy are calculated using the following formulas: ； ； ； 。