A multi-modal small sample plant disease identification method

Abstract

The application discloses a kind of multi-modal small sample plant disease identification methods, the method is: through collection and acquisition plant disease image, data pretreatment is carried out to picture;Propose VSF model, including semantic upgrade module, utilize large language model and prompt word to plant disease definition upgrade generation more real and conform to agricultural scene disease description;Design two-stage modal fusion module, effectively improve the consistency and discriminant ability of cross-modal information;The image data after pretreatment is input into model and is trained and verified, obtains optimal model;Finally, the image to be classified is input into optimal model to obtain classification result.The multi-modal small sample plant disease identification method proposed in the application can ensure rapid identification of plant diseases and obtain the correct type, and has good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of natural image classification and relates to an efficient identification method for classifying plant diseases. Background Technology

[0002] Plant diseases are a significant threat to agricultural production and food security, and their early identification and accurate diagnosis are crucial for ensuring crop yield and quality. Existing plant disease identification methods primarily rely on the manual experience of agricultural experts or shallow image processing techniques based on statistics. However, these methods are not only inefficient but also struggle to maintain stable recognition performance in complex and ever-changing farmland environments. With the rapid development of deep learning technology, image recognition methods based on convolutional neural networks (CNNs) have gradually become the mainstream for plant disease detection. These methods can automatically extract image features, reducing manual intervention and improving recognition accuracy to some extent. However, agricultural scenarios generally suffer from a lack of high-quality, large-scale labeled data: firstly, the diverse types of plant diseases and their significant long-tailed distribution lead to high data collection and labeling costs; secondly, factors such as varying light levels, complex backgrounds, and heterogeneity of lesions in the natural environment limit the model's generalization ability. Therefore, deep learning models that rely solely on image modalities often perform poorly under small sample conditions, exhibiting insufficient feature representation, poor cross-domain adaptability, and low robustness. To alleviate the challenges posed by data scarcity, few-sample learning has gradually become a research hotspot. Typical methods include prototype networks based on metric learning, relational modeling based on graph neural networks, and transfer learning. These methods have improved the modeling effect under few sample conditions to a certain extent, but they still have shortcomings in real farmland scenarios: (1) they have limited ability to model complex backgrounds and subtle lesion features; (2) they rely on a single visual modality and are difficult to effectively utilize other information sources; (3) their cross-domain recognition performance is insufficient and their generalization ability needs to be improved. In recent years, the development of multimodal learning has provided new ideas for the identification of plant diseases with few samples. By integrating visual and textual information, the model can supplement the lack of visual features with semantic knowledge, thereby enhancing the feature representation ability and robustness. For example, combining textual descriptions of crop diseases can not only make up for the lack of image features, but also help the model capture the fine-grained differences of diseases. However, the application of existing multimodal methods in agricultural small sample identification still faces challenges: how to construct high-quality semantic descriptions that fit the actual agricultural situation, how to achieve efficient alignment of image and text feature spaces under limited sample conditions, and how to avoid information redundancy and modal conflicts in the process of multimodal fusion are all technical problems that need to be solved. Therefore, there is an urgent need for a method that can effectively integrate multimodal information and improve the performance of plant disease identification under small sample conditions, so as to achieve higher accuracy, robustness and generalization ability, thereby promoting the development and application of smart agriculture. Summary of the Invention

[0003] This invention aims to address the limitations of existing plant disease identification methods in modeling complex backgrounds and subtle lesion features under small sample conditions, their over-reliance on single visual modal information, poor generalization in cross-domain identification tasks, and insufficient robustness. It proposes a multimodal small-sample plant disease identification method combining visual and semantic approaches to improve the model's accuracy, robustness, and generalization ability in complex backgrounds and diverse lesion morphology classification scenarios. The proposed multimodal small-sample plant disease identification method includes the following steps:

[0004] (1) First, collect a dataset of plant disease images and preprocess the dataset;

[0005] (2) In response to the challenges of plant disease classification and the lack of research in this task area, a plant disease identification model VSF combining vision and semantics is proposed, including a semantic upgrade module (US), which uses a large language model and prompt words to upgrade the definition of plant diseases to generate richer, more realistic and agricultural descriptions; and a two-stage modality fusion module TSMF is introduced to effectively improve the consistency and discriminative ability of cross-modal representation.

[0006] (3) The preprocessed plant disease image data is fed into the VSF model for training and validation, and the final performance of the model is evaluated through multiple evaluation indicators;

[0007] (4) Input the plant disease images to be classified into the VSF model trained in step (3) above, and output the classification results through forward calculation.

[0008] Furthermore, the preprocessing method described in step (1) specifically includes:

[0009] (1.1) Before the experiment, the plant diseases in each image were first classified accordingly; the images input into the network were... The image size is normalized to 224*224 pixels.

[0010] (1.2) The dataset is randomly divided into training set and validation set in a ratio of 6:4.

[0011] Furthermore, the plant disease identification model VSF combining vision and semantics described in step (2) specifically includes:

[0012] (2.1) Input plant disease category C and input C into the semantic upgrade module US. The semantic upgrade module will retrieve the basic definition B related to the category from internationally recognized agricultural organization websites and agricultural journals based on the existing large language model LLM and public knowledge base. It is usually a concise description of category C. Then we define the prompt word Prompt. The prompt word will tell the large model to generate a more detailed plant disease text feature with agricultural practical significance based on the basic definition. Finally, we will obtain the text features The text features S are fed into the text encoder to obtain the latest text features.

[0013]

[0014]

[0015]

[0016] (2.2) Next, the plant disease image I is input into the network, and its image features are denoted as V. The cluster center VC of the image is calculated. The image features V will be processed by the image encoder ViT to obtain the latest image features. Next, the TSMF module uses a linear mapping to project the text features S onto the feature space to obtain text features of the same dimension. Next, the projected semantic features will be... With the latest original image features The initial joint representation X is obtained by splicing the two representations.

[0017]

[0018]

[0019]

[0020] This indicates a splicing operation, and the spliced ​​semantic information is then further fused through a two-stage progressive fusion process. and Feature fusion is performed in the first stage. Initial integration of features is achieved through normalization and nonlinear activation function operations, followed by fusion in the second stage. In the process, deep modeling and feature compression are performed on the preliminary fusion results to further harmonize the distribution differences between modes.

[0021]

[0022]

[0023] in, and Represents the ReLU activation function. and This represents the LayerNorm normalization operation; finally, the features obtained through fusion are calculated. Plant disease categories The distance between them is used to train the loss function L for model optimization, and min means minimizing the loss function;

[0024]

[0025] Furthermore, the specific training and validation methods for the VSF model described in step 3 are as follows:

[0026] (3.1) Input the preprocessed plant disease dataset in step 1 into the VSF model in step 2 for training. Set an iteration cycle of 300 rounds. After each iteration, use the validation set to verify the performance of the model generated in each iteration. Save the optimal model weight file by comparison.

[0027] (3.2) After the training iteration is completed, the optimal model obtained in (3.1) is used to evaluate the performance of the improved model by the number of model parameters, floating-point computation, and accuracy, so as to verify the effectiveness of the model.

[0028] Furthermore, the method for classifying plant disease images using the proposed model in step 4 is as follows: First, the plant disease image to be classified is input into the model and loaded into (3.1) to obtain the optimal model weights. Finally, the correct plant disease category is identified through prediction.

[0029] This invention has the following characteristics:

[0030] 1. This invention proposes a multimodal small-sample plant disease identification method. This method integrates key technologies of the semantic upgrade module (US) and the two-stage modality fusion module (TSMF), enabling accurate classification of lesion features at different scales. It improves identification accuracy while maintaining model efficiency and deployment capability, making it suitable for disease monitoring tasks in complex agricultural environments.

[0031] 2. This invention designs a semantic upgrade module (US) that uses a large language model combined with an authoritative agricultural knowledge base and prompt words to generate high-quality semantic descriptions of plant diseases, effectively making up for the lack of information in single-category labels and enhancing the model's ability to understand fine-grained disease features.

[0032] 3. This invention proposes a two-stage modality fusion module (TSMF), which achieves deep interaction and consistency alignment between visual and semantic features through semantic feature projection, splicing, normalization, nonlinear activation and feature compression, significantly improving the discriminative power and robustness of cross-modal representation.

[0033] 4. Under the multimodal few-shot learning framework, this invention combines complementary information from visual and semantic modalities, effectively solving the problems of scarce disease samples, complex backgrounds, and insufficient cross-domain generalization ability in complex farmland scenarios. It achieves significantly better recognition results than existing technologies on both standard datasets and real-world scenario datasets. Attached Figure Description

[0034] Figure 1 This is a flowchart of the plant disease classification algorithm proposed in this invention.

[0035] Figure 2 This is an example of plant disease images from the PlantVillage dataset used in this invention.

[0036] Figure 3 Examples of plant disease images from the PlantDoc dataset used in this invention.

[0037] Figure 4 This is a structural diagram of the semantic upgrade US module proposed in this invention.

[0038] Figure 5 This is a diagram of the overall framework of the VSF model proposed in this invention.

[0039] Figure 6 This is a structural diagram of the two-stage modal fusion (TSMF) module proposed in this invention.

[0040] Figure 7 This invention provides a performance evaluation of the different semantic guidelines proposed in this paper on the PlantVillage data.

[0041] Figure 8 This paper presents a performance evaluation of the different semantic guidelines proposed in this invention on PlantDoc data.

[0042] Figure 9 Examples of plant disease images collected in the natural environment according to the present invention.

[0043] Figure 10 This is an example of the recognition results of VSF on plant disease images proposed in this invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0045] This invention proposes an efficient plant disease classification algorithm, combining... Figures 1 to 10 Detailed explanation is as follows:

[0046] like Figure 1The flowchart shown is a presentation of the multimodal small-sample plant disease identification algorithm proposed in this invention. In this process, plant disease image data is first obtained through internet searches, and the dataset is preprocessed. Next, a VSF model is constructed to improve the accuracy of the plant disease identification task. This mainly includes introducing a semantic upgrade module (US) to upgrade the plant disease definition using a large language model and prompt words, generating richer, more realistic, and agriculturally relevant disease descriptions; and introducing a two-stage modality fusion module (TSMF) to effectively improve the consistency and discriminative ability of cross-modal representations. Then, the image data is input into the VSF model at a fixed size of 224*224 for training and validation. Finally, the plant disease images to be classified are input into the model weights of the tomato disease identification method proposed in this invention, and the classification result is output through forward calculation.

[0047] like Figure 2 The image shown is an example of plant disease images from the PlantVillage dataset used in this invention. This dataset is a standard publicly available dataset in the field of agricultural vision research, covering a variety of common economic crops and their various disease types. The dataset is divided into a basic set and a new set to simulate the identification needs of new diseases in real-world scenarios.

[0048] like Figure 3 The image shown is an example of a plant disease image from the PlantDoc dataset used in this invention. PlantDoc is a plant disease image dataset with real-world scene characteristics. This dataset is mainly designed for actual farmland applications and features complex backgrounds, lighting variations, diverse viewing angles, and leaf occlusion, which are consistent with the characteristics of real agricultural scenarios.

[0049] like Figure 4 The diagram shows the structure of the semantic upgrade module US proposed in this invention. The US module aims to automatically generate high-quality semantic descriptions for each plant disease category to compensate for the shortcomings of category names or brief definitions in information expression. First, based on existing large language models, plant disease categories are retrieved. The large language model searches internationally recognized agricultural organization websites and agricultural journals in public knowledge bases to obtain basic definitions related to the categories. These definitions are relatively concise and lack sufficient description of visual features. Then, the large language model uses standardized prompts to guide the model to generate richer, more realistic descriptions that conform to agricultural scenarios. In this process, the large language model combines basic definitions, official definitions, and potential agronomic common sense to expand into a paragraph-style description that includes key information such as symptoms and affected parts.

[0050] This invention proposes the VSF model. For example... Figure 5The diagram shows the structure of the VSF model proposed in this invention. Training phase: VSF processes and abstracts image and text features through a visual encoder and a text encoder. The processed image and text features are then fed into the two-stage modality fusion module TSMF to align feature dimensions and perform modality fusion. Finally, the fused features are fed into the feature space. Testing phase: Support set images, query set images, and disease text are randomly selected from the categories in the new set and fed into the visual encoder and text encoder for encoding. Next, the image and text features are input into the TSMF module to obtain the reconstructed prototypes. Finally, the query set image is selected to calculate the nearest prototype to obtain the most likely category of the query image.

[0051] like Figure 6 The diagram shows the structure of the TSMF module proposed in this invention. The TSMF module uses a linear mapping to project semantic features into the feature space, giving the representations of image features and text features a unified dimension. Subsequently, by concatenating the projected text features with the original visual features, the concatenated semantic information undergoes feature fusion through a two-stage progressive fusion step to extract deep cross-modal interaction information. Finally, the fused features are mapped back to the original feature space to reconstruct the prototype training loss function. Compared to methods that rely solely on shallow concatenation, TSMF can more fully reconcile modal differences, improve the consistency and discriminative ability of cross-modal representations, and thus exhibit stronger robustness and generalization in multimodal few-shot tasks.

[0052] To further verify the synergistic optimization effect of each innovative module on model performance, a stepped ablation experiment was designed to evaluate the effectiveness of the improvement strategy. A systematic comparative analysis of the two core components, the Semantic Upgrade Module (US) and the Two-Stage Modality Fusion Module (TSMF), was conducted using the controlled variable method. As shown in Table 1, when using only the backbone network, the accuracies for the 5-way 1-shot and 5-way 5-shot tasks were 74.68% and 86.64%, respectively. After introducing the Semantic Upgrade Module, the accuracies increased to 75.89% and 87.71%, respectively, indicating that the high-quality semantic descriptions generated by US can effectively supplement the semantic information of the category labels. When only the Two-Stage Modality Fusion Module (TSMF) was introduced, the accuracies for 1-shot and 5-shot tasks increased to 76.52% and 88.84%, respectively, verifying the role of multimodal feature alignment and deep interaction modeling in alleviating the modality gap. When both US and TSMF modules are introduced simultaneously, the model performance is further improved to 78.29% and 89.72%, respectively, representing improvements of 3.61% and 3.08% compared to the baseline model in 1-shot and 5-shot scenarios. This result demonstrates that the fine-grained semantic description provided by US and the cross-modal alignment mechanism of TSMF have a synergistic effect in terms of information complementarity: the former enriches the semantic expression of categories, while the latter ensures the effective fusion and alignment of semantic and visual features, thereby significantly improving the model's discriminative ability and generalization performance in small-sample plant disease identification.

[0053] Table 1 Ablation experiments on the PlantVillage dataset

[0054] Model Structure 5-way 1-shot accuracy rate / % 5-way 5-shot accuracy rate / % Baseline 74.68 86.64 Baseline + US 75.89 87.71 Baseline + TSMF 76.52 88.84 Baseline + US + TSMF 79.01 90.51

[0055] To further verify the synergistic optimization effect of image and text feature fusion on model performance, feature ablation experiments were designed to evaluate the effectiveness of the improvement strategy. A systematic comparative analysis of image features V and text features S was conducted using the controlled variable method. As shown in Table 2, when using only image features, the accuracies for the 5-way 1-shot and 5-way 5-shot tasks on the PlantVillage dataset were 72.98% and 88.89%, respectively, while the accuracies for the 5-way 1-shot and 5-way 5-shot tasks on the PlantDoc dataset were 47.06% and 65.67%, respectively. Compared to the complete method, this approach shows a significant performance decrease, indicating that relying solely on visual information is insufficient to fully reconstruct the category prototype. When using only text features, the 5-way 5-shot tasks on the PlantVillage and PlantDoc datasets only achieved 57.25% and 58.14%, respectively. Although semantic information contains category representativeness, the performance of the single modality is limited. Finally, by combining visual and semantic modalities, it was found that visual and semantic modalities are complementary in prototype reconstruction and can enhance the robustness and discriminative power of category representation.

[0056] Table 2. Comparison of different features on the PlantVillage (PV) and PlantDoc (PD) datasets.

[0057] Feature Source (PV) 5-way 1-shot accuracy / % (PV) (PV) 5-way 5-shot accuracy / % (PV) (PD) 5-way 1-shot accuracy / % (PD) (PD) 5-way 5-shot accuracy / % (PD) V 72.98 88.89 47.06 65.67 S 58.45 57.25 46.93 58.14 V&S 79.01 90.51 59.82 70.61

[0058] To verify the superiority of this invention over other advanced classification algorithms, comparative experiments were added, and the results are shown in Tables 3 and 4. The VSF model proposed in this invention achieves 79.01% and 90.51% accuracy in the 5-way 1-shot and 5-way 5-shot tasks on the PlantVillage dataset, respectively, and 59.82% and 70.61% accuracy in the 5-way 1-shot and 5-way 5-shot tasks on the PlantDoc dataset, respectively. Compared to other advanced classification algorithms, it demonstrates superior classification performance and achieves a balance between classification accuracy and speed.

[0059] Table 3. Comparison of few-shot classification performance of each model on the PlantVillage dataset. — Indicates that the N-way K-shot classification task was not performed.

[0060] Model computational load Parameters 5-way 1-shot accuracy rate / % 5-way 5-shot accuracy rate / % Mahalanobis 3.60 G 24.88 M 46.60 63.50 CMSFF+CA 4.70 G 12.75 M 60.70 78.10 Frequency+GC 4.70 G 12.75 M 64.50 80.90 FREN 4.70 G 12.75 M 66.10 84.20 Prune-FSL 4.70 G 12.75 M 77.17 88.86 PMF+FA 4.24 G 21.67 M —— 86.79 SMKD 4.50 G 23.00 M 71.51 86.71 ECKD 4.50 G 23.00 M 74.98 88.28 SemFew 4.50 G 23.00 M 75.62 87.15 VSF (This invention) 4.49 G 28.29 M 79.01 90.51

[0061] Table 4. Comparison of few-shot classification performance of each model on the PlantDoc dataset. — This indicates that the N-way K-shot classification task was not performed.

[0062] Model computational load Parameters 5-way 1-shot accuracy rate / % 5-way 5-shot accuracy rate / % TC-Loss 0.30 G 3.40 M 46.49 57.36 LitePlantProto 0.27 G 4.16 M —— 63.20 PlantCaFo 1.13 G 25.60 M —— 58.31 STV2F6 3.50 G 16.63 M 36.84 55.38 STV2F6+PDFC 3.70 G 16.63 M 49.29 65.88 SemFew 4.50 G 23.00 M 52.87 62.84 VSF (This invention) 4.49 G 28.29 M 59.82 70.61

[0063] like Figure 7 This paper presents the performance evaluation of different semantic guidance proposed in this invention on the PlantVillage dataset. In a 5-way 1-shot configuration, the accuracy of the method without semantics is 69.27%, the accuracy with category semantics is 76.52%, and the accuracy with semantic upgrade is 78.29%. In a 5-way 5-shot configuration, the accuracy of the method without semantics is 72.98%, the accuracy with category semantics is 88.84%, and the accuracy with semantic upgrade further improves to 90.51%. This demonstrates that introducing semantic information can significantly improve the model's discriminative performance in few-shot classification tasks, and the semantic upgrade method is superior to the category semantic method, further enhancing the model's generalization ability in cross-class few-shot tasks.

[0064] like Figure 8 This paper evaluates the performance of different semantic guidance methods proposed in this invention on the PlantDoc dataset. Experimental results show that semantic guidance significantly improves model accuracy in both 5-way 1-shot and 5-way 5-shot tasks. In the 5-way 1-shot setting, the accuracy of the method without semantic guidance is 47.06%, while the accuracy of the categorical semantic guidance and semantic upgrade methods are improved to 58.58% and 59.82%, respectively, with the semantic upgrade method performing best. In the 5-way 5-shot scenario, the performance of all methods generally improves: the accuracy of the method without semantic guidance rises to 65.67%, the categorical semantic guidance reaches 68.73%, and the semantic upgrade method leads with the highest accuracy of 70.61%. The overall trend shows that upgrading the textual semantic information of plant diseases can effectively enhance the robustness of the model, further highlighting the application potential of semantic upgrade (US) in plant disease diagnosis tasks.

[0065] The following is combined with Figure 9 and Figure 10 Here is an application example:

[0066] like Figure 9 Examples of plant disease images in natural scenes obtained by the image acquisition module are shown in (a), (b), (c), and (d). To verify the robustness of the proposed method in real-world scenarios, [the following text is incomplete and requires further context]. Figure 9 The plant disease images (a) and (b) to be classified are input into the weights of the trained VSF model, and the classification results are output through calculation.

[0067] Classification results as follows Figure 10 As shown, Figure 9Examples of plant disease images in the image: (a) has a recognition probability of 0.9012 and is classified as apple black rot; (b) has a recognition probability of 0.9007 and is classified as corn gray spot. Figure 9 The plant disease images (a) and (b) in the examples can be effectively classified and have practical application significance.

Claims

1. A multi-modal small sample plant disease recognition method, characterized in that, The method comprises the following steps: a. First, collect a plant disease image dataset , plant disease images of the input network , and obtain a processed image dataset ; b. Constructing a plant disease recognition model VSF combining vision and semantics, including a semantic upgrading module US, using a large language model and a prompt word Prompt to upgrade the definition of plant diseases to generate more rich, real and agricultural scene-compliant disease descriptions; A two-stage modality fusion module (TSMF) is introduced to effectively improve the consistency and discriminative ability of cross-modal representations; the semantic upgrade module (US) in step b specifically includes: the plant disease image input to the network in step a. Define the plant disease category name as C, and calculate the plant disease image. The cluster center VC is used as a prototype. Then, the plant disease category C is input into the large language model LLM. The large language model retrieves the basic definition B related to category C by searching the websites of international official agricultural organizations and agricultural journals. This B is a concise description of category C. Then, a custom prompt word is used to tell the large language model to generate a more detailed plant disease text feature with practical agricultural significance based on the basic definition. Finally, the obtained text features The text is fed into the text encoder to obtain the latest text feature S: , , , wherein the large language model LLM uses GPT-4o, Prompt represents a prompt word, and CLIP is a text encoder; the two-stage modal fusion module TSMF in step b specifically comprises: the plant disease image of the input network in step a is processed by an image encoder ViT to obtain the latest image feature , the image feature is processed by the image encoder ViT to obtain the latest image feature , the TSMF module first projects the text feature S to the feature space by linear mapping to obtain a text feature of the same dimension , next, the projected text feature is spliced with the latest image feature to obtain an initial joint representation X, denotes the concatenation operation, the concatenated semantic information is then fused by a two-stage progressive fusion procedure and The first stage fusion achieves the preliminary integration of features through normalization and nonlinear activation function operations, followed by the second stage fusion in which the preliminary fusion results are deeply modeled and feature compressed to further reconcile the distributional differences between modalities;​​​ , , wherein, and denote the ReLU activation function, and denote the LayerNorm normalization operation; finally the model is optimized by computing the loss function L that fuses the features and the distance between the features and the plant disease class minimizes the loss function; c. The preprocessed plant disease image data is sent into the VSF model for training and verification, the optimal model weight is saved through iterative training, and the model performance is evaluated using the verification set, and the evaluation indicators include parameter quantity, floating point calculation quantity and accuracy; d. The plant disease image to be classified is input into the VSF model trained in step c, and the classification result is output through forward calculation.

2. The method of claim 1, wherein, The method for collecting plant disease image data set in step a specifically comprises: Images inputted into the network The image size is normalized to 224*224 pixels, and the dataset is randomly divided into training and validation sets in a 6:4 ratio.

3. The method of claim 1, wherein, The method for sending the preprocessed plant disease image data into the VSF model for training and verification in step c specifically comprises: The preprocessed plant disease data set is input into the model for training for 300 iteration cycles; after each iteration, the performance of the model generated by each iteration is verified using the verification set, the optimal model weight file is saved by comparison; the improved model performance is evaluated by the parameter quantity, floating point calculation quantity and accuracy of the model.

4. The method of claim 1, wherein, The method for inputting the plant disease image to be classified into the VSF model trained in step c specifically comprises: The plant disease image to be classified G is adaptively scaled to a resolution size of 224*224 pixels, and then the image G is input into the model VSF loaded with weights, and each layer module of the model extracts features from the image using the weights to identify the key features in the image; secondly, the features are calculated through a full connection layer to convert the features into class probabilities; finally, the model selects the class with the highest probability as the prediction result, that is, the input plant disease image G is considered to belong to the class.

Images