Crop disease diagnosis method and system based on multi-mode automatic prompt optimization
By using a multimodal automatic prompting optimization method, a few-sample image-label example set and optimized disease description text are constructed, which solves the problems of data dependence and knowledge gap in crop disease identification and achieves efficient and accurate disease identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-07
AI Technical Summary
Existing technologies rely on large-scale labeled data for crop disease identification, and large language models lack agricultural expertise, resulting in low identification accuracy and poor generalization ability, especially in fine-grained tasks.
A multimodal automatic prompting optimization method is adopted. A small sample image-label example set is constructed through two-stage screening and the disease description text is optimized. Combined with evolutionary algorithm, multimodal prompts are generated, and visual and text features are automatically optimized to reduce the dependence on large-scale labeled data.
It significantly improves the accuracy and generalization ability of large language models in crop disease identification, reduces data and computing costs, is applicable to different crops and datasets, and improves cross-model performance consistency.
Smart Images
Figure CN121811253A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a crop disease diagnosis method and system based on multimodal automatic prompting optimization. Background Technology
[0002] Agriculture is the cornerstone of the global economy, directly impacting food security and ecological sustainability. However, crop diseases pose a serious threat to agricultural production; reports indicate that pests and diseases can cause yield losses exceeding 50% in developing countries. Therefore, rapid and accurate identification of crop diseases is crucial for ensuring food production and advancing precision agriculture.
[0003] Currently, deep learning-based visual recognition technology for crop diseases has made significant progress. Models such as convolutional neural networks and Transformers have demonstrated excellent classification performance on specific datasets. However, these methods have three key limitations: First, they heavily rely on large-scale, high-quality labeled datasets, which are costly and difficult to obtain for many diseases, especially rare ones. Second, the models have poor generalization ability and are sensitive to environmental changes such as light intensity, shooting angle, and crop growth stage, and their performance may drop sharply in new scenarios. Third, for diseases with similar symptoms or in their early stages, the models struggle to distinguish subtle features, leading to a high misclassification rate.
[0004] Multimodal large language models (LLMs), especially those with visual understanding capabilities such as GPT-4o, offer a new paradigm for addressing the aforementioned data dependency problem. They can handle image classification tasks through zero-shot or few-shot learning without specialized visual model training. However, these general-purpose models are pre-trained on general-domain data, lacking agricultural expertise, resulting in significant "knowledge blind spots" and limited accuracy when handling fine-grained tasks such as crop disease identification. Furthermore, the performance of few-shot learning is highly dependent on the quality of the provided examples; randomly selected examples often contain noise or are unrepresentative, thus limiting the model's learning effectiveness.
[0005] To bridge this performance gap, research has gradually shifted from data-intensive model fine-tuning to data-efficient cue engineering. While supervised fine-tuning effectively injects domain knowledge, it reintroduces high costs for data acquisition and annotation, contradicting the original intention of using pre-trained LLMs. As an alternative, training-free cue engineering guides the inference process of the LLM by designing guided inputs without modifying model parameters. Early cue engineering relied on experts manually writing text cues or selecting a small number of images; this approach was not only time-consuming and labor-intensive, but its effectiveness also depended on the expert's experience level, resulting in cues with poor generalization.
[0006] To overcome the limitations of manual design, Automatic Prompt Optimization (APO) technology has emerged, refactoring prompting engineering into an optimization problem by automatically searching for the optimal prompt through algorithms. However, existing APO methods primarily focus on optimizing plain text prompts, which is insufficient for visual tasks such as crop disease identification, as plain text cannot fully capture key visual details such as lesion shape and color gradation. Furthermore, relying solely on a few random images fails to provide stable visual references. Therefore, there is an urgent need in the current technological field for a framework that can automatically and collaboratively optimize multimodal prompts (i.e., combining high-quality image examples with highly informative text descriptions) to fully leverage the potential of LLMs in specialized visual tasks. Summary of the Invention
[0007] To address the problems in the background technology, this invention proposes a crop disease diagnosis method and system based on multimodal automatic prompting optimization. It aims to solve the problems of existing technologies' reliance on large-scale labeled data and the knowledge gap in fine-grained agricultural identification using large language models.
[0008] The technical solution adopted in this invention is: A crop disease diagnosis method based on multimodal automatic prompting optimization includes the following steps: S1. Obtain crop disease images for each disease category, and sequentially perform semantic relevance screening and visual diversity screening on the crop disease images for each disease category to construct a small sample example set for each disease category. S2. Obtain the initial disease description text for each disease category, combine the few sample sets of each disease category with the initial disease description text, and construct the initial multimodal test prompt. S3. Obtain several crop disease images with labeled disease categories, and then construct a dataset; S4. Optimize the disease description text based on the initial multimodal test prompts and dataset to obtain the optimized disease description text for each disease category; S5. Combine the few sample example sets of each disease category with the optimized disease description text to construct an optimized multimodal test prompt. Input the crop disease image to be diagnosed and the optimized multimodal test prompt into the large language model and process it to obtain the crop disease category of the crop disease image to be diagnosed.
[0009] Specifically, S1 is: S1.1 Obtain crop disease images for each disease category and construct a candidate image pool for each disease category; S1.2 Calculate the semantic similarity between all crop disease images and the corresponding disease category text labels in the candidate image pool for each disease category, and select crop disease images with semantic similarity higher than a preset threshold as the semantically related image set for each disease category. S1.3. Extract features from crop disease images in the semantically related image set for each disease category, and cluster them to obtain several cluster groups. Select the image with the smallest feature distance from the cluster center from each cluster group as a representative sample. S1.4. Representative samples from all clusters of each disease category constitute a small sample instance set for the corresponding disease category.
[0010] The initial disease description text for each disease category is obtained in the following way: each disease category is input into a large language model for processing to generate the initial disease description text for the corresponding disease category.
[0011] Specifically, S4 is: S4.1 Input the multimodal test prompts and dataset into the large language model for training to obtain the predicted crop disease categories of all crop disease images in the training set, and then calculate the classification accuracy based on the predicted crop disease categories and the actual crop disease categories. S4.2 Input the disease description text of each disease category into multiple parallel mutation operations. The mutation operations replace the disease description words in the disease description text of each disease category according to the classification accuracy to generate mutated disease description text. Input the mutated disease description text output by all mutation operations into the cross operation to generate new disease description text for each disease category. S4.3 Combine the few sample sets of each disease category with the new disease description text and update the multimodal test prompts; S4.4 Repeat steps S4.1 to S4.3 until the preset number of repetitions is reached. Use the disease description text used for each disease category when the classification accuracy is highest in each cycle as the optimized disease description text for each disease category.
[0012] The mutation operation specifically involves: randomly sampling a few images from the candidate image pool as the current round; based on the few images of the current round, the classification accuracy and confusion matrix generated in the current round, and combined with the classification accuracy, confusion matrix and historical disease description vocabulary of the previous rounds, replacing the disease description vocabulary in the disease description text of each disease category to generate a mutated disease description text.
[0013] The crossover operation specifically involves selecting the disease description text with the highest classification accuracy in each disease category from all the variant disease description texts output by the mutation operation, and using it as the disease description text for the current disease category in the new comprehensive disease description text set.
[0014] The large language model used is the GPT-4o model.
[0015] A crop disease diagnosis system includes: The few-sample example filtering unit is used to perform semantic relevance filtering and visual diversity filtering on crop disease images with labeled disease categories, and to construct a few-sample example set for each disease category. The symptom description automatic optimization unit is used to optimize the preset initial disease description text for each disease category using an evolutionary algorithm, and generate optimized disease description text for each disease category. A large language model is used to determine the crop disease category of an input crop disease image based on a small sample set of examples for each disease category and disease description text.
[0016] The beneficial effects of this invention are: Compared to traditional deep learning methods, this invention requires only a small number of image samples for the suggestion selection and optimization process, greatly reducing the dependence on large-scale, finely labeled datasets, making it particularly suitable for data-scarce scenarios in agriculture. The automatic suggestion optimization agent proposed in this invention fully automates the suggestion design process, which originally relied on expert experience, allowing it to be easily extended to new crop varieties or disease types. Users only need to provide a small number of unlabeled images, and the system can automatically generate efficient diagnostic suggestions.
[0017] Furthermore, the optimized multimodal suggestions generated by this invention exhibit excellent transferability, demonstrating robustness across different crops and datasets, and effectively improving the performance of large language models with varying architectures. This proves the effectiveness and robustness of this invention as a general enhancement strategy. By automatically optimizing multimodal suggestions, this invention significantly improves the performance of general large language models on fine-grained disease identification tasks. Without requiring any parameter fine-tuning of the large language model, it increases the accuracy of GPT-4o on soybean disease classification from 40.3% to 90.3%, saving substantial computational resources and data costs. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the overall framework of the present invention.
[0019] Figure 2 This is a flowchart illustrating the process of optimizing disease description text in this invention.
[0020] Figure 3This is a schematic diagram of the process for generating a small sample set in this invention.
[0021] Figure 4 This is a comparison chart showing the accuracy of the method of this invention with that of various baseline large language models on soybean disease classification tasks.
[0022] Figure 5 The figure shows the experimental results of the scalability of the method of the present invention on newly added crop diseases.
[0023] Figure 6 The figure shows the experimental results of the generalization ability of the optimization hints generated by the method of the present invention on different models.
[0024] Figure 7 The figure shows the experimental results of the cross-dataset generalization ability of the optimization hints generated by the method of this invention on the PlantSeg dataset. Detailed Implementation
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0027] like Figure 1 As shown, this embodiment includes the following steps: S1. Obtain crop disease images for each disease category, and sequentially perform semantic relevance screening and visual diversity screening on the crop disease images for each disease category to construct a small sample example set for each disease category. That is, a two-stage screening strategy is adopted to select a small sample image-label example set that is both representative and diverse for each disease category from the candidate image pool; S2. Obtain the initial disease description text for each disease category, combine the few sample example set of each disease category with the initial disease description text of each disease category, and construct the initial multimodal test prompt. S3. Obtain several crop disease images with labeled disease categories, and then construct a dataset; S4. Optimize the initial disease description text based on the initial multimodal test prompts and dataset to obtain optimized disease description text for each disease category; That is, to build and run an automatic prompting optimization agent, which is based on an evolutionary algorithm and uses visual feedback to iteratively generate and optimize a set of fine-grained symptom description texts corresponding to each disease category; S5. Combine the few sample example sets of each disease category with the optimized disease description text of each disease category to construct the optimized multimodal test prompt. Input the crop disease image to be diagnosed and the optimized multimodal test prompt into the large language model, and process it to obtain the crop disease image analysis and the corresponding crop disease category.
[0028] The process involves combining a set of few-sample images and labels with an optimized set of symptom description texts into a multimodal cue. This cue, along with the crop disease image to be diagnosed, is then input into a disease classification agent based on a large language model. The disease classification agent uses thought chain reasoning to analyze the image to be diagnosed based on the multimodal cue and outputs the disease identification result.
[0029] S1 specifically refers to: S1.1 Obtain crop disease images for each disease category and construct a candidate image pool for each disease category; S1.2 Calculate the semantic similarity between all crop disease images and the corresponding disease category text labels in the candidate image pool for each disease category, and select crop disease images with semantic similarity higher than a preset threshold as the semantically related image set for each disease category. S1.3. Extract features from crop disease images in the semantically related image set for each disease category, and cluster them to obtain several cluster groups. Select the image with the smallest feature distance from the cluster center from each cluster group as a representative sample. S1.4. Representative samples from all clusters of each disease category constitute a small sample instance set for the corresponding disease category.
[0030] The initial disease description text for each disease category is obtained in the following way: each disease category is input into a large language model for processing to generate the initial disease description text for the corresponding disease category.
[0031] The two-stage screening strategy includes: Stage 1: Semantic representativeness screening, which uses a pre-trained visual-language model to calculate the semantic similarity between candidate images and disease category text labels, and selects candidate images that are highly semantically related to the disease category; Stage 2: Visual diversity screening, which uses a pre-trained convolutional neural network to extract the deep visual features of the candidate images that are highly semantically related, and combines a clustering algorithm to group the images, and selects a representative sample from each group to form a few-sample image-label example set.
[0032] like Figure 3As shown, this two-stage screening strategy aims to address the problem of inconsistent example quality misleading the model in few-shot learning. The first stage is semantic representativeness screening based on CLIP. This stage utilizes a pre-trained visual-language model (such as CLIP) to calculate the semantic relevance between each image in the candidate image pool for a specific disease category (e.g., "soybean rust") and the disease text label (e.g., "images of soybean rust"). It then selects the images with the highest semantic similarity scores, forming a high-quality, thematically clear candidate pool to ensure the semantic purity of the examples. The second stage is visual diversity screening based on ResNet and K-Means. This stage selects a set of examples from the semantically relevant candidate pool that can cover multiple visual manifestations of the disease. First, a pre-trained convolutional neural network (such as ResNet) is used to extract deep visual feature vectors from the candidate images. Second, the feature vectors are input into a K-Means clustering algorithm for grouping, with each cluster representing a unique visual sub-mode of the disease. Finally, the image closest to the centroid of each cluster is selected as a representative sample, collectively forming the final few-shot image-label example set.
[0033] Each category's few-sample example set corresponds to a disease description text, and crop disease images in the same category's few-sample example set use the same disease description text.
[0034] S4 specifically refers to: S4.1 Input the multimodal test prompts and dataset into the large language model for training to obtain the predicted crop disease categories of all crop disease images in the training set, and then calculate the classification accuracy and confusion matrix based on the predicted crop disease categories and the real crop disease categories. S4.2 Input the disease description text of each disease category into multiple parallel mutation operations. The mutation operations replace the disease description words in the disease description text of each disease category according to the classification accuracy and confusion matrix to generate mutated disease description text. Input the mutated disease description text output by all mutation operations into the cross operation to generate new disease description text for each disease category. S4.3 Combine the few sample instances of each disease category with the new disease description text of each disease category, and update the multimodal test prompts; S4.4 Repeat steps S4.1 to S4.3 until the preset number of repetitions is reached. Use the disease description text used for each disease category when the classification accuracy is highest in each cycle as the optimized disease description text for each disease category.
[0035] The iterative optimization process of the auto-suggestion optimization agent includes: executing a loop comprising generation, evaluation, optimization, and selection phases; in the evaluation phase, its quality is quantified by testing the classification performance of the current symptom description text set on a training set, and generating visual feedback including classification accuracy and confusion matrix. The auto-suggestion optimization agent also includes a memory module, which dynamically records descriptive features related to improvements or declines in classification performance during iterations, and guides the generation of new descriptions in subsequent optimization phases.
[0036] The optimization phase includes mutation and crossover operations; mutation is used to generate multiple candidate description sets, and crossover is used to combine the advantages of multiple candidate description sets to synthesize a better new description set.
[0037] like Figure 2 As shown, this Automatic Prompt Optimization (APO) agent is implemented based on GPT-4o and integrates an evolutionary algorithm framework. Its core workflow is an iterative loop of "generation-evaluation-optimization-selection". First, initial description generation is performed, providing representative images for each disease category and instructing GPT-4o to generate an initial list containing multiple observable visual features. Second, visual feedback evaluation is performed, using the current text description set as the classification basis, conducting classification tests on a small training set, calculating performance metrics such as overall accuracy and confusion matrix, and translating these metrics into natural language feedback to guide optimization. Next, iterative evolutionary optimization is performed. This process simulates mutation and crossover operations in biological evolution. In each round of optimization, the agent samples a new set of images as dynamic visual cues to avoid overfitting. Mutation operations are used to generate multiple candidate description sets, while crossover operations combine the advantages of multiple candidate sets to synthesize a better new description set. Furthermore, the agent can also include a memory to dynamically record descriptive features related to improvements or declines in classification performance, thereby improving optimization efficiency and stability. The entire iteration loop terminates after reaching a preset number of iterations or when performance convergence, and outputs the set of symptom description texts that showed the best performance throughout the entire optimization history.
[0038] The mutation operation is as follows: randomly sample the candidate image pool as the few-sample image for the current round, and based on the few-sample image of the current round, the classification accuracy and confusion matrix generated in the current round, and combined with the classification accuracy, confusion matrix and historical disease description vocabulary of the previous rounds, replace the disease description vocabulary in the disease description text of each disease category to generate a mutated disease description text.
[0039] The mutation operation, in this embodiment, aims to mine more fine-grained discriminative symptom features through local search. The large language model is configured as the optimizer, which, at the start of each optimization cycle, receives the currently optimal disease description text, natural language visual feedback derived from classification accuracy and confusion matrix, and feature validity information recorded in memory. Simultaneously, to ensure that text generation is always based on diverse visual evidence and to avoid overfitting, the system provides the optimizer with a set of randomly sampled disease images from the current cycle as dynamic visual cues. The optimizer identifies ambiguous descriptions that cause category confusion based on the feedback information and extracts new visual features from the dynamic visual cues, such as lesion edge morphology, distribution patterns, or color gradients, to replace them. In this embodiment, the number K of the candidate disease description text set is set, and the maximum number of feature replacements for each disease category is 5.
[0040] The crossover operation is as follows: among the variant disease description texts output by all mutation operations, select the disease description text with the highest classification accuracy in each disease category, and use it as the disease description text for the current disease category in the new comprehensive disease description text set.
[0041] The crossover operation, as described in this embodiment, aims to synthesize the best-performing suggestion scheme through global selection. The system first evaluates the classification performance of the K candidate disease description text sets generated by the mutation operation, calculating their classification accuracy on the dataset. Then, all candidate sets and their corresponding accuracies are fed back to the optimizer, which executes the feature recombination logic. The optimizer analyzes and identifies effective feature combinations that co-occur in multiple high-accuracy candidate sets, and synthesizes these preferred features across sets into a new comprehensive disease description text set. If the classification accuracy of this comprehensive disease description text set is better than any single candidate set generated by the mutation operation, it is adopted as the current best result and proceeds to the next iteration; otherwise, the best-performing single candidate set from the mutation stage is retained. This crossover mechanism based on "natural selection" ensures that the disease description texts can effectively integrate multi-dimensional domain knowledge, thereby continuously approaching the global optimum.
[0042] The selected image examples and optimized text descriptions are combined into a multimodal prompt, and the disease classification agent is guided to make a diagnosis using thought chain reasoning.
[0043] Specifically, the disease diagnosis unit integrates optimized image-label pairs and optimized text descriptions into a structured multimodal cue. When classifying an image of a crop disease to be diagnosed, this multimodal cue, along with the image, is input into a classification agent based on a large language model. The classification agent employs thought chain (CoT) reasoning guidance. The model is instructed to perform a step-by-step logical reasoning process: 1) First, describe in detail the key visual features in the image to be diagnosed; 2) Then, compare the observed features one by one with the image examples and text descriptions provided in the multimodal cue; 3) Finally, based on detailed comparative analysis, make a final classification judgment and output the disease identification result. This structured reasoning process not only improves the accuracy of classification but also enhances the interpretability of the results.
[0044] The large language model adopts the GPT-4o model.
[0045] A crop disease diagnosis system includes: The few-sample example filtering unit is used to perform semantic relevance filtering and visual diversity filtering on crop disease images with labeled disease categories, and to construct a few-sample example set for each disease category. The symptom description automatic optimization unit is used to optimize the preset initial disease description text for each disease category using an evolutionary algorithm, and generate optimized disease description text for each disease category. A large language model is used to generate initial disease description text and determine the crop disease category of the input crop disease image based on a small sample set of examples for each disease category and the disease description text.
[0046] Specifically, the few-sample example screening unit executes a two-stage screening strategy to select a few-sample image-label example set that is both representative and diverse for each disease category; the symptom description automatic optimization unit is configured as an automatic prompting optimization agent based on an evolutionary algorithm, which uses visual feedback to iteratively generate and optimize a fine-grained symptom description text set corresponding to each disease category; the disease diagnosis unit is configured as a disease classification agent based on a large language model, which receives multimodal prompts consisting of the few-sample image-label example set and the optimized symptom description text set, and uses thought chain reasoning to diagnose the input crop disease image to be diagnosed, and outputs the disease identification result.
[0047] The symptom description automatic optimization unit also includes a memory module and an evolutionary optimization module that performs mutation and crossover operations.
[0048] This embodiment first employs a two-stage screening strategy to construct a high-quality, low-sample image example library; secondly, it utilizes an evolutionary algorithm-based agent autonomous optimization to generate a fine-grained disease description text library; finally, it integrates image examples and text descriptions into a multimodal prompt, and combines thought chain reasoning to guide a large language model for accurate diagnosis.
[0049] The performance effectiveness of the method and system (AgriMAPO) proposed in this invention will be evaluated below.
[0050] like Figure 4 As shown, in the eight-category soybean disease classification task, the method of this invention achieved an average accuracy of 90.3%, which is 50 percentage points higher than the baseline GPT-4o model's 40.3%, and significantly better than other mainstream models. This proves that the present invention effectively bridges the knowledge gap in the agricultural vertical field of general large language models through prompting engineering.
[0051] To verify the scalability of the method of this invention, the performance of the framework was tested on a newly added crop (apple and tomato) disease dataset. Experimental results are as follows: Figure 5 As shown, in a task involving the identification of apple and tomato diseases across 15 categories, the method of this invention improved the average accuracy of GPT-4o from 44.9% to 84.6%, an increase of 39.7 percentage points. At the fine-grained identification level, the optimized model demonstrated excellent discrimination ability in categories where the baseline model performed extremely poorly (such as apple black rot and tomato mosaic virus), with accuracy improving from near zero to near 1.0. This proves that the method of this invention can effectively guide the model to focus on key disease characteristics, achieving accurate identification.
[0052] To verify the effectiveness of the method of this invention as a general enhancement strategy, the same set of optimization suggestions generated by the method of this invention were applied to baseline models with different architectures (Claude-3.5-sonnet and Gemini-1.5-pro). Experimental results are as follows. Figure 6 As shown, the performance of both models was significantly improved. The accuracy of Claude-3.5-sonnet increased from 21.3% to 75.3%, and the accuracy of Gemini-1.5-pro increased from 42.4% to 76.9%. After applying the strategy of this invention, the two models with large initial performance differences eventually reached similar high-performance levels, exhibiting a "performance convergence" effect. This result shows that the optimization hints generated by this invention are the core driving force for performance improvement, effectively addressing the common shortcomings of different models in agricultural disease identification tasks, and demonstrating that the method of this invention has strong cross-model generalization ability.
[0053] To verify the robustness and transferability of the optimized suggestions generated by the method of this invention on different visual feature datasets, cross-dataset generalization experiments were conducted. The optimized suggestions generated on the CDIP dataset were directly applied to the PlantSeg dataset, which has significantly different image features, for testing. Experimental results are as follows: Figure 7 As shown, the introduction of multimodal prompts resulted in a general and significant improvement in the classification performance of all tested models. Specifically, GPT-4o's accuracy improved by 18.7 percentage points, Gemini-1.5-pro by 23.9 percentage points, and Claude-3.5-sonnet by 21.7 percentage points. This result demonstrates that the domain knowledge contained in the optimized prompts generated by this invention has strong universality and does not overfit to the image features of specific datasets. It can effectively compensate for the knowledge gaps of general models in different agricultural scenarios, proving that the method of this invention has excellent cross-dataset generalization ability.
[0054] The above detailed embodiments illustrate the technical solution and beneficial effects of the present invention. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A crop disease diagnosis method based on multimodal automatic prompting optimization, characterized in that, The method includes the following steps: S1. Obtain crop disease images for each disease category, and sequentially perform semantic relevance screening and visual diversity screening on the crop disease images for each disease category to construct a small sample example set for each disease category. S2. Obtain the initial disease description text for each disease category, combine the few sample sets of each disease category with the initial disease description text, and construct the initial multimodal test prompt. S3. Obtain several crop disease images with labeled disease categories, and then construct a dataset; S4. Optimize the disease description text based on the initial multimodal test prompts and dataset to obtain the optimized disease description text for each disease category; S5. Combine the few sample example sets of each disease category with the optimized disease description text to construct an optimized multimodal test prompt. Input the crop disease image to be diagnosed and the optimized multimodal test prompt into the large language model and process it to obtain the crop disease category of the crop disease image to be diagnosed.
2. The crop disease diagnosis method based on multimodal automatic prompting optimization according to claim 1, characterized in that: Specifically, S1 is: S1.1 Obtain crop disease images for each disease category and construct a candidate image pool for each disease category; S1.2 Calculate the semantic similarity between all crop disease images and the corresponding disease category text labels in the candidate image pool for each disease category, and select crop disease images with semantic similarity higher than a preset threshold as the semantically related image set for each disease category. S1.
3. Extract features from crop disease images in the semantically related image set for each disease category, and cluster them to obtain several cluster groups. Select the image with the smallest feature distance from the cluster center from each cluster group as a representative sample. S1.
4. Representative samples from all clusters of each disease category constitute a small sample instance set for the corresponding disease category.
3. The crop disease diagnosis method based on multimodal automatic prompting optimization according to claim 1, characterized in that: The initial disease description text for each disease category is obtained in the following way: each disease category is input into a large language model for processing to generate the initial disease description text for the corresponding disease category.
4. The crop disease diagnosis method based on multimodal automatic prompting optimization according to claim 1, characterized in that: Specifically, S4 is: S4.1 Input the multimodal test prompts and dataset into the large language model for training to obtain the predicted crop disease categories of all crop disease images in the training set, and then calculate the classification accuracy based on the predicted crop disease categories and the actual crop disease categories. S4.2 Input the disease description text of each disease category into multiple parallel mutation operations. The mutation operations replace the disease description words in the disease description text of each disease category according to the classification accuracy to generate mutated disease description text. Input the mutated disease description text output by all mutation operations into the cross operation to generate new disease description text for each disease category. S4.3 Combine the few sample sets of each disease category with the new disease description text and update the multimodal test prompts; S4.4 Repeat steps S4.1 to S4.3 until the preset number of repetitions is reached. Use the disease description text used for each disease category when the classification accuracy is highest in each cycle as the optimized disease description text for each disease category.
5. The crop disease diagnosis method based on multimodal automatic prompting optimization according to claim 1, characterized in that: The mutation operation specifically involves: randomly sampling a few images from the candidate image pool as the current round; based on the few images of the current round, the classification accuracy and confusion matrix generated in the current round, and combined with the classification accuracy, confusion matrix and historical disease description vocabulary of the previous rounds, replacing the disease description vocabulary in the disease description text of each disease category to generate a mutated disease description text.
6. The crop disease diagnosis method based on multimodal automatic prompting optimization according to claim 1, characterized in that: The crossover operation specifically involves selecting the disease description text with the highest classification accuracy in each disease category from all the variant disease description texts output by the mutation operation, and using it as the disease description text for the current disease category in the new comprehensive disease description text set.
7. The crop disease diagnosis method based on multimodal automatic prompting optimization according to claim 1, characterized in that: The large language model used is the GPT-4o model.
8. A crop disease diagnosis system applied to the crop disease diagnosis method as described in any one of claims 1 to 7, characterized in that, include: The few-sample example filtering unit is used to perform semantic relevance filtering and visual diversity filtering on crop disease images with labeled disease categories, and to construct a few-sample example set for each disease category. The symptom description automatic optimization unit is used to optimize the preset initial disease description text for each disease category using an evolutionary algorithm, and generate optimized disease description text for each disease category. A large language model is used to determine the crop disease category of an input crop disease image based on a small sample set of examples for each disease category and disease description text.