Zero-sample plant disease image representation description generation method based on concept enhancement

Abstract

The invention discloses a zero-sample plant disease image representation description generation method based on concept enhancement. The method comprises the following steps: acquiring a plant disease image; obtaining a plant disease image representation description data set and an original text based on the plant disease image; obtaining a support text based on the representation description data set; based on the support text and the original text, a dependency structure directed graph is constructed, and the dependency structure directed graph is used for indicating a corresponding dependency relationship between the support text and the original text; obtaining a key concept pair according to the dependency structure directed graph; and inputting the key concept pair into a large language LLAMA3 model to obtain a real description text. The invention aims to improve the accuracy and comprehensiveness of plant pathological shape description and make up the defects of numerous methods at present, thereby constructing an efficient and accurate plant disease image phenotype description method.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and in particular relates to a zero-sample plant disease image representation description generation method based on concept enhancement. Background Art

[0002] Crops are often threatened by various diseases during their growth, which affects their yield and quality. Therefore, timely and accurate analysis and description of the pathological traits of staple crops are of great significance for their high-quality growth monitoring and disease diagnosis. Through accurate description of pathological traits, early disease warning and precise disease prevention and control can be achieved, thereby ensuring the healthy growth and high yield of crops. Through accurate description of pathological traits, early disease warning and precise disease prevention and control can be achieved, thereby ensuring the healthy growth and high yield of crops. For example, by describing the lesions on rice leaves in detail, farmers can be helped to discover and take measures in time to prevent the further spread of the disease. Similarly, for wheat rust, by accurately describing its symptoms and occurrence sites, farmers can choose appropriate pesticides for prevention and control, thereby reducing yield losses.

[0003] Plant disease image representation and description technology has broad application potential in crop disease and pest identification, agricultural remote sensing image analysis, agricultural robot visual navigation, and agricultural product quality assessment. By learning shared features between different categories, this technology can generate accurate natural language descriptions to help farmers identify and respond to new diseases and pests, analyze complex agricultural scenarios, and provide farmers with more comprehensive and accurate information to help them make better decisions. The widespread application of zero-sample image description technology in the agricultural field is expected to promote the development of smart agriculture and inject new vitality into modern agriculture.

[0004] In recent years, research on the generation of pathological shape descriptions of plant disease image phenotypes has mainly been done by using pre-trained visual encoders to construct features of the input image, and using adapters to map these features to embeddings, which are then processed as the input part of the language model. However, although these methods have great potential in theory, there are still some problems in practical applications:

[0005] 1. Although the visual encoder based on the attention mechanism can effectively extract local salient features, it is difficult to fully grasp the global representation of the image, resulting in the generated description text often focusing on a certain salient point and ignoring other important features; secondly, this limitation makes the description text appear single and lacks information when it comes to complex disease features, and cannot provide a comprehensive and accurate disease description.

[0006] 2. High computing resource consumption. In order to adapt the pre-trained model to a specific task, a lot of computing resources and time are usually required for fine-tuning. This process requires not only high-performance computing equipment, but also high-quality image-text pair datasets to ensure accurate image feature learning and description generation. However, the collection and annotation of datasets are time-consuming and labor-intensive, and the data quality directly affects the performance of the model. Even if a lot of resources are invested, the improvement in model performance after fine-tuning may still be weak and difficult to meet actual needs. These problems have brought huge challenges to research teams and individuals with limited resources, limiting the widespread application of zero-shot learning.

[0007] In summary, the generation of pathological shape descriptions of plant disease image phenotypes is a topic with important research value and development prospects. Summary of the invention

[0008] In order to solve the above technical problems, the present invention will propose a zero-sample plant disease image representation description generation method based on concept enhancement. The present invention aims to improve the accuracy and comprehensiveness of plant pathological shape description, make up for the defects of many current methods, and thus construct an efficient and accurate plant disease image phenotypic description method.

[0009] The present invention provides a method for generating a zero-shot plant disease image representation description based on concept enhancement, comprising:

[0010] Obtain plant disease images;

[0011] Based on the plant disease image, obtaining a plant disease image representation description dataset and original text;

[0012] Based on the representation description dataset, obtaining supporting text;

[0013] Based on the supporting text and the original text, construct a dependency structure directed graph, wherein the dependency structure directed graph is used to indicate the corresponding dependency relationship between the supporting text and the original text;

[0014] According to the dependency structure directed graph, obtaining key concept pairs;

[0015] The key concept pairs are input into the Large Scale Language Architecture (LLAMA) 3 model to obtain the real description text.

[0016] Optionally, obtaining a plant disease image representation description dataset includes:

[0017] Collecting plant disease images and labeling the plant disease images by category;

[0018] Data processing is performed on the labeled plant disease images to obtain a plant disease image representation description data set.

[0019] Optionally, performing data processing on the labeled plant disease image to obtain a plant disease image representation description data set includes:

[0020] Performing target image removal processing on the labeled plant disease image and correcting the corresponding disease type;

[0021] Based on the corrected plant disease image, generating label information using a deep learning algorithm;

[0022] A description template is constructed according to the label information, and a plant disease image representation description dataset is obtained based on the description template.

[0023] Optionally, based on the plant disease image, obtaining the original text includes:

[0024] Inputting the plant disease image into a second pre-trained model using a frozen image encoder and a large language model to guide the language image, to obtain the original text, wherein the second pre-trained model is composed of a visual encoder, a query generator, and a language decoder;

[0025] The visual encoder is used to extract high-dimensional features of the image;

[0026] The query generator is used to generate a joint multimodal representation based on the high-dimensional features;

[0027] The language decoder is used to generate a text description.

[0028] Optionally, based on the representation description dataset, obtaining supporting text includes:

[0029] Using a text encoder of a contrastive language-image pre-training model, respectively encoding the plant disease image representation description dataset and the plant disease image;

[0030] The similarity between the encoded representation description data set and the plant disease image is calculated, and based on the calculation result, the target text is screened to obtain the supporting text.

[0031] Optionally, constructing a dependency structure directed graph based on the supporting text includes:

[0032] Using the spaCy library to perform grammatical analysis on the original text and the supporting text to obtain the dependency structure;

[0033] A directed edge is added between two nodes in the dependency structure, and an attribute is added to the directed edge to obtain the dependency structure directed graph.

[0034] Optionally, obtaining key concept pairs according to the dependency structure directed graph includes:

[0035] Performing concept filtering on the dependency structure directed graph to obtain concept types;

[0036] Calculating a similarity score between the concept type and the plant disease image using a contrastive language-image pre-trained model;

[0037] Screening the concepts according to the similarity scores to obtain the optimal key concepts;

[0038] The optimal key concepts are mapped to generate the key concept pairs.

[0039] Optionally, the large language LLAMA3 model is trained using a cross entropy loss function.

[0040] Compared with the prior art, the present invention has the following advantages and technical effects:

[0041] The present invention solves the problem that the traditional method is limited by excessive reliance on a single crop, resulting in insufficient generalization ability of the model when facing other crops and difficulty in adapting to the diversified crop disease identification needs. The present invention is based on the image description model of zero-sample learning, which ensures the generalization ability of the model on different crop diseases, can effectively handle unseen crop disease types, and avoids overfitting of a single crop data set. In addition, the present invention incorporates a concept introduction correction module into the architecture, which corrects the original text generated by other methods. This module not only supplements the key details that may be omitted in the description generation process, but also specifically introduces professional terms in the field of plant pathology, so that the generated description not only contains the basic characteristics of the disease, but also covers professional information such as the specific morphology and development degree of the disease, so that the generated description is more in line with the field needs and enhances the accuracy and professionalism of the description. Through these improvements, the present invention realizes a method for describing plant disease images with good generalization and detailed information. This will help to more accurately identify and describe plant diseases and provide more reliable support for agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 It is a flow chart of a method for generating a zero-sample plant disease image representation description based on concept enhancement according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of obtaining a data set according to an embodiment of the present invention;

[0045] Figure 3 is a diagram showing an example of a data set according to an embodiment of the present invention;

[0046] Figure 4 It is a schematic diagram of the overall framework of an embodiment of the present invention. DETAILED DESCRIPTION

[0047] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] This paper proposes a method for generating zero-shot plant disease image representation descriptions based on concept enhancement. Figure 1 As shown, the specific steps include:

[0050] Obtain plant disease images;

[0051] Based on plant disease images, obtain plant disease image representation description dataset and original text;

[0052] Based on the representation description dataset, obtain supporting text;

[0053] Based on the supporting text and the original text, a dependency structure directed graph is constructed, wherein the dependency structure directed graph is used to indicate the corresponding dependency relationship between the supporting text and the original text;

[0054] Obtain key concept pairs based on the dependency structure directed graph;

[0055] The obtained key concept pairs are input into the Large Scale Language Architecture (LLAMA) 3 model to obtain the real description text.

[0056] Specifically, the method in this embodiment is described in detail:

[0057] S1 dataset construction: collect plant images with diseases, store them in different folders according to species and disease categories, generate labels through automatic annotation methods and conduct manual review;

[0058] S2 Support Text Acquisition: Use CLIP's text encoder to encode the support set, calculate the similarity with the input image, and select the text with the highest score as the support text;

[0059] S3 concept graph construction: Use the spaCy library to perform grammatical analysis on the original text and supporting text, obtain the dependency structure, and build a directed graph;

[0060] S4 concept fusion optimization: filter and retain the "amod" and "compound" relationships, use the CLIP model to calculate the similarity between concepts and images, and extract key concept pairs;

[0061] S5 Sentence Reorganization: Fine-tune the LLAMA3 model, take the key concept mapping results and the real description text as input and output, and optimize the model through the cross entropy loss function.

[0062] Furthermore, obtaining a plant disease image representation description dataset includes:

[0063] Collect plant disease images and label them by category;

[0064] Data processing is performed on the labeled plant disease images to obtain a plant disease image representation description dataset.

[0065] Furthermore, the labeled plant disease images are processed to obtain a plant disease image representation description data set including:

[0066] removing the labeled plant disease image and correcting the labeled plant disease image removal process;

[0067] Based on the corrected plant disease images, deep learning algorithms are used to generate label information;

[0068] A description template is constructed according to the label information, and based on the description template, a plant disease image representation description dataset is obtained.

[0069] Specifically, the acquisition of plant disease image representation description dataset is described in detail:

[0070] Step 1.1 Data acquisition: Collect data sets on the Internet and scattered plant disease images, and record the category corresponding to each image.

[0071] Step 1.2 Data preprocessing: Process the diseases obtained from different data sources, remove damaged and blurred images through image processing technology, and correct the disease type corresponding to each image.

[0072] Step 1.3 Generate labels: Use deep learning algorithms to generate labels for each image that contain disease representation information such as color, texture, location, and distribution of disease symptoms.

[0073] Step 1.4 Label processing: Since the machine-generated labels were incorrect, manual review was performed and two additional labels were manually generated for the images: “leaf / plant color” and “leaf / plant shape”.

[0074] Step 1.4: Build the template: Design the description template according to the scientific method. Make sure each description template contains all the label information.

[0075] Step 1.5 Description generation: In order to ensure the repeatability and consistency of random selection, the present invention sets the random seed to 42. In this way, the same random number sequence will be generated each time the program is run, and it is also ensured that each tag information can randomly generate a unique description according to the preset template style.

[0076] Step 1.6: Build the dataset: organize the images and corresponding descriptions and divide them into training set, support set and test set in the ratio of 7:2:1.

[0077] Furthermore, based on the plant disease image, obtaining the original text includes:

[0078] Input the plant disease image into the BLIP-2 model to obtain the original text, where the BLIP-2 model consists of a visual encoder, a query generator, and a language decoder;

[0079] Visual encoder, used to extract high-dimensional features of images;

[0080] A query generator, used to generate joint multimodal representations based on high-dimensional features;

[0081] A language decoder to generate text descriptions.

[0082] Furthermore, based on the representation description dataset, the supporting text is obtained including:

[0083] The text encoder of the CLIP model is used to encode the plant disease image representation description dataset and plant disease images respectively;

[0084] Calculate the similarity between the encoded representation description dataset and the plant disease image, and obtain the supporting text based on the calculation results.

[0085] Specifically, the acquisition of supporting text is described in detail:

[0086] Step 2.1 Build the support database: Use CLIP's text encoder to encode the support set in the dataset produced in step 1.

[0087] Step 2.2 Similarity calculation: Encode the image using CLIP and calculate the similarity score with the image in the support database obtained in step 2.1.

[0088] Step 2.3 Obtain supporting texts: Filter the first few texts with the highest scores as supporting texts;

[0089] Furthermore, based on the supporting text, constructing a dependency structure directed graph includes:

[0090] Use the spaCy library to perform grammatical analysis on the original text and supporting text to obtain the dependency structure;

[0091] Add a directed edge between two nodes in the dependency structure, add the attribute of the directed edge, and obtain the directed graph of the dependency structure.

[0092] Specifically, obtaining the directed graph of the dependency structure is described in detail:

[0093] Step 3.1 Dependency structure acquisition: Use the spaCy library to perform grammatical analysis on the original text and supporting text to obtain the dependency structure.

[0094] Step 3.2 Concept graph construction: For each pair of dependency relationships, add a directed edge between the corresponding two nodes. The direction of this edge is from the dominant word to the subordinate word, and the specific type of the dependency relationship is attached as the attribute of the edge, so that the analyzed grammar is constructed as a directed graph.

[0095] Furthermore, according to the dependency structure directed graph, key concept pairs are obtained including:

[0096] Perform concept filtering on the dependency structure directed graph to obtain concept types;

[0097] The CLIP model was used to calculate the similarity scores between concept types and plant disease images;

[0098] Filter concepts based on similarity scores to obtain the best key concepts;

[0099] Map the optimal key concepts and generate key concept pairs.

[0100] Specifically, the key concept pairs are elaborated in detail:

[0101] Step 4.1 Concept filtering: When constructing the concept map, the present invention retains all dependency structures, including multiple relationships such as root (sentence center word), nsubj (noun subject), dobj (direct object), etc. After analysis, it was found that the two relationships "amod" (adjective modification) and "compound" (compound word) have the greatest impact on the quality of plant disease description generation. Therefore, the present invention performs concept filtering and only retains the relationships of "amod" and "compound" types.

[0102] Step 4.2 Similarity calculation: Use the CLIP model to calculate the similarity score between the concept and the image.

[0103] Step 4.3 Key concept extraction: Rank the concepts based on similarity scores and select the highest scoring ones.

[0104] Step 4.4 Key concept mapping: Map the selected relations back to the modified nouns to generate key concept pairs.

[0105] Furthermore, the LLAMA3 model is trained using the cross entropy loss function.

[0106] Specifically, fine-tune the model: use the key concept mapping results obtained in step 4 and the actual description text as the input and output of the model to fine-tune the model, calculate the cross entropy loss, and obtain the model with the best reconstruction result after multiple cycles of iteration.

[0107] A method for generating zero-sample plant disease image representation description based on concept enhancement proposed in this embodiment is described in detail:

[0108] First, the present invention uses a self-made dataset for training and testing, which contains 20,943 images from PDDD, PlantDOC and PlantVillage, covering 60 crop types and more than 300 disease characteristics. In order to better meet the research objectives of the present invention, the present invention classifies and organizes the images according to crop type and disease manifestation, and stores them in named folders using appropriate formats. The method of making the dataset is shown in the attached figure. Figure 3 As shown. The present invention uses a deep learning model to generate labels for each image, detailing the color, texture, location, and distribution of disease symptoms. To ensure accuracy, these automatically generated labels are reviewed and corrected by domain experts, who also add two additional labels: "leaf / plant color" and "leaf / plant shape." Following these steps, each image is associated with eight specific labels. The present invention then invites domain experts to create descriptive text for each type of plant disease image, combining these eight labels. This process produced ten descriptive templates that integrated optimized and enhanced labels to generate detailed text descriptions for each picture. A total of 20,943 descriptions were created. (As shown in the attached Figure 2 Finally, the 20,943 image-text pairs were divided into training set, support set and test set in a ratio of 7:2:1 to support model training and testing, thereby enhancing the model's ability to identify emerging diseases in practical applications. (As shown in the attached Figure 3 (shown)

[0109] The present invention develops a framework based on the existing zero-shot image caption model (see attached Figure 4As shown in Figure 2), it can generalize to multiple crop diseases and effectively handle unseen disease types, thus preventing overfitting of specific crop datasets. To improve the model's ability to capture details, the present invention introduces three key components: Supporting Text Acquisition (STA), which uses CLIP to retrieve some of the most relevant texts to the image from the augmented database as supporting text; Concept Graph Builder (CGC), which utilizes spaCy to extract complex data relationships from the text and then converts these relationships into a graphical representation for subsequent processing; and Concept Injection Refinement (CIR) module, a custom component used to supplement missing details and correct inaccuracies while incorporating professional plant pathology terminology. This dual approach ensures the comprehensiveness, accuracy, and domain relevance of the description, while improving plant disease diagnosis by leveraging text-dependent analysis and graph-based representation. In addition, considering the computational cost of large-scale model training, the present invention optimizes the design of the CIR module to effectively integrate domain-specific knowledge without significantly increasing the computational overhead. This design strategy not only reduces the overall training cost, but also ensures that the model can understand and utilize domain-specific professional vocabulary, thereby improving the quality and efficiency of description generation.

[0110] Step 1 Data preprocessing: This paper uses CLIP to preprocess the validation set of the self-made dataset to enhance its richness in visual concept representation. The preprocessed validation set serves as a support database and contains visually relevant sentences related to key detail information. These sentences are crucial to remove the illusion of no training cases and reduce knowledge forgetting in pure text training cases. The whole process can be expressed as:

[0111]

[0112] SD=T'

[0113] Among them, T represents the support set in the data set, and E T Refers to the process of encoding text using the text encoder in the CLIP model. The formula represents that for each text t in T i , calculate the text t i The preprocessed text vector is stored in the support database SD.

[0114] Step 2: In order to accurately retrieve text descriptions closely related to the input image, key concepts are extracted from them and injected into the zero-shot description model to ensure that the description is accurate and detailed and avoid "description illusion". Specifically, given an image I, the present invention uses CLIP to evaluate image-text similarity and determine the relevant descriptions of the Top-N images to ensure the description quality and avoid wasting resources. Finally, the most relevant description is used as the supporting text to prepare for subsequent operations.

[0115] I'=E I (I i )

[0116] S ( I,d ) = CLIP(I',v j )

[0117] D topN =argsord d∈D S(I,v j )[1:N]

[0118] Among them, I i represents the current image, and E I It refers to the process of using the image encoder in the CLIP model to encode the image I. The present invention uses the CLIP mechanism to calculate the similarity score between the image and the text, and takes the top N items with the highest scores as supporting texts.

[0119] Step 3: Get the original text. Use the pre-trained BLIP-2 to get the original text that needs to be corrected. BLIP-2 combines language and visual information and can perform well in image description generation tasks. The model is pre-trained on large-scale image-text data and learns rich semantic representations and image features. It consists of a visual encoder, a query generator, and a language decoder. The visual encoder is based on a powerful visual backbone network (such as ViT or SwinTransformer) to extract high-dimensional feature representations of the image. The query generator is a Transformer model that processes the input text query and fuses it with the image features generated by the visual encoder through a self-attention mechanism to generate a joint multimodal representation. The language decoder is based on a powerful language model (such as OPT or T5) and generates a coherent and accurate text description word by word based on the multimodal representation generated by Q-Former.

[0120] T ori =BLIP_2(I i )

[0121] T ori Represents the use of BLIP2 to obtain the original text.

[0122] Step 4 Dependency Structure Acquisition: The dependency structure is extracted from the original text description and supporting materials using the general dependency analysis tool provided by spaCy. It can identify grammatical associations between words, such as subject-predicate agreement, the modification effect of adjectives on nouns, and noun-noun combinations, and convert these associations into a series of word pairs, each of which consists of a dominant word and a corresponding subordinate word.

[0123] W(T)={ω1 ,ω 2 ,…,ω n}

[0124] R(T)={Dep(ω i ,ω j ,r)|ω i ,ω j ∈W(T), r∈R}

[0125] Among them, W(T) is for a given text T, using spaCy's dependency analysis tool to parse the text, extract all the words, and analyze the dependency relationship between each word. R refers to the set of all possible dependency types, and then all dependencies are collected to form a dependency set.

[0126] Step 5 The present invention also designs a visualization component to verify the interpretability of the model. The component uses graph visualization technology to generate a visual representation of the graph and highlight the relationship between nodes. In addition, the visual display can be customized to highlight the key elements of the diagram, such as the most important nodes or the closest relationships. For each pair of dependencies, a directed edge is added between the corresponding two nodes. The direction of this edge is from the dominant word to the subordinate word, and the specific type of the dependency is attached as an attribute of the edge. This graph is a visual representation of the semantic structure of the text.

[0127] Step 6 To further improve the quality of the description, the present invention introduces a concept fusion optimization (CIR) module based on CGC, which focuses on adding missing details, correcting inaccuracies in the description, and integrating specialized plant pathology terms. Specifically, the present invention first inputs the reference text extracted from the model and database into spaCy for dependency parsing. The process includes three main steps:

[0128] Step 6.1: Relationship filtering, CIR first analyzes the relationships extracted by CGC. Through empirical testing, the present invention finds that 'amod' (adjective modification) and 'compound' (noun compound) relationships are particularly informative for attribute optimization. Therefore, CIR filters the extracted relationships and only retains relationships of type 'amod' and 'compound', as they are more likely to provide accurate description attributes.

[0129] R filter ={r∈R(T)|type(r}∈{'amod','compound'}

[0130] Step 6.2: Similarity Calculation,CIR computes the similarity score between each filtered relation and the corresponding image using the CLIP (Contrastive Language-Image Pre-training) model. CLIP is particularly effective in this case because it is trained to align textual and visual representations and is able to accurately assess the relevance of each attribute to the image. CIR ranks the relations based on the similarity scores and selects those with the highest scores to be included in the description. This step ensures that the attributes included in the description are not only grammatically correct but also semantically relevant to the image.

[0131] S(r,I i )=CLIP(r,I i )

[0132]

[0133] R select ={r∈R filter |S r is among the top k socres}

[0134] Among them, S(r,I i ) refers to the similarity score between the calculated dependency relationship and the image, and only the top k relationship groups with higher scores are retained as key concepts to guide subsequent operations.

[0135] Step 6.3: Mapping to key concepts: Finally, CIR maps the selected relations back to the nouns (objects) they modify, generating a set of key concept pairs. Based on the grammatical structure of the text and the visual content of the image, these concept pairs represent the most salient and relevant attributes of the image.

[0136] C key ={mapToConcepts(r)|r∈R select}

[0137] According to the dependency graph constructed in step 5, the key concepts are mapped back to the nouns they modify to generate key concept pairs.

[0138] Step 7 In order to reassemble the key concepts extracted from the image into a semantically rich, fluent and visually relevant title, the present invention uses a pre-trained large language model LLAMA3. LLAMA3 adopts a decoder-only Transformer architecture and uses an innovative rotational position encoding (RoPE) scheme instead of traditional absolute or relative position embedding. This position embedding method uses a rotation matrix to encode the absolute position and directly integrates the relative position information into the self-attention operation, thereby showing significant extrapolation advantages in long text tasks. Since the pre-trained LLAMA3 model lacks expertise in the field of plants, the present invention needs to fine-tune it first. It is a unimodal model, so the input is set to the key concepts extracted in step 6, and the output is the true description text of the corresponding image. The loss function uses cross entropy loss, and the model with the best reconstruction result is obtained after multiple cycles of iteration.

[0139]

[0140]

[0141]

[0142]

[0143] In the inference phase, key concepts are first extracted from the image, and then these concepts are fed into LLAMA3 in a specific format. The model then generates descriptive text associated with these concepts.

[0144] Through the above steps, the present invention realizes a zero-sample plant disease image representation description generation method based on concept enhancement, which effectively alleviates the problem of single description generation that is prevalent in current smart agriculture practice by using concept enhancement technology, and significantly improves the interpretability of disease identification and diagnosis. This innovative framework provides strong support for multimodal disease identification and precise drug use recommendations in modern smart agriculture, significantly improves the overall diagnosis and management efficiency, and makes important contributions to the sustainable development and efficient management of modern agriculture.

[0145] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A zero-shot plant disease image representation description generation method based on concept enhancement, characterized in that: include: Obtain plant disease images; Based on the plant disease image, obtaining a plant disease image representation description dataset and original text; Based on the representation description dataset, obtaining supporting text; Based on the supporting text and the original text, construct a dependency structure directed graph, wherein the dependency structure directed graph is used to indicate the corresponding dependency relationship between the supporting text and the original text; According to the dependency structure directed graph, obtaining key concept pairs; The obtained key concept pairs are input into the Large Scale Language Architecture (LLAMA) 3 model to obtain the real description text.

2. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 1, characterized in that: Obtaining plant disease image representation description datasets includes: Collecting plant disease images and labeling the plant disease images by category; Data processing is performed on the labeled plant disease images to obtain a plant disease image representation description data set.

3. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 2, characterized in that: Performing data processing on the labeled plant disease images to obtain a plant disease image representation description data set includes: Performing target image removal processing on the labeled plant disease image and correcting the corresponding disease type; Based on the corrected plant disease image, generating label information using a deep learning algorithm; A description template is constructed according to the label information, and a plant disease image representation description dataset is obtained based on the description template.

4. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 1, characterized in that: Based on the plant disease image, obtaining the original text includes: Inputting the plant disease image into a pre-trained model using a frozen image encoder and a large language model to guide a language image, to obtain the original text, wherein the pre-trained model is composed of a visual encoder, a query generator, and a language decoder; The visual encoder is used to extract high-dimensional features of the image; The query generator is used to generate a joint multimodal representation based on the high-dimensional features; The language decoder is used to generate a text description.

5. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 1, characterized in that: Based on the representation description dataset, obtaining supporting text includes: Using a text encoder of a contrastive language-image pre-training model, respectively encoding the plant disease image representation description dataset and the plant disease image; The similarity between the encoded representation description data set and the plant disease image is calculated, and based on the calculation result, the target text is screened to obtain the supporting text.

6. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 1, characterized in that: Based on the supporting text, constructing a dependency structure directed graph includes: Using the spaCy library to perform grammatical analysis on the original text and the supporting text to obtain the dependency structure; A directed edge is added between two nodes in the dependency structure, and an attribute is added to the directed edge to obtain the dependency structure directed graph.

7. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 1, characterized in that: According to the dependency structure directed graph, obtaining key concept pairs includes: Performing concept filtering on the dependency structure directed graph to obtain concept types; Calculating a similarity score between the concept type and the plant disease image using a contrastive language-image pre-trained model; Screening the concepts according to the similarity scores to obtain the optimal key concepts; The optimal key concepts are mapped to generate the key concept pairs.

8. The method for generating zero-shot plant disease image representation description based on concept enhancement according to claim 1, characterized in that: The large language LLAMA3 model is trained using a cross entropy loss function.

Images