Whole-crop explainable disease and pest diagnosis method and system based on multi-modal large model

Abstract

The invention provides a whole-crop explainable disease and pest diagnosis method and system based on a multi-mode large model. The method comprises the following steps: constructing a knowledge extraction model of a two-way cross attention mechanism based on relation guidance, extracting structured knowledge from authoritative agricultural data, and constructing a pest and disease knowledge enhancement database to dynamically retrieve prior knowledge of a target crop; a hierarchical image processing strategy is adopted, and global, local and target area multi-level feature information is extracted from an input image; inputting and priori knowledge are integrated into a comprehensive diagnosis instruction, and a thinking chain guiding module is introduced to guide a large model to carry out multi-step reasoning according to a reasoning path; and performing unified reasoning by using the multi-modal large model, and outputting a disease and pest diagnosis result and a diagnosis basis thereof. The method does not need manual marking of multi-modal data or retraining, can realize efficient diagnosis of whole crop diseases and insect pests under the condition that the multi-modal data and computing resources are limited, has low cost, strong generalization ability and high interpretability, and is suitable for large-scale agricultural production practice.

Description

Technical Field

[0001] This invention relates to the field of intelligent diagnosis technology for agricultural pests and diseases, and in particular to a method and system for interpretable diagnosis of pests and diseases across all crops based on a multimodal large model. Background Technology

[0002] Pest and disease control plays a crucial role in agricultural production, directly impacting crop yield and quality. With global climate change and the development of pesticide resistance, traditional pest and disease control methods face increasing challenges. Therefore, leveraging intelligent algorithms to assist in pest and disease diagnosis and control has become an important direction for modern agricultural development. Especially driven by computer vision and artificial intelligence technologies, agricultural pest and disease diagnosis has gradually evolved from relying on traditional expert experience and simple statistical analysis to automated methods based on artificial intelligence.

[0003] In recent years, computer vision technology has made significant progress in the application of pest and disease identification. Through the automatic feature extraction and multi-level representation learning capabilities of deep neural networks, it can effectively identify and classify crop pests and diseases, significantly improving diagnostic efficiency and accuracy. However, these methods generally rely on large amounts of labeled data and perform poorly when faced with unseen crops or a small number of samples. Furthermore, as "black box" models, they lack transparent decision-making processes, cannot provide reliable pathological reasoning, and cannot meet the personalized diagnostic needs of farmers.

[0004] Large language models (such as the GPT series and BERT) have achieved remarkable results in multiple fields due to their superior semantic understanding and reasoning capabilities, providing strong technical support and broad application prospects for agricultural pest and disease diagnosis and management. However, the direct application of general-purpose models in vertical agricultural fields remains limited, mainly because agricultural tasks typically have highly specialized requirements and lack targeted data support. To improve the performance of large language models in specialized fields such as plant disease management, many studies have further constructed specialized large models for agricultural diseases. Due to the lack of cross-modal crop pest and disease datasets, and the relative ease of obtaining high-quality large-scale text data, many studies have constructed agricultural knowledge large language models using text data such as books, web pages, and documents. However, training solely based on text data cannot fully capture the visual features of pests and diseases, limiting the model's performance in actual disease diagnosis. Based on this, multimodal large models are gradually becoming a development trend. One type of research reduces the dependence on multimodal datasets by performing reasoning on images and text separately in independent branches and then merging them, but its fusion of visual and linguistic features is relatively shallow, making it difficult to achieve deep semantic alignment across modalities. Another approach attempts to construct end-to-end multimodal large models for crop disease and pest diagnosis, but the crop and disease coverage of these models remains limited, failing to achieve broad adaptation across all crops. Furthermore, image processing methods often employ fixed-resolution compression, leading to the loss of crucial details such as lesions, thus affecting the model's accuracy in recognizing minute symptoms.

[0005] In summary, there are four key limitations in the current field of agricultural pest and disease diagnosis: (1) Existing deep learning models generally lack interpretability and transparency in the decision-making process, making it difficult to provide farmers with reliable pathological reasoning and limiting the widespread application of models in actual agricultural production. (2) General large models do not perform well in the vertical field of agriculture, and the single-modal language large model methods derived from this lack the ability to capture visual features. (3) Existing multimodal large models have insufficient cross-modal collaborative capabilities, and the fusion of visual and linguistic features is relatively shallow, making it difficult to achieve deep semantic alignment between cross-modalities. (4) Very few end-to-end multimodal large models focus on specific crops and cannot cover all crops.

[0006] Therefore, there is an urgent need for a method that can achieve interpretable diagnosis of diseases and pests across all crops without the need for manual sample labeling or model retraining, even when multimodal data is limited and computational resources are scarce. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems in the related art.

[0008] This invention proposes a method for interpretable disease and pest diagnosis of the whole crop based on a multimodal large model, which can perform high-precision diagnosis of the whole crop and explain the rationale and thought process for disease and pest diagnosis.

[0009] Another objective of this invention is to propose a whole-crop interpretable disease and pest diagnosis system based on a multimodal large model.

[0010] To achieve the above objectives, this invention proposes a method for interpretable disease and pest diagnosis of all crops based on a multimodal large model, comprising: S1. Design a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism, and use the joint knowledge extraction model to process the collected multi-source authoritative agricultural data to extract structured triple knowledge related to pests and diseases, so as to construct an enhanced knowledge database of crop pests and diseases. S2 performs hierarchical image processing on the input high-resolution crop image to generate a global thumbnail, local sub-images, and a semantically guided target image, and integrates the image data to obtain hierarchical image information. S3 introduces a thought chain guidance module and integrates it with prior knowledge retrieved from hierarchical image information, text descriptions, and crop pest and disease knowledge enhancement databases to construct a comprehensive instruction and guide the multimodal large model to carry out multi-step reasoning according to the reasoning path. S4. Input the comprehensive instructions into the multimodal large model for reasoning, and output the disease and pest diagnosis results and the corresponding interpretable diagnostic basis.

[0011] The whole-crop interpretable disease and pest diagnosis method based on a multimodal large model according to embodiments of the present invention may also have the following additional technical features: In one embodiment of the present invention, S1 includes: S11. Use OCR technology or document parsing tools to extract text information from multi-source authoritative agricultural data, and perform data cleaning and structuring to extract core agricultural pest and disease information for subsequent knowledge extraction. S12 combines ontology construction methods with the thesaurus system of the agricultural domain, designs a knowledge graph ontology of diseases and pests based on the disease triangle principle, defines concept types including crop types, pathogens, symptom characteristics and their semantic relationships, in order to construct a semantically complete domain knowledge ontology model. S13, Based on the semantically complete domain knowledge ontology model, construct a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism to identify and extract structured triples related to diseases and pests from the core agricultural disease and pest information. S14, the structured triples are stored in a graph database to construct a knowledge-enhanced database for crop disease and pest diagnosis tasks.

[0012] In one embodiment of the present invention, constructing the joint knowledge extraction model based on the relation-guided bidirectional cross-attention mechanism includes: Knowledge extraction training set construction: Integrate public knowledge graphs in the agricultural field and self-collected crop diagnosis question and answer data, combine manual annotation to annotate named entities and their structured triple relationships in the text, and use BIO tagging method and standard triple format to generate knowledge extraction training corpus; Text semantic encoding: Input the natural language text sequence in the knowledge extraction training corpus into the RoBERTa pre-trained language model to obtain a representation vector containing global semantic context; Entity boundary and type identification: Two independent Sigmoid classifiers are used to identify whether each position of the representation vector containing the global semantic context is the start or end point of an entity, and candidate entity fragments are identified; the start and end position vectors of each entity fragment are extracted and concatenated, and the entity type is determined by a Softmax classifier; Relationship-guided bidirectional cross-attention modeling: A relationship-guided bidirectional cross-attention mechanism is designed to construct attention paths that guide the subject to the object and the object to the subject, respectively, to enhance the semantic interaction and direction awareness between entity pairs. In path one, the subject entity is used as the query and the text is used as the key. Relationship-aware attention weights are generated by combining relation type embedding to update the object representation. In path two, the object entity is used as the query to update the subject representation in reverse. The updated entity representations are fused through residual connections and concatenated with the relation vectors to obtain the joint representation of the entity pairs. Relationship classification and triplet recognition: The joint representation of entity pairs is input into a multilayer perceptron, and the existence of a certain relationship is determined by the activation function; if the predicted probability is higher than a set threshold, a structured triplet containing entity type information is generated. Multi-task joint optimization and output: A multi-task learning approach is adopted to jointly train entity boundary recognition, entity type classification and relation classification tasks, define corresponding loss functions for each task, and perform joint optimization through weighted summation to achieve end-to-end shared learning of model parameters; Output a list of structured knowledge triples, each triple containing a subject, relation, object, and corresponding entity type label, which can be used to directly construct the node and edge structure of the agricultural knowledge graph.

[0013] In one embodiment of the present invention, the design is based on a relation-guided bidirectional cross-attention mechanism, where the subject guides the object and the object guides the subject, respectively, to enhance semantic interaction between entities and update the joint representation of entity pairs, including: Define each pair of candidate entities and ,in, For entities The starting position index, For entities The end position index; the concatenated vector representation of the main entity features are... The concatenated vector of the object entity is represented as The candidate relationship types between subject and object are: Construct the following bidirectional attention path: Path 1: Using the subject entity as the query and the entire text as the key, and introducing relation type embedding as a semantic guide, construct a relation type-aware attention distribution, and then update the object entity representation:

[0014] in, The distribution of the subject's attention to the object; The entire sentence context representation output by the encoding layer, where For the number of tokens, To hide dimensions; The query vector projection matrix is ​​used to represent the subject. Mapped to the query space; Guide the projection matrix for relations, used to embed relation types The mapping is used as a guiding vector, enabling the subject vector to better focus on the object region based on the relation type; This is the key vector projection matrix, used to represent the entire sequence. The mapping is done in a key vector space; after attention is guided, the representation of the entire sequence is weighted and aggregated to obtain the updated representation of the object entity. :

[0015] in, This is the projection matrix of the value vector, used to generate the weighted representation; Path 2: Use the object entity as the query, the subject direction as the key value, add the relation type vector as a guide, and perform reverse information update on the subject representation;

[0016]

[0017] in, The distribution of attention of the object to the subject; The entire sentence context representation output by the encoding layer, where For the number of tokens, To hide dimensions; The query vector projection matrix is ​​used to represent the object. Mapped to the query space; Guide the projection matrix for relations, used to embed relation types The mapping is used as a guiding vector, allowing the object vector to guide the subject region in reverse according to the relation type; This is the key vector projection matrix, used to represent the entire sequence. Mapped to a key vector space; This represents the updated main entity. The projection matrix of the value vector; Residual join fusion: The updated subject-object representations are added to their original vector residuals, and then concatenated with the relation vectors to obtain a joint representation of the entity pairs.

[0018] The final entity pair is represented as: .

[0019] In one embodiment of the present invention, the hierarchical image processing of the input high-resolution crop image includes: S21, dynamically segment the high-resolution image into multiple local sub-images, and generate a global thumbnail based on image scaling; S22, based on semantic guidance, extract high-probability disease areas from the image to obtain the target image.

[0020] In one embodiment of the present invention, a high-resolution image is dynamically segmented into multiple local sub-images, and a global thumbnail is generated based on image scaling, including: A high-resolution image is preset as the original image, with a size denoted as [size not specified]. The image is cut into multiple segments. The image patch should be the correct size, ensuring that the aspect ratio of the cut image remains unchanged; the calculation steps are as follows: The number of image blocks cut vertically and horizontally are respectively:

[0021]

[0022] The total number of image blocks is:

[0023] Calculate the pixel width that needs to be padded: if ,or Then, zero-padding is applied to the right and bottom sides of the original image:

[0024]

[0025] in, , These are the length and width of the original input image, respectively; The size of the image blocks to be divided into This is the pixel width that needs to be padded on the right. The height of the image to be padded at the bottom is denoted as:

[0026]

[0027] Determine the coordinates of the final cut image block to obtain a local sub-image: The completed image Above, define the first Image patches, of which , The coordinates of the top left corner are:

[0028]

[0029] The corresponding cropping area is:

[0030] in Further, a set of local subgraphs is obtained. ; if Scale the original image to Global thumbnail .

[0031] In one embodiment of the present invention, the step of extracting high-probability disease regions from an image based on semantic guidance to obtain a target image includes: A semantically guided object detection algorithm is used to extract disease-related regions based on keywords, and only the detection boxes with the highest confidence are retained as the object map. Finally, image integration is performed to obtain hierarchical image information.

[0032] In one embodiment of the present invention, a thought chain guidance module is introduced and integrated with prior knowledge retrieved from a hierarchical image information, text description, and crop pest and disease knowledge enhancement database to construct a comprehensive instruction. This instruction guides a multimodal large model to perform multi-step reasoning according to the reasoning path, including: S31 introduces a thought chain guidance module, which includes a pre-set reasoning process template of: health assessment → anomaly location → symptom identification → pest and disease attribution → diagnosis summary. This template is used to guide the multimodal large model to generate intermediate reasoning processes and final conclusions in a logical order during the diagnosis task. S32, design a system instruction template with expert role setting and task prompt structure, clarify the model behavior role, and stipulate that its reasoning method must follow the step-by-step thinking chain logic provided by the thinking chain guidance module; S33, integrate user input information, including crop type, text description and image information, and standardize and organize it into multimodal input content; wherein, the image information includes global thumbnail, local image patch and disease target image; S34. Use Cypher query language to dynamically retrieve pest and disease information related to the input crop type from the knowledge-enhanced database, including disease name, symptom description, disease pattern, control methods and graphic examples, and organize them into structured prior knowledge fragments. S35 integrates thought chain guidance instructions, system instructions, user input, and knowledge background to generate structured comprehensive diagnostic instructions, guiding the model to carry out multi-step semantic reasoning and diagnostic results according to the preset thought chain.

[0033] To achieve the above objectives, another aspect of the present invention proposes a whole-crop interpretable disease and pest diagnosis system based on a multimodal large model, comprising: The knowledge-enhanced database retrieval module is used to design a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism, and to use the joint knowledge extraction model to process the collected multi-source authoritative agricultural data to extract structured triple knowledge related to pests and diseases, so as to construct a knowledge-enhanced database of crop pests and diseases. The hierarchical image processing module is used to perform hierarchical image processing on the input high-resolution crop image, generate a global thumbnail, local sub-images and semantically guided target images, and integrate image data to obtain hierarchical image information. The integrated instruction generation module is used to introduce the thought chain guidance module and integrate it with the prior knowledge retrieved from the hierarchical image information, text description and crop pest and disease knowledge enhancement database to construct integrated instructions and guide the multimodal large model to carry out multi-step reasoning according to the reasoning path. The multimodal reasoning output module is used to input the comprehensive instructions into the multimodal large model for reasoning and output the disease and pest diagnosis results and the corresponding interpretable diagnostic basis.

[0034] The present invention relates to a method and system for interpretable disease and pest diagnosis of all crops based on a multimodal large model. By constructing a knowledge-enhanced database and utilizing the small-sample capability of the multimodal large model, it eliminates the need for model retraining, thus reducing time and computational costs. It addresses the problem of scarce multimodal data, covers disease and pest diagnosis across all crops, and has broad applicability. Furthermore, the introduction of a thought chain guidance module (CoT instruction) enables the model to perform diagnostic reasoning according to a logical sequence and output detailed diagnostic evidence, enhancing the reliability of the diagnostic results.

[0035] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a method for interpretable disease and pest diagnosis of all crops based on a multimodal large model, provided in an embodiment of the present invention; Figure 2 A data logic diagram of a whole-crop interpretable disease and pest diagnosis method based on a multimodal large model provided in an embodiment of the present invention; Figure 3 This is a flowchart of the knowledge graph ontology construction process provided in an embodiment of the present invention; Figure 4 This is a flowchart of hierarchical image processing provided in an embodiment of the present invention; Figure 5 This is a structural diagram of a whole-crop interpretable disease and pest diagnosis system based on a multimodal large model, according to an embodiment of the present invention. Detailed Implementation

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] The following description, with reference to the accompanying drawings, describes a method and system for diagnosing diseases and pests in crops based on a multimodal large model, which can be interpreted across the entire crop.

[0040] Figure 1 This is a flowchart of a method for interpretable whole-crop disease and pest diagnosis based on a multimodal large model according to an embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, it includes: S1. Design a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism, and use the joint knowledge extraction model to process the collected multi-source authoritative agricultural data to extract structured triple knowledge related to pests and diseases, so as to construct an enhanced knowledge database of crop pests and diseases.

[0041] Specifically, the core of the dynamic knowledge-enhanced database constructed in this invention lies in extracting structured triples from multi-source authoritative agricultural data and employing a relationship-guided bidirectional cross-attention mechanism to achieve joint recognition of entities and relationships, thereby constructing a crop disease and pest knowledge graph with real-time scalability. This step plays a crucial role in knowledge support and reasoning guidance within the overall technical solution, and is the foundational module for achieving interpretable disease and pest diagnosis across all crops.

[0042] At the technical implementation level, this invention first integrates multi-source data from authoritative agricultural atlases, plant protection station documents, CABIPlantwise, and other sources, covering structured, semi-structured, and unstructured text. After extracting the text content using OCR and document parsing tools, data cleaning and structuring are performed using BIO annotation and structured triple representation. Subsequently, a knowledge graph ontology is designed based on the disease triangle principle, defining entity categories such as "crop type," "disease type," "affected part," and "symptom description," and establishing relationship types such as "caused by," "manifested symptoms," and "control methods," ensuring the semantic integrity and adaptability of the knowledge graph to the agricultural field.

[0043] In entity and relation recognition, this invention employs a relation-guided bidirectional cross-attention mechanism, using RoBERTa for text encoding to obtain contextual semantic representations. For each candidate entity pair, bidirectional attention paths are constructed for subject→object and object→subject, respectively, introducing relation type embeddings as semantic guides to enhance the model's ability to model asymmetric dependencies. Through residual connections and an MLP classifier, joint representation and triple recognition of entity pairs and relations are achieved, ultimately outputting a list of structured knowledge triples.

[0044] At the parameter level, the entity boundary recognition loss adopts binary cross-entropy loss, and the relationship classification loss adopts multi-label sigmoid cross-entropy loss. Through multi-task joint optimization, loss weight coefficients (such as λ1=0.3, λ2=0.4, λ3=0.3) are set for end-to-end training to improve the model's recognition accuracy of complex semantic relationships.

[0045] In application scenarios, this knowledge-enhanced database can access new crop or disease data in real time and dynamically retrieve relevant disease knowledge through the Cypher query language, providing the contextual information needed for few-shot learning of multimodal large models, thereby significantly improving the model's diagnostic capabilities and interpretability in scenarios involving crops or diseases not previously seen.

[0046] The technical value of this step lies in the fact that by constructing and dynamically expanding a structured knowledge graph, it solves the problems of data sparsity and insufficient model generalization ability in the agricultural field. At the same time, it improves the accuracy of entity and relation recognition by leveraging a relationship-guided bidirectional cross-attention mechanism, providing high-quality and interpretable knowledge support for subsequent multimodal reasoning.

[0047] Furthermore, such as Figure 3 As shown, step S1 includes: S11. Use OCR technology or document parsing tools to extract text information from multi-source authoritative agricultural data, and perform data cleaning and structuring to extract core agricultural pest and disease information for subsequent knowledge extraction.

[0048] Understandably, to improve the professionalism and reliability of the general-purpose large-scale model in crop disease diagnosis, a scalable knowledge-enhanced database is constructed to provide the large-scale model with domain-specific knowledge for few-shot learning. This knowledge-enhanced database integrates authoritative crop disease and pest data from multiple sources, ensuring its richness, systematicity, and professionalism. Because this data is decoupled from the diagnostic large-scale model, it possesses real-time scalability, allowing for the real-time inclusion of new crops or diseases.

[0049] Specifically, data was collected from multiple sources (including structured, semi-structured, and unstructured formats) such as authoritative atlases, agricultural websites, and official guidance documents. Examples include the third edition of *Chinese Crop Diseases and Pests*, *Color Atlas of Diseases and Pests of Chinese Grain Crops, Economic Crops, and Medicinal Plants*, *New Color Atlas of Vegetable Diseases and Pests Control*, agricultural knowledge websites, and informational leaflets from the Beijing Plant Protection Station. OCR technology or document parsing tools were used to extract text, followed by data cleaning and structuring to extract the crop name, crop category, pathogen information, disease / pest type, disease symptoms, disease progression, control methods, and images of each crop disease / pest.

[0050] S12 combines ontology construction methods with the thesaurus system of the agricultural domain, and designs a knowledge graph ontology of diseases and pests based on the disease triangle principle. It defines concept types and their semantic relationships, including crop types, pathogens, symptom characteristics, etc., in order to construct a semantically complete domain knowledge ontology model.

[0051] Specifically, combining existing ontology construction methods—the seven-step method and the thesaurus-based method—the ontology layer design is guided by the core theory in plant diseases: the disease triangle principle (environment, host, pathogen). This includes referencing model-level data sources such as the *Thesaurus of Agricultural Sciences*, designing an ontology outline model using Xmind based on the disease triangle principle, and defining entity classes, hierarchical structures, and attributes using Protégé. Based on this, an evaluation process combining self-review and review by agronomic experts is employed, and the model is formally described using the OWL language. The final definition includes concept categories such as "crop type," "disease / pest type," "pathogen," "disease / pest name," "affected part," "phenotypic symptoms," "epidemic environmental conditions," and "control strategies." Relationships such as "caused by," "manifested symptoms," "impact," "epidemic environment," "control methods," "disease pattern," and "onset" are also defined to construct a standard terminology system.

[0052] S13. Based on a semantically complete domain knowledge ontology model, a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism is constructed to identify and extract structured triples related to diseases and pests from the core agricultural disease and pest information.

[0053] Specifically, a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism is constructed. This model performs entity recognition on the text data processed in the first step, while simultaneously extracting structured knowledge such as triples (crop, infected, diseased), thereby achieving instantiation modeling of the crop disease ontology. The construction and reasoning steps of the joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism in this embodiment of the invention are as follows: S131, Knowledge Extraction Training Set Construction: Integrating publicly available knowledge graph datasets in the agricultural field (such as AgriKG) and self-collected crop diagnostic question-and-answer data, and combining manual annotation, each text is annotated with named entities (including entity boundaries and types) and their corresponding structured relation triples. The annotation results adopt standard BIO notation and structured triple representation, adapting to the requirements of model input and supervised training.

[0054] S132, Text Semantic Encoding: Input the natural language text sequence from the knowledge extraction training corpus into the pre-trained language model RoBERTa to obtain the semantic encoding for each position. Corresponding context vector The overall structure represents a matrix. ,in For text length, This is to hide the dimension. The encoder has a multi-layer Transformer structure, which can capture global semantic dependencies and provide a representational foundation for subsequent entity and relationship recognition.

[0055] S133, Entity Boundary and Type Recognition: Further identify all named entities in the text and simultaneously determine their entity category. Two independent Sigmoid binary classifiers are used to determine whether each position is the start or end point of an entity.

[0056] in, and The probability that each position is the start or end point of an entity; For learnable parameters, This is the Sigmoid function. It then retrieves all candidate entity spans.

[0057] in, Represents the entity boundary pairs for each element, indicating that an entity is defined from the first... Starting with the first token, up to the second... The process ends with a token. For each identified entity span, its start and end position vectors are extracted and concatenated to obtain a result. Contextual semantics of entities at boundaries The type category of the entity is output using the Softmax classifier:

[0058] in, Predict probability distributions for entity types. Weight matrix of trainable entity type classifier.

[0059] S134, Relationship-Guided Bidirectional Cross-Attention Modeling: This model aims to enhance the semantic interaction between entity pairs through bidirectional paths and relational semantic guidance mechanisms. It can significantly enhance the model's ability to express asymmetric dependencies and semantic direction differences between entities, and is suitable for semantically diverse structural relationship modeling tasks such as disease-symptom and crop-disease relationships.

[0060] For each pair of candidate entities and ,in, For entities The starting position index, For entities The end position index. The concatenated vector representation of the main entity has the following characteristics. The concatenated vector of the object entity is represented as The candidate relation types between subject and object are: Construct the following bidirectional attention path: Path 1 (Head→Tail): Using the subject entity as the query and the entire text as the key / value pair, and introducing relation type embedding as a semantic guide, a relation type-aware attention distribution is constructed to update the object entity representation:

[0061] in, The distribution of the subject's attention to the object; The entire sentence context representation output by the encoding layer, where For the number of tokens, To hide dimensions; The query vector projection matrix is ​​used to represent the subject. Mapped to the query space; Guide the projection matrix for relations, used to embed relation types The mapping is used as a guiding vector, enabling the subject vector to better focus on the object region based on the relation type; This is the key vector projection matrix, used to represent the entire sequence. This is mapped to a key vector space. After attention is guided, the representation of the entire sequence is weighted and aggregated to obtain the updated representation of the object entity. :

[0062] in, This is the projection matrix of the value vector, used to generate a weighted representation.

[0063] Path 2 (Tail→Head): Symmetrically, the object entity is used as the query, the subject direction is used as the key value, and a relation type vector is also added as a guide to update the subject representation in reverse.

[0064]

[0065] in, The distribution of attention of the object to the subject; The entire sentence context representation output by the encoding layer, where For the number of tokens, To hide dimensions; The query vector projection matrix is ​​used to represent the object. Mapped to the query space; Guide the projection matrix for relations, used to embed relation types The mapping is used as a guiding vector, allowing the object vector to guide the subject region in reverse according to the relation type; This is the key vector projection matrix, used to represent the entire sequence. It is mapped to a key vector space. This represents the updated main entity. It is the projection matrix of the value vector.

[0066] Residual join fusion: The updated subject and object representations are added to their original vector residuals and concatenated with the relation vectors to obtain a joint representation of the entity pairs.

[0067]

[0068] The final entity pair is represented as:

[0069] S135, Relation Classification and Triple Recognition: The fused subject-object representation and relation are embedded into the multilayer perceptron. A linear transformation plus Sigmoid activation is used to determine whether the entity pair constitutes a valid triple under a certain relation. If the predicted value is higher than the threshold, a structured triple is generated, carrying the category label information of each entity.

[0070]

[0071] in, For entity pairs and Having a relationship The probability (confidence level). Let be a linear mapping matrix used to map entity pairs to relation existence scores. If If the result is positive, it is predicted to be a valid triplet.

[0072] S136, Multi-task joint training and optimization: The entire model is trained in a multi-task joint learning manner. The optimization objectives include entity boundary recognition loss, entity type classification loss and entity pair relationship classification loss. Through loss weighted fusion, end-to-end parameter sharing optimization is achieved, enabling the model to learn entity recognition and triple extraction tasks simultaneously within the same framework.

[0073] The entity boundary recognition loss is:

[0074] in, Let be the probability that the i-th token is predicted to be the start / end of an entity; This is the actual label, with a value of 0 or 1.

[0075] The entity type classification loss is:

[0076] in, Probability distribution of entity types; M represents the actual category label of the entity, and M is the number of entities identified.

[0077] The entity-to-relation classification loss is:

[0078] in, To predict the probability of triples; Let K be the true label indicating whether a triple exists, and K be the total number of entity-relation combinations. The three losses are combined and jointly optimized using a weighted approach:

[0079] in, , , This is the loss weighting coefficient.

[0080] S137, Output: The model outputs a set of structured knowledge triples. Each triple contains a subject, relation, object, and semantic category labels for each entity, which are used to directly construct the structure of knowledge graph nodes and edges.

[0081] S14, the structured triples are stored in a graph database to construct a knowledge-enhanced database for crop disease and pest diagnosis tasks.

[0082] Specifically, a graph database (such as Neo4j) is used for entity and relation storage and retrieval. The preceding steps have extracted entities and their structured triples from unstructured text data. This structured data is then imported in batches into the Neo4j database using the py2neo library in Python. The graph database is used to construct entity nodes and relation edges, forming a queryable, visualized, and reasonable graph structure that supports application requirements such as knowledge indexing, subgraph retrieval, and pattern matching. In practical use, Cypher statements are constructed to dynamically retrieve background knowledge related to the input crop. The graph database is then used to extract knowledge fragments related to the crop (such as typical symptoms, known pathogens, and common prevention and control measures). These results are used as contextual information input into a large language model to assist in generating more evidence-based and credible diagnostic conclusions.

[0083] S2 performs hierarchical image processing on the input high-resolution crop image to generate a global thumbnail, local sub-images, and a semantically guided target image, and integrates the image data to obtain hierarchical image information.

[0084] The hierarchical image processing method of this invention aims to solve the problem of detail loss caused by compression in high-resolution agricultural images during multimodal large-scale model inference, while enhancing the model's ability to focus on diseased areas, thereby improving the accuracy and interpretability of pest and disease diagnosis. This step achieves collaborative perception of image structure, texture, and diseased areas through a three-level visual information processing mechanism of "global-local-target".

[0085] Understandably, to enhance the model's fine-grained recognition ability of high-resolution agricultural images and improve the accurate focusing on diseased areas, a hierarchical image processing method is proposed. This method obtains hierarchical visual information in a "global-local-target" manner from both spatial resolution preservation and semantic target guidance, thereby improving the model's perception ability of complex disease images. First, addressing the problem of key information loss caused by high-resolution image compression, this invention employs a dynamic high-resolution strategy. The entire high-resolution image is dynamically divided into multiple small image patches according to the input size, while simultaneously generating a global overview map. The image patches provide rich texture and lesion features while preserving local details, and the thumbnails retain spatial structure and background information.

[0086] This method first dynamically segments the input high-resolution image (e.g., 2048×2048 or higher) to generate multiple local image patches (e.g., 256×256 or 512×512), and retains a global thumbnail (with the same size as the local image patches). The image patch generation employs a non-overlapping sliding window strategy to ensure the integrity of local details. If the original image size is not divisible by the target patch size, zero padding is applied to the right and bottom sides to maintain the image aspect ratio. The coordinates of each image patch are determined by a formula to ensure the accuracy of the cropped area. The global thumbnail is downsampled using a bilinear interpolation algorithm to preserve the overall structure of the crop and background information.

[0087] When using Vision Transformer (ViT) for feature extraction, the input image is typically divided into 16×16 patches, and self-attention is calculated based on the patch dimension. Taking a 2048×2048 image as an example, if the entire image is input directly without dynamic segmentation, the image will be divided into 16384 (128×128) patches, and the computational complexity of self-attention is O(log n). This invention dynamically divides an image into 64 local image blocks of 256×256 pixels. When calculating self-attention for these 64 local image blocks, only... This reduces computational complexity by approximately 64 times, significantly lowering the computational complexity of the self-attention mechanism. Furthermore, a Grounding DINO semantic-guided object detection module is introduced, setting box_threshold and text_threshold to 0.2, retaining only the detection box with the highest confidence level to ensure high-quality extraction of the target region.

[0088] This method is applicable to scenarios such as field pest and disease image acquisition, drone aerial image processing, and high-resolution images uploaded from mobile devices. By preserving global structure, local texture, and diseased areas, the model can achieve more accurate disease identification and localization under different modal inputs (such as image input only), and is especially suitable for disease types with small and sparsely distributed lesions, such as powdery mildew and rust.

[0089] The hierarchical image processing method of this invention effectively alleviates the computational bottleneck of high-resolution images in visual models, enabling the model to achieve efficient inference without losing key information. Through a semantically guided target detection module, the model can automatically focus on diseased areas, improving its ability to identify disease features. This step provides structured, multi-scale visual input for subsequent inference of large multimodal models, and is a key supporting module for achieving interpretable diagnosis of all crops and all diseases.

[0090] Furthermore, such as Figure 4 As shown, S2 involves performing hierarchical image processing on the input high-resolution crop image to generate a global thumbnail, local sub-images, and a semantically guided target image, and then integrating the image data to obtain hierarchical image information. This process may include the following sub-steps: S21, dynamically divide the high-resolution image into multiple local sub-images, and generate a global thumbnail based on image scaling.

[0091] Specifically, given a high-resolution image as the original image, with dimensions denoted as . The goal is to cut the image into multiple segments through image cropping. Image blocks of a certain size, ensuring that the aspect ratio of the cut image remains unchanged. S21 includes the following steps: S211, Calculate the number of image blocks cut vertically and horizontally: The number of image blocks cut vertically and horizontally are respectively:

[0092]

[0093] The total number of image blocks is:

[0094] S212, Calculate the pixel width that needs to be padded: if ,or Then, zero padding needs to be applied to the right and bottom sides of the original image.

[0095]

[0096] in, , These are the length and width of the original input image, respectively; The size of the image blocks to be divided into This is the pixel width that needs to be padded on the right. The height of the image to be padded at the bottom is denoted as:

[0097]

[0098] Determine the coordinates of the final cut image block to obtain a local sub-image: The completed image Above, define the first Image patches, of which , The coordinates of the top left corner are:

[0099]

[0100] The corresponding cropping area is:

[0101] in Further, a set of local subgraphs is obtained. .

[0102] if Scale the original image to Global thumbnail .

[0103] S22, based on semantic guidance, extract high-probability disease areas from the image to obtain the target image.

[0104] Specifically, to further enhance the model's ability to focus on targets from complex backgrounds, the hierarchical image processing method also introduces the Grounding DINO semantic-guided target detection module. Semantic keywords (such as "disease leaves" and "disease fruits") guide visual perception, focusing on and extracting key target regions related to diseases in the image. To ensure good semantic relevance and localization accuracy of the detection boxes, while avoiding the introduction of redundant interference regions, `box_threshold` and `text_threshold` are set to 0.2, retaining only the detection box with the highest confidence as the target image. This enables high-quality filtering and semantic focus of the target region. Text-guided visual preprocessing provides strong priors for subsequent cross-modal alignment.

[0105] Finally, image data integration is performed to obtain hierarchical image information, that is, the global thumbnail, local sub-images, and target image are all used as image input:

[0106] The embodiments of the present invention enable the model to achieve the synergistic fusion of global semantic perception, local detail preservation and key target focusing without sacrificing the original resolution. This supports the visual processing flow from scene-level understanding to pixel-level analysis, improves computational efficiency while ensuring information integrity, and effectively enhances the model's ability to recognize fine-grained symptoms such as tiny lesions.

[0107] The hierarchical image processing method used in this step is more computationally efficient than directly processing the entire original image, as it utilizes multi-level small image patches. Taking Vision Transformer (ViT) as an example, the input image is first divided into several patches, and global self-attention modeling is performed, with a computational complexity of O(n log n). ,in, Let H and W represent the image height and width, P be the patch size, and d be the embedding dimension. Therefore, high-resolution images (e.g., 2048×2048) result in a huge number of patches N in ViT, drastically increasing the computational cost and memory usage of the self-attention layer. After using hierarchical image processing for segmentation, the size of each image block is significantly reduced. Assuming the image is divided into k non-overlapping blocks, the number of patches per block is... The corresponding attention computation complexity is: The computational complexity is reduced to 1 / k times the original complexity, achieving near-linear speedup. Furthermore, since each tile is spatially independent and there are no cross-tile dependencies, feature extraction and inference operations can be performed in parallel on hardware such as GPUs or TPUs, further improving overall throughput.

[0108] The specific process of the Grounding DINO semantic-guided object detection module is as follows: Given the original input image and the keyword text to be identified, the word-level embeddings are first extracted using the frozen CLIP text encoder. Then, the SwinTransformer is used as the backbone to extract multi-scale features from the image. The feature maps of each layer are flattened into a patch sequence, and spatial location encoding and hierarchical embedding are added. Finally, the features of all scales are concatenated into a unified image feature sequence. The image feature sequence is then fed into a standard Transformer Encoder for multi-head self-attention modeling, capturing the contextual relationships between different regions in the image and outputting a semantic representation of the image. .

[0109] The decoder section consists of multiple Transformer layers with the same structure. Each decoder layer contains three sub-modules: multi-head self-attention, image cross-attention, and text cross-attention. Given the query vector of the k-th decoder as input... First, a multi-head self-attention mechanism is used to achieve communication between queries, avoiding duplicate target detection, and then the updated query is obtained: MHSA is a multi-head self-attention function. After image cross-attention, each query focuses on a specific region in the image, resulting in the updated query: MHCA is a multi-head cross-attention function, which is finally obtained after text cross-attention. Each decoder layer receives the query vector from the previous layer and outputs the optimized query. After L layers, the final decoder output is obtained. .

[0110] After decoding, the bounding box and text alignment score are further predicted: the i-th query is processed by the regression head MLP to predict a normalized bounding box. Used to represent the coordinates of the center point and the width and height. Then, the cosine similarity between each query and each text token is calculated. The similarity between the i-th query and the j-th text token is:

[0111] Pick The maximum value is used as the final match confidence score: .

[0112] In the final reasoning phase, the output of all queries is... Perform scoring and sorting; set a text matching score threshold for filtering; use NMS (non-maximum suppression) to remove overlapping boxes; output the final bounding box and corresponding text label.

[0113] S3 introduces a thought chain guidance module and integrates it with prior knowledge retrieved from hierarchical image information, text descriptions, and crop pest and disease knowledge enhancement databases to construct a comprehensive instruction that guides the multimodal large model to carry out multi-step reasoning according to the reasoning path.

[0114] Specifically, the "integrated instruction design" of this invention is one of the core links in realizing interpretable disease and pest diagnosis for all crops. Its technical implementation principle lies in the organic integration of multimodal input information and domain knowledge through structured Prompt design, thereby guiding the multimodal large model to reason according to the preset chain of thought (CoT) of "health judgment → anomaly location → symptom identification → disease and pest attribution → diagnosis summary", thereby improving the logic, interpretability and diagnostic accuracy of the model output.

[0115] In terms of specific operation, this step first structurally integrates the crop type (e.g., cucumber), text description (e.g., "yellow spots appear on the leaves"), and the hierarchical image information processed in step S2 (including global thumbnails, local image patches, and semantically guided target images). Simultaneously, using the Cypher query language of the knowledge graph, it retrieves crop-related pest and disease knowledge from the knowledge-enhanced database, including disease names, typical symptoms, disease patterns, and control strategies, and embeds this knowledge into the Prompt in a combined text and image format. For example, the system automatically retrieves and inserts text and image descriptions of all cucumber-related pests and diseases (e.g., "cucumber downy mildew" and "cucumber powdery mildew"), providing contextual support for the model.

[0116] The construction of the integrated instructions follows the standard Prompt engineering design principles, employing a modular input structure to ensure the model can clearly identify the boundaries and semantics of each modal input. Image information is processed through... Labels are used for annotation, and text descriptions are presented in the form of structured fields. In the CoT guidance module, each inference step has clear instruction statements, such as "Health judgment: Is there any abnormality in the crop in this image? If so, briefly explain", thereby guiding the model to output intermediate inference results step by step, forming a traceable diagnostic path.

[0117] This invention supports selectable input for each modality, and text descriptions and image data can be input individually or in combination, thereby ensuring the flexibility and adaptability of the instructions.

[0118] This step plays a crucial role in the overall technical solution, integrating multi-source heterogeneous data on the one hand, and activating the small-shot learning capability of the large model through structured prompt design on the other, enabling it to achieve high-precision, full-crop-coverage disease and pest diagnosis even in the absence of large-scale image and text training data. Furthermore, guided by CoT, the model output possesses a clear logical chain and diagnostic basis, significantly improving the system's interpretability and user experience, providing solid support for the practical application of agricultural intelligent diagnostic systems.

[0119] Furthermore, S3 may also include the following sub-steps: S31 introduces a thought chain guidance module, which includes a pre-set reasoning process template of: health assessment → anomaly location → symptom identification → pest and disease attribution → diagnosis summary. This is used to guide the multimodal large model to generate intermediate reasoning processes and final conclusions step by step in a logical order during the diagnosis task.

[0120] Understandably, this invention introduces a thought chain guidance module, which pre-sets a reasoning process template including health assessment → anomaly localization → symptom identification → pest and disease attribution → diagnostic summary. This module guides the multimodal large model to generate intermediate reasoning processes and final conclusions step by step in a logical order during diagnostic tasks. This step-by-step thought chain design ensures the logic and interpretability of the reasoning process, enabling the model to clearly demonstrate the reasoning basis for each step, thereby enhancing the credibility of the diagnostic results.

[0121] Specifically, to ensure that the multimodal large model can logically generate intermediate reasoning processes and final conclusions step by step in diagnostic tasks, this invention introduces a thought chain guidance module. This module pre-sets a reasoning process template containing five key steps: Health assessment: To make a preliminary assessment of the overall health status of the crop and determine if any abnormalities exist.

[0122] Anomaly location: If an anomaly is found, determine its specific location (e.g., leaves, stems, fruits, etc.).

[0123] Symptom identification: Identify specific abnormal symptoms (such as yellow spots, decay, wormholes, etc.).

[0124] Pest and disease attribution: Inferring possible causes of pests and diseases based on symptoms and location.

[0125] Diagnostic Summary: Based on the above information, a final diagnostic conclusion is drawn, and corresponding recommendations are provided.

[0126] This step-by-step thought process design ensures the logic and interpretability of the reasoning process, enabling the model to clearly demonstrate the reasoning basis for each step, thereby enhancing the credibility of the diagnostic results. For example, when diagnosing downy mildew in cucumbers, the model first assesses the overall health of the cucumber, then locates abnormal areas on the leaves, identifies the grayish-white mold layer and irregular lesions, and finally infers that it may be downy mildew, providing corresponding prevention and control suggestions.

[0127] S32, design a system instruction template with expert role setting and task prompt structure, clarify the model behavior role, and stipulate that its reasoning method must follow the step-by-step thinking chain logic provided by the thinking chain guidance module.

[0128] Understandably, designing a system instruction template with expert role settings and task prompts clarifies the model's behavioral roles and stipulates that its reasoning method must follow the step-by-step thought chain logic provided by the thought chain guidance module. The system instruction template integrates user input information with background knowledge through predefined structured fields, providing clear reasoning guidance for the model.

[0129] Here is a specific example: You are a well-trained agricultural pest and disease diagnosis expert. Based on the following input information and referring to the descriptions of relevant pests and diseases of this crop in the encyclopedia, please follow the thought chain of "health assessment → anomaly location → symptom identification → pest and disease attribution → diagnosis summary" to conduct a step-by-step diagnostic analysis, and finally output the diagnosis results and reasons.

[0130] By assigning this expert role, the model can more closely resemble professional practices, ensuring that the diagnostic process meets the professional standards for agricultural pest and disease diagnosis.

[0131] S33, integrate user input information, including crop type, text description and image information, and standardize and organize it into multimodal input content; wherein, the image information includes global thumbnail, local image patch and disease target image.

[0132] To ensure the model can handle multimodal input, this invention integrates user input information, including crop type, text description, and image information, and standardizes and organizes it into multimodal input content. Specifically: Crop type: Clearly identify the target crop for diagnosis, such as "cucumber".

[0133] Text description: A user's textual description of the crop's abnormal condition, such as "yellow spots appear on the leaves, gradually expanding into brown lesions with irregular edges."

[0134] Image information: via Tags are used for modal differentiation, enabling structured embedding of images and text. For example: The global thumbnail of the input image is<image1>Local image patch is <image2> ，……， <imagen>The target image of the disease is as follows:<imageN+1> .

[0135] Image information includes global thumbnails, local image patches, and disease target images. This image information provides the model with rich visual cues, which helps to identify pests and diseases more accurately.

[0136] S34. Use Cypher query language to dynamically retrieve pest and disease information related to the input crop type from the knowledge-enhanced database, including disease name, symptom description, disease pattern, control methods and graphic examples, and organize them into structured prior knowledge fragments.

[0137] To provide the model with real-time updated domain knowledge support, this invention uses the Cypher query language to dynamically retrieve pest and disease information related to the input crop type from a knowledge-enhanced database. This information includes disease names, symptom descriptions, disease patterns, control methods, and illustrated examples, and is organized into structured prior knowledge fragments. Specifically, it implements an agricultural pest and disease knowledge graph built on the Neo4j graph database, and uses the Cypher query language to achieve dynamic retrieval.

[0138] This information is then formatted into text blocks and embedded into the synthesis instructions as contextual knowledge support for multimodal large model inference.

[0139] The following are descriptions and illustrated examples of cucumber-related diseases: 1. Cucumber downy mildew: Symptoms include a grayish-white mold layer on the underside of leaves; lesions are polygonal or irregular in shape and often spread along the veins. (Illustration shown)<imageN+2> ; 2. Cucumber powdery mildew: Primarily infects the leaves, causing a white powdery substance to appear on the leaf surface, which can lead to leaf curling and shedding. (Illustration shown)<imageN+3> ... Through dynamic knowledge retrieval, the model can obtain information on pests and diseases related to the input crop, thereby improving the accuracy and reliability of diagnosis.

[0140] S35 integrates thought chain guidance instructions, system instructions, user input, and knowledge background to generate structured comprehensive diagnostic instructions, guiding the model to carry out multi-step semantic reasoning and diagnostic results according to the preset thought chain.

[0141] Specifically, to guide the model to conduct multi-step semantic reasoning and diagnostic results according to a preset thought chain, this invention integrates thought chain guidance instructions, system instructions, user input, and knowledge background to generate structured comprehensive diagnostic instructions. An example of a comprehensive instruction template is as follows: You are a trained agricultural pest and disease diagnosis expert. Based on the following input information and referring to the relevant pest and disease descriptions for this crop in the encyclopedia, please follow the thought process of "health assessment → anomaly location → symptom identification → pest and disease attribution → diagnosis summary" to conduct a step-by-step diagnostic analysis and finally output the diagnosis results and reasons.

[0142] [Crop Type]: Cucumber; [User Description]: Yellow spots appear on the leaves, gradually expanding into brown lesions with irregular edges.

[0143] [Image Information]: The global thumbnail of the input image is <image1>Local image patch is <image2> ，……， <imagen>The target image of the disease is as follows:<imageN+1> .

[0144] [Background Knowledge on Crop-Related Diseases and Pests]: The following are descriptions and illustrated examples of diseases related to cucumbers: 1. Cucumber downy mildew: Symptoms include a grayish-white mold layer on the underside of leaves; lesions are polygonal or irregular in shape and often spread along the veins. (Illustration shown)<imageN+2> ; 2. Cucumber powdery mildew: Primarily infects the leaves, causing a white powdery substance to appear on the leaf surface, which can lead to leaf curling and shedding. (Illustration shown)<imageN+3> ... [Reasoning Process]: Please think about and answer the following five questions in sequence: 1. Health Assessment: Are there any abnormalities in the crops in this image? If so, please briefly explain.

[0145] 2. Anomaly localization: Identify abnormal areas in the image and describe their characteristics (color, location, distribution).

[0146] 3. Symptom identification: Analyze the specific symptoms of abnormal areas and compare them with the symptoms recorded in the knowledge base.

[0147] 4. Pest and disease attribution: Based on textual and graphic information and knowledge base content, infer the most likely type of disease and provide a brief explanation.

[0148] 5. Diagnostic Summary and Interpretation: Summarize the above analysis results and output a diagnostic conclusion along with the supporting evidence.

[0149] Output format: Please output your diagnosis results in the following format: Diagnosis results: (Disease name or healthy condition); Reasoning: (Explain the symptom characteristics and diagnostic basis according to the above reasoning steps).

[0150] The above steps achieve the following technical effects: Cypher query language dynamically retrieves pest and disease information related to the input crop from the knowledge graph and embeds it into integrated instructions, providing the model with real-time updated domain knowledge support. Standardized user input integration and image information embedding ensure the model can process text and image information simultaneously, avoiding modality confusion. The thought chain guidance module and system instruction templates ensure the model reasons step-by-step according to a preset logical order, generating interpretable diagnostic conclusions.

[0151] By fusing domain knowledge with the semantics of user input, the model's reasoning consistency and diagnostic accuracy across different modal inputs are significantly improved. Through structured prompt design and detailed explanations of reasoning steps, scientifically based diagnostic results are provided to users, advancing the practical application of intelligent agricultural diagnostics.

[0152] S4. Input the comprehensive instructions into the multimodal large model for reasoning, and output the disease and pest diagnosis results and the corresponding interpretable diagnostic basis.

[0153] Specifically, the designed comprehensive prompt is input into the multimodal large model for inference, outputting the final diagnostic results and interpretable diagnostic evidence to the user. The inference process can be implemented by calling the API interface of a general large model (such as GPT-4o or Qwen-VL-Max), or by using a locally deployed open-source multimodal large model (such as InternVL-2.5). Upon receiving the comprehensive prompt, the model automatically parses the image and text information, encodes the images and text separately, and generates natural language diagnostic results and explanatory feedback based on the fused semantic representation. The specific process is as follows: Given crop image and their corresponding symptom descriptions or diagnostic instructions The model first processes the input image through hierarchical image processing in step S2 to generate a set of multi-scale image patches. This allows for the simultaneous capture of global contextual information and local disease details. These image patches are then fed into a frozen pre-trained visual encoder, where they are encoded into dense visual embedding representations, ensuring robust visual representations with good generalization capabilities.

[0154] Meanwhile, text input Word segmentation into discrete text embeddings To align modal differences between images and text, a trainable MLP-based projection module is introduced to map visual features into the same semantic space as the language tokens, resulting in aligned visual embeddings. This projection operation enables visual information to be incorporated into subsequent language processing in a semantically consistent manner.

[0155] Subsequently, the aligned visual embedding With text embedding Both are input into a large language model to achieve multimodal information fusion, reasoning, and generation. To help the language model distinguish between visual and textual modalities, visual tokens are specially... The text sequence is enclosed in tags and symbols. This design allows the model to perform unimodal or multimodal inference simultaneously.

[0156] The multimodal large model employs an autoregressive generative approach to complete the diagnosis and inference tasks related to crop diseases and pests. Specifically, at each time step, the model predicts the probability of the next token based on previously generated sequences and the current multimodal input. Its training objective is to maximize the conditional likelihood at each time step, specifically defined as follows:

[0157] in, Indicates the length of the output sequence. For the first A token generated at each time step. It is a sequence of historical tokens generated previously. This indicates the input text query (such as the comprehensive instruction mentioned in step S3). This represents the visual features extracted from crop images.

[0158] For example, the model output is: The cucumber crop is diagnosed with cucumber powdery mildew, based on the following diagnostic criteria: "A white powdery mildew layer" appears on the "upper surface of the leaves," along with "irregularly distributed patches," "the powdery layer is concentrated between the veins," "the powder is fine-textured and easily detached," and "slight yellowing and chlorosis appear on the leaf surface." These symptoms are highly consistent with the typical manifestations of cucumber powdery mildew. Cucumber powdery mildew primarily affects the leaves, initially forming small white powdery spots on the upper surface of the leaves, which then expand into irregular powdery mildew layers with blurred edges that can merge into larger patches. The areas covered by these lesions are often accompanied by reduced photosynthesis, resulting in chlorosis or yellowing, but the tissue does not immediately die, and the leaves retain a certain degree of integrity. Based on the combined information of leaf location, mildew characteristics, and color changes, a diagnosis of cucumber powdery mildew can be made.

[0159] In summary, the overall workflow of the model is as follows: The user inputs images and text descriptions. Images of diseases, such as downy mildew on cucumbers, are provided. Text descriptions, such as "leaf spots appeared during cucumber harvesting," are also provided. The input images are processed by a hierarchical image preprocessing module. Prior knowledge enhancement modules are used to obtain relevant pest and disease information for the crop. This information is then integrated to obtain a comprehensive prompt input multimodal large model. The model then uses inference to obtain diagnostic results and the basis for the inference.

[0160] The multimodal large model-based interpretable disease and pest diagnosis method for all crops in this invention can achieve high-precision disease and pest diagnosis for all crops and all diseases, and provide interpretable diagnostic basis, effectively overcoming the shortcomings of existing technologies in terms of data dependence, model generalization ability, cross-modal fusion and interpretability.

[0161] The beneficial effects of this invention are as follows: 1) This invention does not require retraining the model, thus having a low-cost advantage.

[0162] 2) Possesses continuous evolution capability: It can directly connect to any open-source or commercial multimodal large model with text and image understanding capabilities (such as InternVL, Qwen-VL, GPT-4o, etc.). As the performance of the base multimodal model continues to improve, the diagnostic capability of this invention can also be improved in sync, realizing a sustainable development capability of self-enhancement as the model evolves.

[0163] 3) Supports input of any modality, making it more suitable for real-world applications: It can support image single-modal input, text single-modal input, or image-text combination input, and can flexibly adapt to different inputs.

[0164] 4) Supports diagnosis of all crops and all diseases: It solves the problem of insufficient data in the current multimodal dataset of all crops. By constructing a knowledge enhancement module that is decoupled from the model and has scalability, and by making full use of the few-shot capability of the large model, it can achieve accurate diagnosis of all crops.

[0165] 5) It has interpretability and interactivity: The CoT instruction design is introduced to guide the large model to output reasoning explanations according to the thought chain.

[0166] 6) More focused on the core area of ​​the image, without damaging details and with higher accuracy: Construct a hierarchical image processing module to achieve multi-level visual information fusion of global, local and target, improve operating efficiency without reducing resolution, and at the same time focus more on the main body of the disease.

[0167] 7) Introducing a relation-aware bidirectional cross-attention mechanism improves knowledge extraction accuracy, thereby strengthening the knowledge enhancement module in the large model. By constructing bidirectional interaction paths from subject to object and from object to subject, it effectively captures the bidirectional dependencies between entities, exhibiting stronger reasoning robustness for semantically complex relationships and avoiding reliance solely on static context features. Simultaneously, it introduces relation type embedding as an attention guide, enabling dynamic control of entity representation by relational semantics, which helps distinguish ambiguous situations involving multiple relations.

[0168] To achieve the above embodiments, such as Figure 5 As shown, this embodiment also provides a whole-crop interpretable disease and pest diagnosis system 10 based on a multimodal large model, including: The knowledge-enhanced database retrieval module 100 is used to design a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism, and to use the joint knowledge extraction model to process the collected multi-source authoritative agricultural data, extract structured triple knowledge related to pests and diseases, so as to construct a knowledge-enhanced database of crop pests and diseases. The hierarchical image processing module 200 is used to perform hierarchical image processing on the input high-resolution crop image, generate a global thumbnail, local sub-images and semantically guided target images, and integrate image data to obtain hierarchical image information. The comprehensive instruction generation module 300 is used to introduce the thought chain guidance module and integrate it with the prior knowledge retrieved from the hierarchical image information, text description and crop pest and disease knowledge enhancement database to construct a comprehensive instruction and guide the multimodal large model to carry out multi-step reasoning according to the reasoning path. The multimodal reasoning output module 400 is used to input the comprehensive instructions into the multimodal large model for reasoning and output the disease and pest diagnosis results and the corresponding interpretable diagnostic basis.

[0169] This invention presents a multimodal large-scale model-based interpretable disease and pest diagnosis system for all crops, eliminating the need for model retraining. By constructing a knowledge-enhanced database and leveraging the limited-sample capability of the multimodal large-scale model, model retraining is unnecessary, reducing time and computational costs. It addresses the problem of scarce multimodal data, covering disease and pest diagnosis across all crops and possessing broad applicability. Furthermore, the introduction of a thought chain guidance module (CoT instruction) enables the model to perform diagnostic reasoning according to a logical sequence and output detailed diagnostic evidence, enhancing the credibility of the diagnostic results. A hierarchical image processing method is employed to preserve image details and focus on diseased areas, improving the accuracy of disease and pest identification. It supports single-modal image input, single-modal text input, or combined image and text input, flexibly adapting to different practical application scenarios. It can also directly connect to any open-source or commercial multimodal large-scale model with image and text understanding capabilities.

[0170] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0171] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.< / imagen> < / image2> < / imagen> < / image2>

Claims

1. A method for whole-crop disease and pest diagnosis based on a multimodal large model, characterized in that, include: S1. Design a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism, and use the joint knowledge extraction model to process the collected multi-source authoritative agricultural data to extract structured triple knowledge related to pests and diseases, so as to construct an enhanced knowledge database of crop pests and diseases. S2 performs hierarchical image processing on the input high-resolution crop image to generate a global thumbnail, local sub-images, and a semantically guided target image, and integrates the image data to obtain hierarchical image information. S3 introduces a thought chain guidance module and integrates it with prior knowledge retrieved from hierarchical image information, text descriptions, and crop pest and disease knowledge enhancement databases to construct a comprehensive instruction and guide the multimodal large model to carry out multi-step reasoning according to the reasoning path. S4. Input the comprehensive instructions into the multimodal large model for reasoning, and output the disease and pest diagnosis results and the corresponding interpretable diagnostic basis.

2. The method as described in claim 1, characterized in that, S1 includes: S11. Use OCR technology or document parsing tools to extract text information from multi-source authoritative agricultural data, and perform data cleaning and structuring to extract core agricultural pest and disease information for subsequent knowledge extraction. S12 combines ontology construction methods with the thesaurus system of the agricultural domain, designs a knowledge graph ontology of diseases and pests based on the disease triangle principle, defines concept types including crop types, pathogens, symptom characteristics and their semantic relationships, in order to construct a semantically complete domain knowledge ontology model. S13, Based on the semantically complete domain knowledge ontology model, construct a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism to identify and extract structured triples related to diseases and pests from the core agricultural disease and pest information. S14, the structured triples are stored in a graph database to construct a knowledge-enhanced database for crop disease and pest diagnosis tasks.

3. The method as described in claim 2, characterized in that, Constructing the joint knowledge extraction model based on the relationship-guided bidirectional cross-attention mechanism includes: Knowledge extraction training set construction: Integrate public knowledge graphs in the agricultural field and self-collected crop diagnosis question and answer data, combine manual annotation to annotate named entities and their structured triple relationships in the text, and use BIO tagging method and standard triple format to generate knowledge extraction training corpus; Text semantic encoding: Input the natural language text sequence in the knowledge extraction training corpus into the RoBERTa pre-trained language model to obtain a representation vector containing global semantic context; Entity boundary and type identification: Two independent Sigmoid classifiers are used to identify whether each position of the representation vector containing the global semantic context is the start or end point of an entity, and candidate entity fragments are identified; the start and end position vectors of each entity fragment are extracted and concatenated, and the entity type is determined by a Softmax classifier; Relationship-guided bidirectional cross-attention modeling: A relationship-guided bidirectional cross-attention mechanism is designed to construct attention paths that guide the subject to the object and the object to the subject, respectively, to enhance the semantic interaction and direction awareness between entity pairs. In path one, the subject entity is used as the query and the text is used as the key. Relationship-aware attention weights are generated by combining relation type embedding to update the object representation. In path two, the object entity is used as the query to update the subject representation in reverse. The updated entity representations are fused through residual connections and concatenated with the relation vectors to obtain the joint representation of the entity pairs. Relationship classification and triplet recognition: The joint representation of entity pairs is input into a multilayer perceptron, and the existence of a certain relationship is determined by the activation function; if the predicted probability is higher than a set threshold, a structured triplet containing entity type information is generated. Multi-task joint optimization and output: A multi-task learning approach is adopted to jointly train entity boundary recognition, entity type classification and relation classification tasks, define corresponding loss functions for each task, and perform joint optimization through weighted summation to achieve end-to-end shared learning of model parameters; Output a list of structured knowledge triples, each triple containing a subject, relation, object, and corresponding entity type label, which can be used to directly construct the node and edge structure of the agricultural knowledge graph.

4. The method as described in claim 3, characterized in that, The design is based on a relation-guided bidirectional cross-attention mechanism, where the subject guides the object and the object guides the subject, respectively, to enhance semantic interaction between entities and update the joint representation of entity pairs, including: Define each pair of candidate entities and ,in, For entities The starting position index, For entities The end position index; the concatenated vector representation of the main entity features are... The concatenated vector of the object entity is represented as The candidate relationship types between subject and object are: Construct the following bidirectional attention path: Path 1: Using the subject entity as the query and the entire text as the key, and introducing relation type embedding as a semantic guide, construct a relation type-aware attention distribution, and then update the object entity representation: in, The distribution of the subject's attention to the object; The entire sentence context representation output by the encoding layer, where For the number of tokens, To hide dimensions; The query vector projection matrix is ​​used to represent the subject. Mapped to the query space; Guide the projection matrix for relations, used to embed relation types The mapping is used as a guiding vector, enabling the subject vector to better focus on the object region based on the relation type; This is the key vector projection matrix, used to represent the entire sequence. The mapping is done in a key vector space; after attention is guided, the representation of the entire sequence is weighted and aggregated to obtain the updated representation of the object entity. : in, This is the projection matrix of the value vector, used to generate the weighted representation; Path 2: Use the object entity as the query, the subject direction as the key value, add the relation type vector as a guide, and perform reverse information update on the subject representation; in, The distribution of attention of the object to the subject; The entire sentence context representation output by the encoding layer, where For the number of tokens, To hide dimensions; The query vector projection matrix is ​​used to represent the object. Mapped to the query space; Guide the projection matrix for relations, used to embed relation types The mapping is used as a guiding vector, allowing the object vector to guide the subject region in reverse according to the relation type; This is the key vector projection matrix, used to represent the entire sequence. Mapped to a key vector space; This represents the updated main entity. The projection matrix of the value vector; Residual join fusion: The updated subject-object representations are added to their original vector residuals, and then concatenated with the relation vectors to obtain a joint representation of the entity pairs. The final entity pair is represented as: 。 5. The method as described in claim 1, characterized in that, The hierarchical image processing of the input high-resolution crop image includes: S21, dynamically segment the high-resolution image into multiple local sub-images, and generate a global thumbnail based on image scaling; S22, based on semantic guidance, extract high-probability disease areas from the image to obtain the target image.

6. The method as described in claim 5, characterized in that, The high-resolution image is dynamically segmented into multiple local sub-images, and a global thumbnail is generated based on image scaling, including: A high-resolution image is preset as the original image, with a size denoted as [size not specified]. The image is cut into multiple segments. The image patch should be the correct size, ensuring that the aspect ratio of the cut image remains unchanged; the calculation steps are as follows: The number of image blocks cut vertically and horizontally are respectively: The total number of image blocks is: Calculate the pixel width that needs to be padded: if ,or Then, zero-padding is applied to the right and bottom sides of the original image: in, , These are the length and width of the original input image, respectively; The size of the image blocks to be divided into This is the pixel width that needs to be padded on the right. The height of the image to be padded at the bottom is denoted as: Determine the coordinates of the final cut image block to obtain a local sub-image: The completed image Above, define the first Image patches, of which , The coordinates of the top left corner are: The corresponding cropping area is: in Further, a set of local subgraphs is obtained. ; if Scale the original image to Global thumbnail .

7. The method as described in claim 6, characterized in that, The step of extracting high-probability disease regions from an image based on semantic guidance to obtain a target image includes: A semantically guided object detection algorithm is used to extract disease-related regions based on keywords, and only the detection boxes with the highest confidence are retained as the object map. Finally, image integration is performed to obtain hierarchical image information. 。 8. The method as described in claim 1, characterized in that, A thought chain guidance module is introduced and integrated with prior knowledge retrieved from a hierarchical image information, text description, and crop pest and disease knowledge enhancement database to construct a comprehensive instruction. This instruction guides the multimodal large model to perform multi-step reasoning according to the reasoning path, including: S31 introduces a thought chain guidance module, which includes a pre-set reasoning process template of: health assessment → anomaly location → symptom identification → pest and disease attribution → diagnosis summary. This template is used to guide the multimodal large model to generate intermediate reasoning processes and final conclusions in a logical order during the diagnosis task. S32, design a system instruction template with expert role setting and task prompt structure, clarify the model behavior role, and stipulate that its reasoning method must follow the step-by-step thinking chain logic provided by the thinking chain guidance module; S33, integrate user input information, including crop type, text description and image information, and standardize and organize it into multimodal input content; wherein, the image information includes global thumbnail, local image patch and disease target image; S34. Use Cypher query language to dynamically retrieve pest and disease information related to the input crop type from the knowledge-enhanced database, including disease name, symptom description, disease pattern, control methods and graphic examples, and organize them into structured prior knowledge fragments. S35 integrates thought chain guidance instructions, system instructions, user input, and knowledge background to generate structured comprehensive diagnostic instructions, guiding the model to carry out multi-step semantic reasoning and diagnostic results according to the preset thought chain.

9. A whole-crop interpretable disease and pest diagnosis system based on a multimodal large model, characterized in that, include: The knowledge-enhanced database retrieval module is used to design a joint knowledge extraction model based on a relation-guided bidirectional cross-attention mechanism, and to use the joint knowledge extraction model to process the collected multi-source authoritative agricultural data to extract structured triple knowledge related to pests and diseases, so as to construct a knowledge-enhanced database of crop pests and diseases. The hierarchical image processing module is used to perform hierarchical image processing on the input high-resolution crop image, generate a global thumbnail, local sub-images and semantically guided target images, and integrate image data to obtain hierarchical image information. The integrated instruction generation module is used to introduce the thought chain guidance module and integrate it with the prior knowledge retrieved from the hierarchical image information, text description and crop pest and disease knowledge enhancement database to construct integrated instructions and guide the multimodal large model to carry out multi-step reasoning according to the reasoning path. The multimodal reasoning output module is used to input the comprehensive instructions into the multimodal large model for reasoning and output the disease and pest diagnosis results and the corresponding interpretable diagnostic basis.

Images