Crop disease diagnosis system based on lightweight visual models and retrieval enhancement

Abstract

This invention relates to the field of crop disease diagnosis, specifically a crop disease diagnosis system based on a lightweight visual model and retrieval-enhanced generation. By organically coordinating a lightweight visual recognition network with a retrieval-enhanced generation system, it achieves high-precision disease diagnosis and professional solution generation with extremely low computational overhead. It can be smoothly deployed on mobile terminals or embedded devices. Visual tags serve as hard-constraint retrieval prefixes, fundamentally eliminating cross-crop misdiagnosis and significantly lowering the threshold for field deployment. By constructing a multi-factor comprehensive scoring and dynamic routing decision mechanism, it achieves quantitative evaluation of retrieval quality and adaptive switching of generation paths. Vector similarity, rearrangement relevance scores, and domain reward scores are weighted and integrated to take into account semantic breadth, logical depth, and factual accuracy. Entity coverage and keyword overlap are used for dual threshold determination to accurately identify knowledge blind spots and trigger backtracking generation, effectively suppressing the illusion of large language models and ensuring the professionalism and factual accuracy of diagnostic results.

Description

Technical Field

[0001] This invention relates to the field of crop disease diagnosis, specifically a crop disease diagnosis system based on lightweight visual models and enhanced retrieval generation. Background Technology

[0002] In the field of smart agriculture, timely and accurate diagnosis of crop diseases and provision of effective prevention and control guidance are crucial to ensuring crop yield and agricultural efficiency. With the application of artificial intelligence technology, disease diagnosis methods based on visual recognition and natural language processing have gradually gained attention, but there are still obvious limitations in actual promotion.

[0003] Currently, high-precision visual models typically have large parameter scales and high computational requirements, making them difficult to run efficiently on mobile devices or embedded terminals with limited storage and computing power, thus restricting their real-time deployment in the field. Furthermore, traditional image classification models generally only output disease categories, failing to provide detailed symptom descriptions and prevention suggestions, resulting in insufficient practicality. Directly using general-purpose large language models to generate agricultural question-and-answer content is also prone to factual discrepancies in key information such as pesticide names and dosages due to insufficient alignment between training data and agricultural expertise, posing a risk to guidance.

[0004] Some existing methods that combine multimodal retrieval and generation attempt to introduce external knowledge to improve the accuracy of responses. However, these systems often rely on large multimodal networks, resulting in high overall computational load and large response latency, making them unsuitable for edge environments. Furthermore, general retrieval mechanisms lack fine-grained guidance based on agricultural characteristics when distinguishing between cases that appear similar but belong to different crops or diseases, which may affect the professionalism and reliability of the diagnosis.

[0005] Therefore, there is an urgent need to develop a lightweight intelligent disease diagnosis solution that can operate stably in resource-constrained environments, and is characterized by accurate identification, professional knowledge, and rapid response, so as to realize a reliable technical closed loop from image perception to the generation of prevention and control suggestions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a crop disease diagnosis system based on lightweight visual models and enhanced retrieval, solving the problem of the lack of a lightweight intelligent disease diagnosis solution that can operate stably in resource-constrained environments and is characterized by accurate identification, professional knowledge, and rapid response.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a crop disease diagnosis system based on a lightweight visual model and enhanced retrieval generation, the system comprising:

[0008] A lightweight visual recognition network is used to extract visual labels containing crop species and disease types from input images;

[0009] The adaptive retrieval enhancement module is used to divide agricultural books into multiple document blocks encapsulated with metadata. It retrieves multiple candidate document blocks from the agricultural knowledge base based on visual tags and metadata. It reorganizes visual tags, user original questions, and system instructions according to preset logic to construct combined tags. It calculates a comprehensive score that reflects the degree of matching between the combined tags and each candidate document block and the retrieval quality. It also uses the parent document retrieval mechanism and the candidate document block with the highest comprehensive score to backtrack the context.

[0010] The agricultural big language model is obtained by supervised fine-tuning of a pre-constructed big language model based on an agricultural knowledge base.

[0011] The routing decision module is used to selectively input visual tags, the user's original question, and the context obtained from backtracking, or only visual tags and the user's original question, into the fine-tuned agricultural big language model to generate diagnostic results, based on the retrieval quality of candidate document blocks.

[0012] Preferably, the specific steps for constructing the agricultural big language model are as follows:

[0013] Select a base large language model for training and use BF16 precision loading;

[0014] Acquire question-and-answer training data in the agricultural field and serialize it into a standard dialogue format according to a predefined dialogue template format;

[0015] The standard dialogue format is input into the basic large language model, and the attention mechanism of the Transformer layer of the basic large language model is fine-tuned through fine-tuning the model.

[0016] Gradient accumulation is used during training, and the fine-tuned model weights with a small size are saved after training. They are then merged into the basic large language model during inference to construct the agricultural large language model.

[0017] Preferably, the specific steps for dividing the agricultural books into blocks are as follows:

[0018] Agricultural books are analyzed to identify their hierarchical heading structure and paragraph boundaries, resulting in analyzed text paragraphs.

[0019] Based on predefined agricultural keyword rules, the parsed text paragraphs are identified in terms of content type, and corresponding knowledge type tags are injected; wherein, the knowledge type tags include at least a first type tag for identifying disease symptoms and a second type tag for identifying prevention and control measures;

[0020] Divide the text paragraph into multiple document blocks of preset length, and dynamically anchor the split point of the document block to the nearest paragraph end character or period;

[0021] Each document block is encapsulated with metadata, which includes at least the name of the corresponding agricultural book, the name of the corresponding crop, the name of the disease, the knowledge type tag, and the parent document ID.

[0022] Preferably, the comprehensive score is a weighted sum of vector similarity score, rearrangement relevance score, and domain reward score;

[0023] The vector similarity score is used to vectorize the combined tags and candidate document blocks, and to calculate the cosine similarity between them.

[0024] The rearranged relevance score is obtained by concatenating the combined label with the candidate document block and inputting it into the cross-encoder model.

[0025] The domain reward score is based on the matching of the visual tags with the crop name and disease name in the candidate document block metadata. According to the preset rules, a preset score is assigned to each successfully matched item, and the sum of the scores of all matched items is calculated.

[0026] Preferably, when calculating the weighted sum of the vector similarity score, the rearranged relevance score, and the domain reward score, the domain reward score has a greater weight than the rearranged relevance score, and the rearranged relevance score has a greater weight than the vector similarity score.

[0027] Preferably, the specific steps of the parent document retrieval mechanism to trace back the context are as follows:

[0028] The candidate document block with the highest overall score is marked as the core block;

[0029] Map back to the complete chapter of the original agricultural book based on the parent document ID in the metadata encapsulated on the core block;

[0030] Centered on the core block, extract multiple adjacent document blocks before and after it in the complete chapter of the original agricultural book to obtain the context containing the core block.

[0031] Preferably, the specific steps for obtaining the diagnostic results are as follows:

[0032] Calculate the entity coverage and keyword overlap between the user's original question and the context obtained from the backtracking;

[0033] If the entity coverage or keyword overlap is lower than the corresponding threshold, the visual labels and the user's original question will be input into the fine-tuned agricultural big language model to generate diagnostic results.

[0034] Otherwise, the agricultural big data language model is fine-tuned by inputting visual labels, the user's original question, and the context obtained from backtracking to generate diagnostic results.

[0035] Preferably, the specific steps for calculating the entity coverage rate are as follows:

[0036] Extract entities from the user's original question to obtain the user entity set, and extract entities from the context obtained through backtracking to obtain the context entity set;

[0037] Calculate the ratio of the intersection of the user entity set and the context entity set to the user entity set to obtain the entity coverage.

[0038] Preferably, the keyword overlap is obtained based on the BM25 algorithm and incorporates agricultural feature reward calculation. The specific calculation steps are as follows:

[0039] The user's original question and the context obtained from the backtracking are vectorized, and the cosine similarity between the two is calculated to obtain a semantic relevance score;

[0040] The BM25 algorithm is used to calculate the degree of matching between the user's original question and the context obtained from the backtracking, in order to obtain a keyword matching score.

[0041] The metadata corresponding to the backtracked context is matched with the entities representing crop names and disease names in the user's original question. Each successfully matched item is assigned a preset score according to preset rules. The sum of the scores of all matched items is calculated, and a bonus score of a preset value is added for each successful match to obtain agricultural feature bonus items.

[0042] The keyword overlap is obtained by calculating the weighted sum of semantic relevance score, keyword matching score, and agricultural feature reward item.

[0043] Compared with existing technologies, this invention provides a crop disease diagnosis system based on a lightweight visual model and enhanced retrieval, which has the following beneficial effects:

[0044] 1. This invention achieves high-precision disease diagnosis and professional solution generation with extremely low computational overhead by organically coordinating a lightweight visual recognition network with a retrieval enhancement generation system. The visual model has only 4.2M parameters and an accuracy rate of over 98%. It can be smoothly deployed on mobile terminals or embedded devices. Visual tags serve as hard-constraint retrieval prefixes, fundamentally eliminating cross-crop misdiagnosis and forming an efficient closed loop of "perception-retrieval-generation", significantly reducing the threshold for field deployment.

[0045] 2. This invention employs LoRA parameter high-efficiency fine-tuning technology, which accurately injects agricultural domain knowledge while maintaining the general capabilities of the base model. Only a very small number of adapter weights need to be trained, which can enable a 7B-level large language model to have profound plant pathology expertise. The gradient accumulation and BF16 accuracy strategy ensure stable training under limited GPU memory resources, which greatly reduces the construction and storage costs of large domain models.

[0046] 3. This invention achieves deep structuring of unstructured agricultural knowledge through intelligent segmentation, metadata encapsulation, and parent document retrieval mechanisms. Boundary protection segmentation based on semantic boundaries ensures the semantic integrity of knowledge fragments. Multi-dimensional metadata such as crops, diseases, and knowledge types provide hard constraint filtering basis for accurate retrieval. Parent document retrieval automatically expands the context around the core block, effectively solving the semantic fragmentation problem caused by knowledge fragmentation and ensuring the coherence and information integrity of the generated answer.

[0047] 4. This invention constructs a multi-factor comprehensive scoring and dynamic routing decision mechanism to achieve quantitative evaluation of retrieval quality and adaptive switching of generation paths. It uses a weighted fusion of vector similarity, reordering relevance score and domain reward score, taking into account semantic breadth, logical depth and factual accuracy. It uses dual thresholds for entity coverage and keyword overlap to accurately identify knowledge blind spots and trigger backtracking generation, effectively suppressing the illusion of large language models and ensuring the professionalism and factual accuracy of key information such as drug names and dosages. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a structural block diagram of the crop disease diagnosis system based on a lightweight visual model and retrieval enhancement, as presented in this invention.

[0050] In the diagram: 1. Lightweight visual recognition network; 2. Adaptive retrieval enhancement module; 3. Agricultural big language model; 4. Routing decision module. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0052] Those skilled in the art will understand that all or part of the steps in the methods of the following embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] To address the lack of a lightweight intelligent disease diagnosis solution in existing technologies that can operate stably in resource-constrained environments while possessing accurate identification, professional knowledge, and rapid response, this invention provides a crop disease diagnosis system based on a lightweight visual model and retrieval enhancement. This system comprises four modules: a lightweight visual recognition network 1, an adaptive retrieval enhancement module 2, an agricultural big data language model 3, and a routing decision module 4. These four modules work collaboratively to form a closed loop from image input to the generation of a professional diagnostic report. Each module is described in detail below:

[0054] 1. Lightweight Visual Recognition Network 1. When a user uploads a partial image of a crop leaf or plant via a mobile terminal or embedded device, the system first calls the lightweight visual recognition network 1 for processing. It extracts visual labels containing crop type and disease type from the input image. This label (e.g., "strawberry gray mold") will serve as a hard constraint, greatly narrowing the scope of subsequent knowledge retrieval and preventing the system from getting confused when dealing with diseases that look similar but belong to different crops. The core of this network is a pre-trained MobileNetV3-Large model. MobileNetV3 was chosen because it achieves an excellent balance between accuracy and efficiency in mobile vision tasks. Compared with traditional large models such as VisionTransformer, MobileNetV3, through deep separable convolutions, neural architecture search (NAS) optimized activation functions (such as h-swish), and efficient attention modules (SE modules), and by being fully trained on a large dataset containing 11 crops and more than 40,000 images, can maintain a strong ability to capture subtle lesion texture features with a very low number of parameters (4.2M parameters in this embodiment, with an accuracy of 98%), achieving lightweight deployment. Of course, models such as ShuffleNetV2 or EfficientNetV2 can also be used, but under the strict constraint of 4M parameters, MobileNetV3 has the best balance between the extraction efficiency and inference speed of subtle features such as agricultural lesions.

[0055] 2. Adaptive Retrieval Enhancement Module 2. After obtaining visual tags (e.g., "strawberry gray mold"), the system will enter the knowledge retrieval stage. This task is undertaken by Adaptive Retrieval Enhancement Module 2. This module is not a simple keyword matching process, but a complex process guided by visual tags and involving multiple filtering and optimization. For example, after a user inputs an image, the natural language question they might ask (e.g., "How to treat it?") and the image's visual tags will together constitute the query intent. This module needs to quickly and accurately locate the most relevant professional knowledge fragments to the current crop disease and the user's intent from an agricultural knowledge base containing a vast amount of agricultural books (usually PDF text).

[0056] Before using this system, the knowledge base needs to be preprocessed. Agricultural books are divided into multiple document blocks encapsulated with metadata. Agricultural knowledge (such as textbooks) is generally characterized by strong logic and close contextual relationships. If fixed-length segmentation is used, the natural connection between symptom descriptions and prevention methods will be easily severed, resulting in fragmented search results. Therefore, this system adopts an intelligent segmentation strategy based on semantic boundary recognition when constructing the knowledge base, and attaches rich metadata (such as crop, disease, knowledge type) to each text block. This makes it possible for subsequent accurate filtering and multi-dimensional scoring. The specific steps are as follows:

[0057] First, agricultural books are analyzed, not only extracting the text content but also identifying the hierarchical heading structure and paragraph boundaries to obtain the analyzed text paragraphs. Considering that agricultural knowledge has a rigorous logical system, it is usually organized in a hierarchy of "crop -> disease -> symptoms / prevention". Identifying these structures is a prerequisite for subsequent meaningful text segmentation that does not disrupt the knowledge logic. This assigns "coordinates" of the position and level of each text unit in the original document, laying the foundation for parent document retrieval in subsequent steps. By combining regular expression matching (used to identify labeling patterns such as "Chapter 1", "1.1", "(I)") and semantic analysis (to determine typesetting information such as font and font size), a tree-like logical structure of the document can be constructed.

[0058] Based on the parsed paragraphs, the system identifies the content type of the parsed text paragraphs according to predefined agricultural keyword rules and injects corresponding knowledge type tags. The knowledge type tags include at least a first type tag for identifying disease symptoms and a second type tag for identifying control measures. Keywords corresponding to the first type tag generally include "harm", "manifestation", "symptoms", "leaf appearance", "lesion", and "characteristics". Keywords corresponding to the second type tag generally include "control methods", "pesticides", "spraying", "dosage", "agricultural control", and "management measures". The system scans the paragraph text, and if it detects that the frequency of a certain type of keyword exceeds a threshold, it can mark the paragraph with the corresponding knowledge type tag. For example, when a user queries with a specific intention (such as "how to treat it?"), the retrieval system can prioritize searching in document blocks with the "control methods" tag, greatly improving the accuracy and efficiency of the retrieval.

[0059] To address the semantic fragmentation caused by fixed-length segmentation, this embodiment divides text paragraphs into multiple document blocks of a preset length. The segmentation point of each document block is dynamically anchored to the nearest paragraph end character or period. For example, a base block size (e.g., 500 characters) is initially set as a reference for the target block size. During actual segmentation, the system does not directly cut at a fixed character count. Instead, it dynamically searches for the nearest paragraph end character or period in the text before and after reaching the base block size, and uses this position as the actual segmentation point. It is important to note that to ensure contextual continuity, this embodiment allows for a small number of overlapping characters (e.g., 50 characters), meaning that adjacent blocks retain a small amount of repeated text at the boundary. This segmentation method ensures that the segmented document blocks are semantically as self-consistent and complete as possible, avoiding the truncation of a complete sentence or key description, thereby improving the quality of subsequent vectorized representation and the accuracy of retrieval.

[0060] Next, metadata needs to be encapsulated for each document block. The metadata should include at least the name of the corresponding agricultural book, the name of the corresponding crop, the name of the disease, the knowledge type tag, and the parent document ID. The name of the agricultural book records which authoritative work the document block is from, and the parent document ID is a unique identifier pointing to the original complete chapter or parent block to which the document block belongs. In the subsequent first-level search, the crop name and disease name can be used for precise matching. When calculating the comprehensive score, the metadata can be matched with visual tags to give domain bonus points and guide the results to shift to the correct domain. When performing context backtracking in the future, the parent document ID can be used to quickly locate and obtain adjacent document blocks, realize parent document retrieval, and quickly restore the complete context.

[0061] After constructing the agricultural knowledge base, a two-level retrieval strategy was adopted. In the first-level retrieval, multiple candidate document blocks were obtained by searching the agricultural knowledge base based on visual tags and metadata. That is, the system first uses the visual tags (such as "strawberry gray mold") output by the lightweight visual recognition network 1 to directly match and filter the metadata of all document blocks in the knowledge base. For example, only those document blocks whose metadata shows the crop type as strawberry and the disease type as gray mold are retained. Its purpose is to quickly narrow the search space from the entire agricultural knowledge base to a specific disease subset directly related to the current image with the lowest computational cost. This fundamentally avoids cross-crop misdiagnosis that may occur in subsequent steps. For example, knowledge of apple diseases is used for strawberries, ensuring the absolute relevance of the retrieval basis. In the secondary retrieval, visual tags, user's original questions, and system instructions are reorganized according to preset logic to construct combined tags. Then, a comprehensive score reflecting the matching degree and retrieval quality between the combined tags and each candidate document block is calculated in turn. The candidate document block with the highest comprehensive score is selected. Based on the user's specific question intent (e.g., asking about symptoms or prevention), the document block that is most semantically close and relevant is further found. Finally, the parent document retrieval mechanism and the candidate document block with the highest comprehensive score are used to backtrack the context.

[0062] In secondary retrieval, filtering is achieved by calculating a comprehensive score. Existing technologies generally use a single similarity metric (such as cosine similarity) for filtering. However, this approach is difficult to comprehensively evaluate the quality of agricultural retrieval. It may not be able to distinguish between semantically similar but crop-related cases, and it may also miss important information due to the failure to match key professional terms. Therefore, a comprehensive score is designed to ensure that the knowledge fragments used to generate the final answer have both breadth and accuracy. The comprehensive score is a weighted sum of vector similarity score, rearranged relevance score, and domain reward score. The vector similarity score is used to vectorize the combined tags and candidate document blocks and calculate their cosine similarity. This score measures the basic proximity between the combined tags and candidate document blocks in the dense vector semantic space. It is mainly used to solve the lexical gap problem caused by different expressions but the same core meaning (e.g., "prevention" and "treatment"), and provides the semantic breadth of retrieval. Vector similarity reflects "whether the meaning is similar", but it cannot determine "whether it really answers the question". For example, for a query about "strawberry gray mold symptoms", the vector might match a text describing "strawberry growth habits". The two are semantically related but do not directly answer the question. Therefore, a deeper semantic understanding needs to be introduced. The reorder relevance score is obtained by concatenating the combined tags and candidate document blocks and inputting them into a cross-encoder model. Through a more complex interactive model, it evaluates the deep logical response relationship between the combined tags and candidate blocks. It can effectively identify and suppress semantically relevant but irrelevant candidate blocks, which is the key to improving retrieval accuracy. This score does not rely on pre-computed vectors, but is calculated in real time using a reordering model with a cross-encoder architecture (such as BAAI / bge-reranker-large). The combined tags and candidate document block texts can be concatenated into a sequence and directly input into the reordering model. The attention mechanism inside the model allows each term in the query and document to interact fully, thereby understanding the complex logical relationship between the two (such as causal relationship, inclusion relationship, etc.). The model directly outputs a scalar score in the interval [0,1].

[0063] Domain reward scores are based on the matching of visual tags with crop and disease names in candidate document block metadata. Each successfully matched item is assigned a preset score according to predefined rules, and the sum of the scores for all matching items is calculated. This ensures that the visual tags output by the lightweight visual recognition network 1 are used as the highest priority retrieval constraint, guaranteeing that the retrieved knowledge is 100% aligned with the input image in terms of crop and disease entities, fundamentally preventing cross-crop misdiagnosis. Domain reward score calculation depends on the matching results between visual tags and candidate block metadata. If the crop name field in the candidate document block metadata is completely consistent with the crop name parsed from the visual tags, a reward score (e.g., 0.3) is awarded; otherwise, it is 0. Similarly, if the disease name field in the candidate document block metadata is completely consistent with the disease name parsed from the visual tags, a reward score (e.g., 0.4) is awarded; otherwise, it is 0. Generally, higher rewards are given for disease matching because the same crop may suffer from multiple diseases, and accurate disease name matching has a greater impact on the professionalism of the final answer than crop name matching. Furthermore, considering the importance of vector similarity score, rearrangement relevance score, and domain reward score in the generation of results, to ensure the accuracy of the output results, the weight of the domain reward score is greater than that of the rearrangement relevance score, and the weight of the rearrangement relevance score is greater than that of the vector similarity score. That is, crop matching and disease matching are the highest constraints, the rearrangement relevance score ensures the relevance of the generated answers, and the vector similarity score, as a basic recall method, mainly plays an auxiliary and supplementary role in the final fine ranking stage to prevent the complete exclusion of some semantically highly relevant results due to possible minor biases in the rearrangement model itself. Therefore, it is given the lowest weight. In this way, the retrieval strategy is shifted from similarity-centered to answer-centered. In this embodiment, the weights of vector similarity score, rearrangement relevance score, and domain reward score are set to 0.3, 0.7, and 1, respectively.

[0064] After calculating the comprehensive score, the candidate document block with the highest comprehensive score is marked as the core block. Furthermore, considering that agricultural knowledge in authoritative textbooks is usually presented in a continuous and logically rigorous manner—for example, "symptom description," "disease pattern," and "control methods" often belong to adjacent paragraphs or chapters—providing only a single document block may not meet the needs of generating a complete and coherent control plan. Therefore, this invention introduces a parent document retrieval mechanism to expand the context of the core block to obtain richer background information. This system will activate the parent document retrieval mechanism to trace back the context, that is, map the parent document ID in the metadata encapsulated on the core block back to the complete chapter of the original agricultural book. Centered on the core block, it extracts multiple (e.g., two) adjacent document blocks before and after it in the complete chapter of the original agricultural book to obtain the context containing the core block. This ensures that even if the most relevant block retrieved is "disease symptoms," the system can automatically capture the immediately following "control suggestions" or the preceding "pathogen description," thus providing a self-consistent, complete set of background knowledge that requires no additional piecing together. The window size can be set to 2, which is an empirical balance that effectively supplements the context while avoiding the introduction of too many documents that may dilute the core relevance.

[0065] 3. Agricultural Large Language Model 3. This model is obtained through supervised fine-tuning of a pre-built large language model based on an agricultural knowledge base. That is, the model is not built from scratch, but rather on an excellent general pre-trained model. Precise agricultural plant pathology knowledge is injected through efficient fine-tuning techniques. The specific construction steps are as follows: First, a base large language model for training is selected. In this embodiment, Qwen2.5-7B-Instruct is chosen as the base model, i.e., the basic large language model. The reasons for choosing this model are: firstly, its 7B parameter count maintains strong inference capabilities while being more conducive to subsequent deployment and application compared to larger-scale models; secondly, the "Instruct" version means that the model has been optimized for instruction-following tasks, enabling it to better understand and execute complex task formats such as "answer questions based on the following information," which highly aligns with the system's requirement to combine visual tags, retrieval context, and user questions to generate structured answers.

[0066] During model loading, this embodiment uses BF16 (bfloat16) precision instead of the more common FP16. This is because, although BF16 and FP16 both occupy 2 bytes of memory, BF16 has the same 8 exponent bits as FP32, thus providing a larger dynamic range. When processing specialized texts in the agricultural field, gradient calculations may involve a wide range of numerical values. The large dynamic range of BF16 can effectively avoid training instability or failure caused by gradient values ​​that are too large (overflow) or too small (underflow) during fine-tuning, ensuring the numerical stability of model weight updates. This is crucial for learning specialized and precise knowledge.

[0067] Next, to enable the model to respond with the tone and knowledge level of an agricultural expert, specialized training data and a unified dialogue format need to be prepared. Therefore, large-scale agricultural question-and-answer training data (QA data) is required, covering various aspects such as symptom description, etiology analysis, prevention and control measures, and pesticide dosage. This data needs to be serialized into a standard dialogue format according to a predefined dialogue template. In this embodiment, the ChatML format template is used to serialize the original QA data into a standard dialogue format. The ChatML format template clearly defines the roles (such as system, user, and assistant) and content boundaries in the dialogue, helping the model clearly distinguish between task instructions, external inputs, and its own output. A key detail is that the system prompt for each data point is consistently written as: "You are a professional agricultural plant pathologist. Please answer the user's questions accurately and in detail based on scientific knowledge." This system prompt continuously guides the model to develop a professional personality or role during fine-tuning, strengthening the domain relevance and rigor of its generated content. Additionally, during this process, the Qwen2.5 tokenizer will be used to segment the formatted text, and a maximum sequence length (Max) will be set. The maximum length (e.g., 1024) is set because converting text into a token ID sequence that the model can process is a necessary step in training. The maximum length is set to ensure training efficiency and to handle potentially long texts. Any text exceeding this length will be truncated to ensure that all training samples have a consistent format.

[0068] Next, the model needs to learn this agricultural knowledge efficiently while avoiding catastrophic forgetting of its original general language capabilities. Therefore, a parameter-efficient fine-tuning method is required. Thus, a standard dialogue format is input into the base large language model, and the attention mechanism of the Transformer layer of the base large language model is fine-tuned through a fine-tuning model. Fine-tuning models can use LoRA, Llama-3-8B-Instruct, InternLM2-7B, P-Tuning v2, and Adapter, etc. This embodiment uses LoRA (Low-Rank Adaptation) technology to fine-tune the base model. The reason for using this technique instead of full-parameter fine-tuning is that full-parameter fine-tuning requires updating all parameters of the model (approximately 7 billion), resulting in extremely high computational and storage costs, and easily causing the model to "forget" its original extensive knowledge. LoRA, on the other hand, injects new knowledge by adding a trainable low-rank matrix next to the original weights, requiring only a very small number of parameters to train (usually less than 1% of the original model parameters), achieving a performance comparable to full-parameter fine-tuning, thus achieving a balance between efficiency and performance. The specific implementation principle of LoRA in this embodiment is described below:

[0069] First, the LoRA adapter is injected into the Query and Value weight matrices of the self-attention mechanism in the Transformer layer and then decomposed into low rank. These two positions are chosen because they are crucial for the model to understand the task and generate content. For a pre-trained weight matrix... LoRA does not update directly. Instead, by Two trainable low-rank matrices are introduced on the side. and To achieve incremental updates, the modified output... The calculation formula is as follows: Where x is the input vector, r is the rank, d is the input feature dimension of the weight matrix, and k is the output feature dimension of the weight matrix. The increment represents the original output. In this embodiment, the rank r is set to 16. It determines the intrinsic dimension of the low-rank matrix and is a key hyperparameter for balancing the number of trainable parameters and the expressive power of the model. A smaller r (such as 8 or 16) can capture enough task-specific information, and only the low-rank matrix is ​​updated during training. And the parameters of B, freeze low-rank matrix It is usually initialized with a random Gaussian distribution. It can be initialized to a zero matrix to ensure that the increment is calculated at the start of training. The initial value is zero, and fine-tuning starts from the original point of the base model. Additionally, a scaling factor related to r can be introduced during training. To control the intensity of incremental updates, where It is a hyperparameter, which can be set to 32 in this embodiment. It is used to adjust the influence of LoRA weights on the original output of the model. This helps to stabilize training and make the behavior of adjusting the learning rate under different r more consistent.

[0070] After determining an efficient fine-tuning method, a specific training strategy is needed to guide model optimization. In this embodiment, the AdamW optimizer is used during training. However, an excessively large learning rate will drastically change the weights and destroy the model's original general capabilities. Therefore, the learning rate is set to a very small value, such as... To prevent the disruption of pre-trained weights and thus gently and purposefully guide the model toward agricultural applications, a gradient accumulation mechanism can be used for memory optimization to obtain a stable optimization direction under limited hardware resources. Its calculation formula can be expressed as: ,in, Indicates global batch. Indicates a single card batch. The cumulative number of steps can be set to 4 for a single card and 4 for the cumulative number of steps, thereby simulating 16 global batches with limited video memory (such as a single 24G graphics card) to ensure the stability of the gradient descent direction.

[0071] After training, only the small-sized fine-tuned model weights (i.e., the parameters of all low-rank matrices A and B) are saved, thus achieving lightweight deployment of the model. During inference, the saved fine-tuned model weights are merged into the base large language model to build the agricultural large language model 3. This avoids saving the entire huge fine-tuned model (about 14GB), and only the adapter weights (usually only tens of MB) need to be saved, which greatly reduces storage and transmission overhead.

[0072] 4. Routing Decision Module 4. After the adaptive retrieval enhancement module 2 completes the context expansion, the system obtains an expanded text fragment centered on the core block and containing the preceding and following neighboring blocks, which is the backtracked context. However, the quality of this context is not necessarily high. It may be highly relevant to the user's question and contain all the entities in the user's question, or it may be information-sparse due to insufficient coverage of the agricultural knowledge base or retrieval bias. If low-quality retrieval results are forcibly input into the agricultural big language model 3, it may cause the model to generate biased answers based on incomplete or weakly related information. Therefore, this invention designs a routing decision module 4, which selectively inputs visual tags, the user's original question, and the backtracked context, or only visual tags and the user's original question, into the fine-tuned agricultural big language model 3 according to the retrieval quality of the candidate document blocks, in order to generate diagnostic results. Through this method, the system combines the accuracy of retrieval enhancement with the robustness of the pure agricultural big language model, effectively avoiding the illusion phenomenon. In this embodiment, entity coverage and keyword overlap are used as evaluation criteria for retrieval quality. The specific steps are as follows:

[0073] First, calculate the entity coverage and keyword overlap between the user's original question and the context obtained from the backtracking. If the entity coverage or keyword overlap is lower than the corresponding threshold (e.g., the entity coverage is set to 0.5 and the keyword overlap is set to 0.4), then input the visual labels and the user's original question into the fine-tuned Agricultural Big Language Model 3 to generate a diagnostic result. Otherwise, input the visual labels, the user's original question, and the context obtained from the backtracking into the fine-tuned Agricultural Big Language Model 3 to generate a diagnostic result.

[0074] The generation of agricultural disease diagnosis and control recommendations requires extremely high precision in terminology. The reliability of an answer often depends on whether it accurately mentions the correct crop name, disease name, and specific pesticide name. Entity coverage measures the degree to which search results cover the user's question at the key domain entity level. Its role is to identify and penalize search results that are semantically related but lack core entities or have mismatches, ensuring that the final generated context highly matches the user's question. The specific steps for calculating entity coverage are as follows:

[0075] First, extract entities from the user's original query to obtain the user entity set. Then, extract entities from the backtracked context to obtain the context entity set. Next, calculate the intersection of the user entity set and the context entity set, and then calculate the ratio of this intersection to the user entity set to obtain the entity coverage.

[0076] Keyword overlap is a composite metric that integrates information from three dimensions: semantic similarity, exact keyword matching, and prior knowledge in the agricultural field. Its purpose is to comprehensively evaluate the degree of alignment between search results and user queries at both the "meaning" and "wording" levels. It also strengthens cross-crop and cross-disease differentiation capabilities through domain-specific rewards. Keyword overlap is calculated based on the BM25 algorithm and incorporates agricultural feature rewards. The specific calculation steps are as follows:

[0077] The user's original query and the backtracked context are vectorized, and their cosine similarity is calculated to obtain a semantic relevance score. The BM25 algorithm is used to calculate the matching degree between the user's original query and the backtracked context to obtain a keyword matching score. The metadata corresponding to the backtracked context is matched against the entities representing crop and disease names in the user's original query. Each successfully matched item is assigned a preset score according to preset rules, and the sum of the scores for all matched items is calculated. An additional preset bonus score is added for each successful match to obtain an agricultural feature bonus item. For example, if a crop name is successfully matched, a preset score of 0.3 is added. After a successful match, an additional bonus score of 0.3 is given, resulting in an increase of 0.6 points after each match. Finally, the weighted sum of the semantic relevance score, keyword matching score, and agricultural feature bonus item is calculated to obtain the keyword overlap.

[0078] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A crop disease diagnosis system based on lightweight visual models and retrieval enhancement, characterized in that, The system includes: A lightweight visual recognition network (1) is used to extract visual labels containing crop type and disease type from input images; The adaptive retrieval enhancement module (2) is used to divide agricultural books into multiple document blocks encapsulated with metadata, retrieve multiple candidate document blocks in the agricultural knowledge base based on visual tags and metadata, reorganize visual tags, user original questions and system instructions according to preset logic to construct combined tags, calculate a comprehensive score reflecting the matching degree and retrieval quality between the combined tags and each candidate document block, and use the parent document retrieval mechanism and the candidate document block with the highest comprehensive score to backtrack the context; the comprehensive score is a weighted sum of vector similarity score, rearrangement relevance score and domain reward score; the vector similarity score is used to vectorize the combined tags and candidate document blocks and calculate their cosine similarity; the rearrangement relevance score is used to concatenate the combined tags and candidate document blocks and input them into the cross-encoder model; the domain reward score is based on the matching of the visual tags and the crop name and disease name in the metadata of the candidate document blocks, assigns a preset score to each successfully matched item according to preset rules, and calculates the sum of the scores of all matched items; The agricultural big language model (3) is obtained by supervised fine-tuning of a pre-constructed big language model based on an agricultural knowledge base; The routing decision module (4) is used to generate diagnostic results based on the retrieval quality of candidate document blocks. The specific steps are as follows: calculate the entity coverage and keyword overlap between the user's original question and the context obtained by backtracking; if the entity coverage or keyword overlap is lower than the corresponding threshold, input the visual labels and the user's original question into the fine-tuned agricultural big language model (3) to generate diagnostic results; otherwise, input the visual labels, the user's original question and the context obtained by backtracking into the fine-tuned agricultural big language model (3) to generate diagnostic results.

2. The crop disease diagnosis system according to claim 1, characterized in that, The specific steps for constructing the agricultural big language model (3) are as follows: Select a base large language model for training and use BF16 precision loading; Acquire question-and-answer training data in the agricultural field and serialize it into a standard dialogue format according to a predefined dialogue template format; The standard dialogue format is input into the basic large language model, and the attention mechanism of the Transformer layer of the basic large language model is fine-tuned through fine-tuning the model. Gradient accumulation mechanism is used during training, and the small-volume fine-tuned model weights are saved after training and merged into the basic large language model during inference to construct the agricultural large language model (3).

3. The crop disease diagnosis system according to claim 1, characterized in that, The specific steps for dividing the agricultural books into blocks are as follows: Agricultural books are analyzed to identify their hierarchical heading structure and paragraph boundaries, resulting in analyzed text paragraphs. Based on predefined agricultural keyword rules, the parsed text paragraphs are identified in terms of content type, and corresponding knowledge type tags are injected; wherein, the knowledge type tags include at least a first type tag for identifying disease symptoms and a second type tag for identifying prevention and control measures; Divide the text paragraph into multiple document blocks of preset length, and dynamically anchor the split point of the document block to the nearest paragraph end character or period; Each document block is encapsulated with metadata, which includes at least the name of the corresponding agricultural book, the name of the corresponding crop, the name of the disease, the knowledge type tag, and the parent document ID.

4. The crop disease diagnosis system according to claim 1, characterized in that, When calculating the weighted sum of the vector similarity score, the rearranged relevance score, and the domain reward score, the domain reward score has a greater weight than the rearranged relevance score, and the rearranged relevance score has a greater weight than the vector similarity score.

5. The crop disease diagnosis system according to claim 1, characterized in that, The specific steps for the parent document retrieval mechanism to trace back the context are as follows: The candidate document block with the highest overall score is marked as the core block; Map back to the complete chapter of the original agricultural book based on the parent document ID in the metadata encapsulated on the core block; Centered on the core block, extract multiple adjacent document blocks before and after it in the complete chapter of the original agricultural book to obtain the context containing the core block.

6. The crop disease diagnosis system according to claim 1, characterized in that, The specific steps for calculating the entity coverage rate are as follows: Extract entities from the user's original question to obtain the user entity set, and extract entities from the context obtained through backtracking to obtain the context entity set; Calculate the ratio of the intersection of the user entity set and the context entity set to the user entity set to obtain the entity coverage.

7. The crop disease diagnosis system according to claim 1, characterized in that, The keyword overlap is obtained based on the BM25 algorithm and incorporates agricultural feature reward calculation. The specific calculation steps are as follows: The user's original question and the context obtained from the backtracking are vectorized, and the cosine similarity between the two is calculated to obtain a semantic relevance score; The BM25 algorithm is used to calculate the degree of matching between the user's original question and the context obtained from the backtracking, in order to obtain a keyword matching score. The metadata corresponding to the backtracked context is matched with the entities representing crop names and disease names in the user's original question. Each successfully matched item is assigned a preset score according to preset rules. The sum of the scores of all matched items is calculated, and a bonus score of a preset value is added for each successful match to obtain agricultural feature bonus items. The keyword overlap is obtained by calculating the weighted sum of semantic relevance score, keyword matching score, and agricultural feature reward item.

Images