Multimodal data distillation agricultural pest and disease detection large model construction method and system

Abstract

The application discloses a multi-modal data distillation agricultural pest and disease detection large model construction method and system, wherein the method comprises the following steps: obtaining pre-retained data sets S1, S11 and S12 through preprocessing; using a base large model to distinguish the data RS1 that can correctly answer and the data IS1 that cannot correctly answer from S11, and finally obtaining a pre-retained data set S3 through large model scoring; merging the data set S1 with a long thinking chain and the data set S12 to obtain a pre-retained data set S2; performing LoRA fine-tuning based on an honest large model to obtain a data distillation large model; determining the quantization parameters and quantization algorithms of the data distillation large model, and performing quantization perception training optimization and precision calibration to obtain a quantized large model; and capturing images automatically for user input information or a camera, respectively performing standardization processing and binary classification judgment to obtain a final answer. The application effectively solves the problems of unprofessional, difficult deployment and illusion of traditional large models in the field of agricultural pest and disease.

Description

Technical Field

[0001] The present invention relates to the intersection of artificial intelligence and the agricultural Internet of Things, and specifically to a method and system for constructing a large agricultural pest and disease detection model using multimodal data distillation. Background Art

[0002] In the field of agricultural pest and disease identification and detection, traditional large models, despite being pre-trained and fine-tuned on massive amounts of data, still have significant shortcomings in the vertical field of agricultural pests and diseases. They face problems such as a lack of dispersed expertise, inaccurate feature extraction, and insufficient scene adaptation, making it difficult for the models to accurately identify pests and diseases. At the same time, the large number of parameters exceeding 500B imposes extremely high requirements on hardware computing power, storage resources, and energy consumption. The huge economic cost seriously hinders its deployment and promotion in agricultural scenarios. In addition, the "hallucination problem" makes it easy for the model to make erroneous inferences that are out of touch with the actual scene when processing user or camera input, which can cause significant losses to agricultural production.

[0003] To this end, the present invention proposes a method and system for constructing a large model for agricultural pest and disease detection based on multimodal data distillation to solve the problems of unprofessionalism, difficulty in deployment, and hallucinations in traditional large models in terms of agricultural pests and diseases. Summary of the Invention

[0004] The present invention provides a method and system for constructing a large agricultural pest and disease detection model using multimodal data distillation, which improves the overall professionalism of model detection, reduces the number of model parameters, and enhances the accuracy and credibility of model output. This method and system can solve at least one of the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for constructing a large agricultural pest and disease detection model using multimodal data distillation includes the following steps:

[0007] Step 1: Collect and organize the agricultural pest and disease image-text paired datasets, and pre-process them to obtain the reserved datasets S1, S11, and S12;

[0008] Step 2: Use the base large model to distinguish the data RS1 that can be answered correctly from the data IS1 that cannot be answered correctly from S11. Finally, the pre-reserved dataset S3 is obtained by scoring the large model. Then, the DPO algorithm is fine-tuned on the pre-reserved dataset S3 using the base large model to obtain the honest large model with binary classification judgment. At the same time, the datasets S1 and S12 with long thought chains are merged to obtain the pre-reserved dataset S2. LoRA is fine-tuned based on the honest large model to obtain the data distillation large model.

[0009] Step 3: Determine the quantization parameters and quantization algorithm of the data distillation large model, and perform quantization-aware training optimization and accuracy calibration to obtain the quantized large model;

[0010] Step 4: Standardize the information input by the user or the images automatically captured by the camera, use a large quantitative model for preliminary reasoning, and then perform a binary classification judgment to obtain the final answer.

[0011] Furthermore, the Step 1 further includes:

[0012] Step 11. Data Collection and Organization: Because text is easier to obtain comprehensively, we first obtained the text names of agricultural pests and diseases through various channels. We then organized the dataset using a two-layer labeling system. The bottom layer was based on the biological classification system, while the top layer was based on practical agricultural production classification standards. This enabled multi-dimensional, standardized data classification and storage. Entity similarity filtering and manual review were used to remove duplicates and obtain unambiguous text names of agricultural pests and diseases.

[0013] Step 12: Image-text association expansion: The collected agricultural pest and disease text names are used as entities to search for keywords in professional image databases and internet resources for corresponding images. The large-scale model is first used to determine whether the pest and disease name and image match each other, and the image-text datasets with successful matching are selected. The large-scale model is then used to score the image-text datasets that fall below a threshold of 5. Finally, through manual screening, the image-text matching datasets that are considered to be correct are selected, and the high-quality image-text datasets are retained.

[0014] Step 13. Large model-guided completion: Design a standardized information template, which is used to clarify the key fields that need to be supplemented in the graphic data of agricultural pests and diseases, and input the graphic data into the large model with deep thinking. The standardized information template guides the large model to carry out targeted completion of the missing or incomplete content in the graphic data, and generate the corresponding reasoning process data set as the long thinking chain data, which together constitute and are saved as the pre-reserved data set S1. At the same time, in order to obtain a diverse and high-quality data set, S1 is used as the main data set, and no more than 15% of the dictionary-like data pairs are selected as question-answer pairs to be generated again, and generated twice respectively to obtain the pre-reserved data sets S11 and S12.

[0015] Furthermore, the Step 2 further includes:

[0016] Step 21: Question the pre-reserved dataset S11 via the large base model to ensure the quality of the dataset;

[0017] Step 22: Save the dataset corresponding to the questions answered incorrectly by the base model as IS1 and add the label "label": 0 to the dataset. Save the dataset corresponding to the questions answered correctly by the base model as RS1 and add the label "label": 1 to the dataset to obtain a dataset S3 with binary classification labels. Use the DPO algorithm to train an honest large model for binary classification to distinguish between knowledge that the base model can answer and knowledge that it cannot answer.

[0018] Step 23: Merge the pre-reserved datasets S1 and S12 to form the complete agricultural pest and disease image-text pairing dataset S2 with long thought chains for the second stage of LoRA fine-tuning.

[0019] Step 24. Use the improved LoRA low-rank adapter fine-tuning method to add a multi-head low-rank weight matrix to the attention layer of the frozen honest large model Transformer to adapt to the downstream tasks of the agricultural pest and disease image-text pairing dataset, and decompose the low-rank matrix into multiple heads. Each head focuses on learning the features of different dimensions in the agricultural pest and disease image data to further improve the low-rank matrix's ability to extract information granularity.

[0020] Furthermore, in Step 22, the loss function expression of the DPO algorithm is:

[0021]

[0022] In the above formula, x is the problem input of the pre-reserved dataset S3, exp is the exponential function, log is the logarithmic function, and y is the preferred It is RS1,y rejected is IS1, β is the average score obtained by using the large model to score the pre-reserved dataset S11, which is between 0 and 10. The RS1 data score is above 5 points, and the IS1 data score is below 5 points.

[0023] Furthermore, in Step 24, the expression of the multi-head low-rank weight matrix is:

[0024]

[0025] The forward propagation method of the multi-head low-rank weight matrix to change the model is:

[0026]

[0027] In the above two formulas, A j and B iRepresent multi-head low-rank matrices, i and j represent the number of multi-head low-rank matrices, h represents the maximum number of multi-head low-rank matrices, x represents the input features of the large model with parameters > 500B after removing data with scores below 5, W represents the frozen honest large model, ΔW represents the updated multi-head low-rank weight matrix, and f is the output feature of the large agricultural pest and disease model after LoRA fine-tuning;

[0028] Perform image preprocessing and normalization on the user input feature x that is input to the quantized large model for inference to increase the generalization of the model. The steps are as follows:

[0029] Image preprocessing operations include resizing and denoising;

[0030] Resizing: The input image is uniformly scaled to the input size H×W specified by the model. Using the bilinear interpolation algorithm, for the pixel at the coordinate (x, y) in the input image, the pixel value I(u, v) at the corresponding position (u, v) in the target image is calculated as follows:

[0031]

[0032]

[0033]

[0034] In the above three formulas, W src 、H src 、W dst and H dst Represent the width and height of the input large model image and the width and height of the target image during inference, respectively. (x i ,y j ) are the four neighboring pixels around (u,v), w i and w j are the linear weights in the horizontal and vertical directions calculated based on the pixel coordinate distance;

[0035] Denoising: To address the noise that may be generated during the shooting process, a median filter algorithm is used. For each pixel point (x, y) in the image, the median of all pixel values ​​within an n×n neighborhood is taken as the output value of the point, effectively removing impulse noise.

[0036] Normalization operation: Normalize the preprocessed image and map the pixel value range to the interval required by the model. In order to map to [-1, 1], the linear normalization formula is used:

[0037]

[0038]

[0039] In the above two formulas, I (x, y) is the pixel value of the pixel point in the resized image, min (I) is the minimum pixel value of the resized image, max (I) is the maximum pixel value of the resized image, and I norm (x, y) is the pixel value finally normalized to the interval [-1, 1];

[0040] Through the above standardization processing, the differences in size and noise between images taken by different devices are eliminated, and the processed information is used as the input feature x input information of the base large model.

[0041] Furthermore, the Step 3 further includes:

[0042] Step 31. Determine quantization parameters: Quantization parameters determine the numerical accuracy of model weights and activation values, directly affecting model size, inference speed, and accuracy loss.

[0043] Step 32. Determine the quantization granularity: The quantization granularity determines the granularity of grouping and quantizing the model weights, affecting the accuracy loss and compression efficiency.

[0044] Step 33. Determine the quantization algorithm. To alleviate the accuracy loss caused by quantization, adopt the algorithm of layer-by-layer quantization + quantization-aware training optimization + accuracy calibration.

[0045] Furthermore, the Step 4 further includes:

[0046] Step 41: Standardize input data such as text or image information input by the user, or real-time images of the current crop status captured by the camera, to ensure that the input data meets the input requirements of the quantitative large-scale model;

[0047] Step 42: Input normalization is performed on the normalized data. When the quantitative large model receives the input processed data and generates reasoning and correct answers, it first performs a binary classification judgment to determine whether the question can be answered correctly by itself. If the model can answer correctly, it directly infers the corresponding correct answer to guide the user. For questions that the quantitative large model answers incorrectly, it is necessary to call an external API or provide suggested measures rather than direct methods to guide the user.

[0048] The multimodal data distillation agricultural pest and disease detection large model system is applicable to the multimodal data distillation agricultural pest and disease detection large model construction method, including:

[0049] Large model data distillation module M1, which is used to collect data sets, build an agricultural pest and disease database, and conduct image-text matching questions on the large model to construct pre-reserved data sets S1, S11 and S12;

[0050] The data set processing module M2 is connected to the large model data distillation module M1 and inputs data to the data set processing module M2. The data set processing module M2 uses the base large model to perform binary classification judgment on the pre-reserved data set S11, obtains the data sets IS1 and RS1 with long thought chains, and uses IS1 and RS1 for DPO fine-tuning of the honest large model of the large model data distillation module M1;

[0051] The large model quantization module M3 is connected to the data set processing module M2 and inputs data to the large model quantization module M3. The large model quantization module M3 is used to determine the quantization parameters and quantization algorithm, and perform quantization perception training optimization and accuracy calibration to obtain a quantized large model;

[0052] The large model reasoning module M4 is connected to the data set processing module M2 and the large model quantization module M3, and is used to standardize the data input into the quantitative large model, and then introduce a binary classification judgment mechanism to complete the reasoning and answer of the quantitative large model.

[0053] Furthermore, in the large model reasoning module M4, the quantitative large model first performs classification judgment based on the binary classification judgment mechanism introduced by the preference answer to determine whether the large model can correctly solve the problem, and then selects the answer method. For problems that the quantitative large model can solve using its own knowledge, direct reasoning is used to obtain the correct answer. For problems that the quantitative large model cannot solve using its own knowledge, external APIs are called or suggested measures are given instead of direct methods to assist in reasoning and obtain the best answer.

[0054] Furthermore, the parameter amount of the large model is greater than 500B, and the parameter amount of the base large model is less than 32B.

[0055] The beneficial effects of the present invention are embodied in:

[0056] 1. This paper adopts a multimodal data distillation method to combine image and text information, which can better capture the fine-grained feature differences of agricultural pests and diseases in visual and semantic modes. Compared with traditional single-modality models, it can perform pest and disease detection more comprehensively and accurately, improve the professionalism and detection accuracy of the model, and solve the problem of the unprofessionalism of traditional large models in the field of agricultural pest and disease detection.

[0057] 2. This invention significantly reduces the number of parameters in large models through data distillation and quantization techniques, enabling them to run efficiently under low-resource (such as CPU) conditions. This solves the problem of traditional large models being difficult to deploy in the field of agricultural pest and disease detection, reduces hardware costs, and facilitates widespread application on lightweight terminal devices in agricultural production sites, thus having important practical significance.

[0058] 3. The present invention introduces a binary preference judgment mechanism in the reasoning module, which first determines whether the model can correctly solve the problem, and then performs reasoning or calls an external API to assist in answering. This design can effectively solve the hallucination problem that is prone to occur in traditional large models in the field of agricultural pest and disease detection, improve the reliability and accuracy of the model output, and enhance the credibility of the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0060] Figure 1 It is a schematic diagram of the overall process of the method for constructing a large agricultural pest detection model according to an embodiment of the present invention.

[0061] Figure 2 2 is a schematic diagram of the processing flow of the large model data distillation module M1 according to an embodiment of the present invention.

[0062] Figure 3 Schematic diagram of the processing flow of the data set processing module M2 according to an embodiment of the present invention.

[0063] Figure 4 3 is a schematic diagram of the processing flow of the large model quantization module M3 according to an embodiment of the present invention.

[0064] Figure 5 It is a schematic diagram of the processing flow of the large model reasoning module M4 in an embodiment of the present invention.

[0065] Figure 6 It is a structural block diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0067] It should be noted that the meaning of "and / or" appearing throughout the text includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions in which both A and B are satisfied. In addition, "multiple" refers to more than two. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0068] See also Figure 1 The embodiment of the present invention provides a method for constructing a large model for agricultural pest and disease detection using multimodal data distillation, comprising the following steps:

[0069] Step 1: Collect and organize the agricultural pest and disease image-text paired datasets, and pre-process them to obtain the reserved datasets S1, S11, and S12;

[0070] Step 2: Use the base large model to distinguish the data RS1 that can be answered correctly from the data IS1 that cannot be answered correctly from S11. Finally, the pre-reserved dataset S3 is obtained by scoring the large model. Then, the DPO algorithm is fine-tuned on the pre-reserved dataset S3 using the base large model to obtain the honest large model with binary classification judgment. At the same time, the datasets S1 and S12 with long thought chains are merged to obtain the pre-reserved dataset S2. LoRA is fine-tuned based on the honest large model to obtain the data distillation large model.

[0071] Step 3: Determine the quantization parameters and quantization algorithm of the data distillation large model, and perform quantization-aware training optimization and accuracy calibration to obtain the quantized large model;

[0072] Step 4: Standardize the information input by the user or the images automatically captured by the camera, use a large quantitative model for preliminary reasoning, and then perform a binary classification judgment to obtain the final answer.

[0073] This invention aims to improve the model's professionalism, ease of deployment, and low hallucination capabilities in the field of agricultural pest and disease detection through a complete set of processes from data collection to training.

[0074] See also Figure 2 In this embodiment, Step 1 further includes:

[0075] Step 11. Data Collection and Organization: Because text is easier to obtain comprehensively, we first obtained the text names of agricultural pests and diseases through various channels. We then organized the dataset using a two-layer labeling system. The bottom layer was based on the biological classification system (kingdom > phylum > class > order > family > genus > species). The top layer was combined with practical agricultural production classification standards (such as food crops, cash crops, and other categories) to achieve multi-dimensional and standardized data classification and storage. Entity similarity filtering and manual review were used to remove duplicates and obtain unambiguous text names of agricultural pests and diseases.

[0076] To ensure the diversity and authority of data sources, we first systematically sorted and sorted the partially complete pest and disease image-text paired data from Internet resources to construct dataset D1. Next, we collected agricultural pest and disease datasets from agricultural research institutions, online databases, and professional literature on the Internet to form dataset D2. Finally, we collected pest and disease datasets from paper-based references to form dataset D3. For duplicate datasets, the credibility was ranked according to D3>D2>D1.

[0077] Step 12: Image-text association expansion: The collected agricultural pest and disease text names are used as entities to search for keywords in professional image databases and internet resources for corresponding images. The large-scale model is first used to determine whether the pest and disease name and image match each other, and the image-text datasets with successful matching are selected. The large-scale model is then used to score the image-text datasets that fall below a threshold of 5. Finally, through manual screening, the image-text matching datasets that are considered to be correct are selected, and the high-quality image-text datasets are retained.

[0078] To improve data diversity, different images were selected from each type of pest and disease dataset based on different developmental stages, pixels, and shooting angles. The number of images selected in this application is tentatively set at 100.

[0079] Step 13. Large model-guided completion: Design a standardized information template, which is used to clarify the key fields that need to be supplemented in the graphic data of agricultural pests and diseases, and input the graphic data into a large model with deep thinking (the number of parameters is greater than 500B). The standardized information template guides the large model to carry out targeted completion of the missing or incomplete content in the graphic data, and generate the corresponding reasoning process data set as the long thinking chain data, which together constitute and are saved as the pre-reserved data set S1. At the same time, in order to obtain a diverse and high-quality data set, S1 is used as the main data set, and no more than 15% of the dictionary-like data pairs are selected as question-answer pairs to be generated again, and generated twice to obtain the pre-reserved data sets S11 and S12.

[0080] The following is an example of a standardized information template designed in this application:

[0081] {"Plant name": "", "Plant category": "", "Plant variety": "", "Name of the causative pathogen":"", "Scientific name of the pathogen": "", "Classification of the pathogen": "", "Biological characteristics of the pathogen": {"Growth temperature range": "", "Conidia germination conditions": "", "Overwintering method": ""}, "Disease name": [], "Affected parts": "", "Appearance characteristics": {"Color": "", "Shape": "", "Texture": "", "Edge": ""}, "Symptom details": {"Leaves": "", "Fruit": "", "Branches": ""}, "Scene": "", "Cause of disease": "", "Disease cycle": "", "Infection route": [], "Period of disease onset": {"Leaves": "", "Fruit": "", "Branches":""}, "Pathogenesis pattern": {"Budding period": "", "Flowering period": "", "Young fruit period": "", "Fruit expansion period to maturity period": ""}, "Transmission method": [], "Host range": [], "Field identification characteristics": "", "Hazard level":"", "Resistant varieties": [], "Preventive measures": "", "Ecological control measures": {"Grass cultivation": "", "Reasonable load": "", "Fruit bagging": ""}, "Treatment measures": "", "Mechanism of action of pesticides": "", "Resistance management": {"Rotation of pesticides": "", "Mixed pesticides": "", "Monitoring and early warning": ""}, "Economic threshold": "", "Critical period for prevention and control": [], "Economic impact": [], "Global distribution": "", "Main affected areas in China": "", "Legal and regulatory management": "", "Post-disaster management": {"Severely affected orchards": "", "Trees with severe defoliation": ""}, "References":[], "Data source": "", "Update date": "", "Data version": "", "Image": ""}.

[0082] See also Figure 3 In this embodiment, Step 2 further includes:

[0083] Step 21: The pre-reserved dataset S11 is questioned via the base model, questions answered correctly by the base model are discarded, and questions answered incorrectly by the base model are retained to ensure the quality of the dataset;

[0084] Step 22: Save the dataset corresponding to the questions answered incorrectly by the base model as IS1 and add the label "label": 0 to the dataset. Save the dataset corresponding to the questions answered correctly by the base model as RS1 and add the label "label": 1 to the dataset. This results in a dataset S3 with binary classification labels. Use the DPO algorithm (Direct Preference Optimization) to train an honest large model for binary classification to distinguish between knowledge that the base model can answer and knowledge that it cannot answer.

[0085] Step 23: Merge the pre-reserved datasets S1 and S12 to form the complete agricultural pest and disease image-text pairing dataset S2 with long thought chains fine-tuned by the second stage LoRA (Low-Rank Adaptation).

[0086] Step 24: Adopt the improved LoRA low-rank adapter fine-tuning method. By adding a multi-head low-rank weight matrix to the attention layer of the frozen honest large model's Transformer, we adapt it to the downstream task of the agricultural pest and disease image-text pairing dataset. The low-rank matrix is ​​decomposed into multiple heads, with each head focusing on learning features of different dimensions in the agricultural pest and disease image-text data, to further improve the low-rank matrix's ability to extract information granularity.

[0087] The low-rank matrix is ​​decomposed into multiple heads, with each head focusing on learning features from different dimensions of agricultural pest and disease image and text data. For example, some heads focus on visual features such as the shape and color of the pests and diseases in the image, while others target semantic features such as disease patterns and control methods in the text descriptions. This approach allows the model to deeply mine image and text information from multiple perspectives, achieving more granular capture of pest and disease characteristics.

[0088] In this embodiment, in Step 22, the loss function expression of the DPO algorithm is:

[0089]

[0090] In the above formula, x is the problem input of the pre-reserved dataset S3, exp is the exponential function, log is the logarithmic function, and y is the preferred It is RS1,y rejected is IS1, β is the average score obtained by using the large model to score the pre-reserved dataset S11, which is between 0 and 10. The RS1 data score is above 5 points, and the IS1 data score is below 5 points.

[0091] In this embodiment, in Step 24, the expression of the multi-head low-rank weight matrix is:

[0092]

[0093] The forward propagation method of the multi-head low-rank weight matrix to change the model is:

[0094]

[0095] In the above two formulas, A j and B i Represent multi-head low-rank matrices respectively, i and j represent the number of multi-head low-rank matrices respectively, h represents the maximum number of multi-head low-rank matrices, x represents the large model with parameter quantity > 500B, which will retain the input features of the dataset S2 after removing data with scores below 5, W represents the frozen honest large model, ΔW represents the updated multi-head low-rank weight matrix, and f is the output feature of the large agricultural pest and disease model (parameter quantity < 32B) after LoRA fine-tuning.

[0096] In this embodiment, image preprocessing and normalization operations are performed on the user input feature x input to the quantized large model for inference to increase the generalization of the model. The operation steps are as follows:

[0097] Image preprocessing operations include resizing and denoising;

[0098] Resizing: The input image is uniformly scaled to the input size H×W specified by the model (for example, 448×448). For the pixel at the coordinate (x, y) in the input image, the pixel value I(u, v) at the corresponding position (u, v) in the target image is calculated as follows:

[0099]

[0100]

[0101]

[0102] In the above three formulas, W src 、H src 、W dst and H dst Represent the width and height of the input large model image and the width and height of the target image during inference, respectively. (x i ,y j ) are the four neighboring pixels around (u,v), w i and w j are the linear weights in the horizontal and vertical directions calculated based on the pixel coordinate distance;

[0103] Denoising: To address the noise that may be generated during the shooting process, a median filter algorithm is used. For each pixel (x, y) in the image, the median of all pixel values ​​within an n×n neighborhood (e.g., 3×3) is taken as the output value for that pixel, effectively removing impulse noise such as salt and pepper noise.

[0104] Normalization operation: Normalize the preprocessed image and map the pixel value range to the interval required by the model. In order to map to [-1, 1], the linear normalization formula is used:

[0105]

[0106]

[0107] In the above two formulas, I (x, y) is the pixel value of the pixel point in the resized image, min (I) is the minimum pixel value of the resized image, max (I) is the maximum pixel value of the resized image, and I norm (x, y) is the pixel value finally normalized to the interval [-1, 1];

[0108] Through the above standardization processing, the differences in size and noise between images taken by different devices are eliminated, and the processed information is used as the input feature x input information of the base large model.

[0109] At this point, after completing the data set collection, preprocessing and fine-tuning operations in Step 1 and Step 2, the data distillation of the large model has been preliminarily completed, solving the unprofessional and hallucination problems of traditional large models in the field of agricultural pest and disease identification and detection.

[0110] See also Figure 4 In this embodiment, Step 3 further includes:

[0111] Step 31. Determine quantization parameters: Quantization parameters determine the numerical accuracy of model weights and activation values, directly affecting model size, inference speed, and accuracy loss.

[0112] The available quantization levels include FP32, FP16, BF16, INT8, INT4, and mixed precision. To balance speed, accuracy, and cost, this application chooses INT8 as the quantization level. Compared to the original FP32, the model size is compressed by 4 times, the inference speed is greatly improved, and there may be a slight loss of accuracy. This makes it easy to deploy on economical GPUs, facilitating agricultural dissemination and use.

[0113] Step 32. Determine the quantization granularity: The quantization granularity determines the granularity of grouping and quantizing the model weights, affecting the accuracy loss and compression efficiency.

[0114] The available granularities include layer-by-layer quantization, channel-by-channel quantization, and element-by-element quantization. Considering that the target device (such as a camera with a low-end ARM CPU) only supports layer-by-layer quantization and fast deployment or has limited hardware resources;

[0115] Step 33. Determine the quantization algorithm. To alleviate the accuracy loss caused by quantization, adopt the algorithm of layer-by-layer quantization + quantization-aware training optimization + accuracy calibration.

[0116] The following is a detailed description of the above determination process:

[0117] 1. Quantization bit selection and compression ratio analysis

[0118] Objective: To balance speed, accuracy, and cost while keeping the number of model parameters W less than 32B.

[0119] Quantization formula: The floating point parameter W FP32 ∈R maps to integer W INT8 , the expression is:

[0120]

[0121]

[0122] Among them, W INT8 ∈{-128,-127,...,127}, S is the scaling factor (Scale), , is the zero point (ZeroPoint, which is 0 for symmetric quantization).

[0123] Compression ratio calculation:

[0124]

[0125] Storage cost optimization:

[0126]

[0127] 2. Quantization Granularity Selection and Layer-by-Layer Quantization Implementation

[0128] Goal: Adapt to the hardware constraints of low-end ARM CPUs (such as camera processors and ordinary laptops).

[0129] Specific operations of layer-by-layer quantization:

[0130] The weight matrix W of the lth layer (l) ∈R m×n , using a uniform scaling factor S (l) and zero point Z (l) :

[0131]

[0132] Hardware adaptability: Low-end ARM CPUs generally lack dedicated instruction support for channel-by-channel or element-by-element quantization. Layer-by-layer quantization can directly utilize general matrix multiplication optimization to reduce deployment complexity.

[0133] 3. Accuracy Calibration Solution

[0134] Goal: Minimize the accuracy loss introduced by quantization .

[0135] Calibration process:

[0136] Data collection: Using a representative sample set D calid ={x1,x2,...,x m Here, you can first select the common crops in the region, and then select the representative pests and diseases of the crops. For example, in East China, the crops that can be considered first are wheat, corn and rice, and then representative pests and diseases can be selected according to the crops, such as wheat powdery mildew, corn leaf spot, rice planthoppers, etc.

[0137] Activation value statistics: layer, calculate activation value The optimal scaling factor is determined by minimizing the KL divergence. :

[0138]

[0139] Layer-by-layer compensation: For sensitive layers (such as the attention mechanism), retain FP32 accuracy:

[0140]

[0141] in, A set of sensitive layers.

[0142] To further reduce the accuracy loss, quantization-aware training optimization is used, especially for modules that are sensitive to INT8 quantization.

[0143] Forward propagation modification: insert pseudo-quantization nodes during training to simulate quantization errors:

[0144]

[0145] Backpropagation optimization: Use straight-through estimators to handle non-differentiable quantization operations:

[0146]

[0147] Through the above operations, the model's reasoning speed is improved:

[0148]

[0149] Memory bandwidth optimization:

[0150]

[0151] GPU deployment: Use Tensor Core to accelerate INT8 matrix operations.

[0152] ARM CPU deployment: Use the NEON instruction set to optimize INT8 computing and combine it with OpenMP multi-threaded parallelism.

[0153] Deployment verification and evaluation indicators:

[0154] 1. Model volume:

[0155] 2. Inference latency: (Depends on hardware architecture)

[0156] 3. Loss of precision: (Third step can be repeated with calibration control)

[0157] At this point, after completing the quantification plan of Step 3, the model deployment cost can be reduced by more than 75% while ensuring the accuracy of pest and disease identification, significantly improving the accessibility of deployment and dissemination of large agricultural pest and disease models, and solving the difficult deployment problem of traditional large models in the field of agricultural pest and disease identification and detection.

[0158] See also Figure 5 In this embodiment, Step 4 further includes:

[0159] Step 41: Normalize the input data, such as text or image information input by the user, or the real-time status image of the current crop captured by the camera, and scale the image size to the input size specified by the model. (e.g. 224×224), remove irrelevant characters, punctuation marks, extra spaces, etc., and only retain text content that is meaningful to the description of crops to ensure that the input data meets the input requirements of the large model;

[0160] Step 42: Perform input normalization on the normalized data. When the quantitative model receives the input processed data and generates inferences and correct answers, it first performs a binary classification judgment to determine whether the question can be answered correctly:

[0161] If the model can answer the question correctly, it directly infers the correct answer to guide the user. Here, "guiding the user" means that when the quantitative model believes it can answer the question, the confidence level of the answer it generates is 100%, and the user can directly follow the guidance.

[0162] For questions that the quantitative model answers incorrectly, it is necessary to call an external API or provide suggested measures rather than direct methods to guide the user. Here, "guiding the user" means that the quantitative model believes that it may answer this question incorrectly, and the generated answer has low credibility and is only used as a reference. Users can take further action or not take action based on the guidance content.

[0163] Before the model obtains the correct inference answer, it first performs a binary preference judgment. To do this, simply add the following prompt: "First, you are a person who can judge whether you can solve this problem correctly based on the input information. If you can solve it correctly, please think step by step and output the correct answer. If there is a possibility of error, you can call the API of an external large model (with more than 500B parameters) for auxiliary answering. When outputting, please add a prefix prompt: 'Call the external large model to generate the answer. The answer may be wrong:' Or if you think you can answer it yourself, please add a prefix prompt: 'Self-answer: I am not sure about this question. You can look up the answer in the Encyclopedia'." This can greatly solve the illusion problem that traditional large models have in the field of agricultural pest and disease identification and detection. Finally, the model obtains the best correct answer through its own knowledge or by calling an external API.

[0164] See also Figure 2-Figure 5 The embodiment of the present invention further provides a multimodal data distillation agricultural pest and disease detection large model system, which is applicable to the multimodal data distillation agricultural pest and disease detection large model construction method, including:

[0165] Large model data distillation module M1, which is used to collect data sets, build an agricultural pest and disease database, and conduct image-text matching questions on the large model to construct pre-reserved data sets S1, S11 and S12;

[0166] The data set processing module M2 is connected to the large model data distillation module M1 and inputs data to the data set processing module M2. The data set processing module M2 uses the base large model to perform binary classification judgment on the pre-reserved data set S11, obtains the data sets IS1 and RS1 with long thought chains, and uses IS1 and RS1 for DPO fine-tuning of the honest large model of the large model data distillation module M1;

[0167] The large model quantization module M3 is connected to the data set processing module M2 and inputs data to the large model quantization module M3. The large model quantization module M3 is used to determine the quantization parameters and quantization algorithm, and perform quantization perception training optimization and accuracy calibration to obtain a quantized large model;

[0168] The large model reasoning module M4 is connected to the data set processing module M2 and the large model quantization module M3, and is used to standardize the data input into the quantitative large model, and then introduce a binary classification judgment mechanism to complete the reasoning and answer of the quantitative large model.

[0169] See also Figure 5 In this embodiment, in the large model reasoning module M4, the quantitative large model first performs classification judgment based on the binary classification judgment mechanism introduced by the preference answer to determine whether the large model can correctly solve the problem, and then selects the answer method. For problems that the quantitative large model can solve using its own knowledge, direct reasoning is used to obtain the correct answer. For problems that the quantitative large model cannot solve using its own knowledge, external APIs are called or suggested measures are given instead of direct methods to assist in reasoning and obtain the best answer. The specific judgment and reasoning of the large model reasoning module M4 is detailed in the corresponding content of Step 41-Step 42 of the method embodiment, which will not be repeated here.

[0170] In this embodiment, the above-mentioned deep thinking large model and large model are both selected as available general large models, whose parameter volume is greater than 500B. The present invention does not specifically limit the specific type of such large models, such as GPT4o, DeepSeek, etc. The base large model can select an open source pre-trained model, whose parameter volume is less than 32B. Similarly, there is no special limitation on the specific type of the base large model, such as Qwen2.5-VL-7B-Instruct, etc.

[0171] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method for constructing a large model for agricultural pest and disease detection by multimodal data distillation as described above.

[0172] See also Figure 6 An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for constructing a large model for agricultural pest and disease detection by multimodal data distillation.

[0173] An embodiment of the present invention also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned method for constructing a large agricultural pest and disease detection model by multimodal data distillation.

[0174] It is understandable that the system, device and storage medium provided by the embodiments of the present invention correspond to the method provided by the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above-mentioned multimodal data distillation agricultural pest and disease detection large model construction method.

[0175] It should be noted that those skilled in the art will appreciate that all or part of the steps implemented in the embodiments of the present invention can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. When implemented using hardware, the steps can be implemented in whole or in part as purchased standard components or modified parts. When implemented using software, the steps can be implemented in whole or in part as a computer program product. The computer program product comprises one or more computer instructions. When loaded and executed on a computer, the steps or functions described in the embodiments of the present application are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0176] In summary, this invention addresses the unprofessionalism, difficulty in deployment, and hallucinations that plague traditional large-scale models in the field of agricultural pest and disease identification and detection. It provides a lightweight large-scale agricultural pest and disease detection model construction method driven by multimodal data distillation. This method comprises a large-scale model data distillation module M1, a dataset processing module M2, a large-scale model quantization module M3, and a large-scale model inference module M4, which together form a large-scale agricultural pest and disease detection model system. This invention utilizes a multimodal large-scale model-based approach to better capture the information between images and text. The use of data distillation and quantization reduces the number of model parameters while preserving the model's detection accuracy and generalization capabilities. This allows the use of high-quality large-scale agricultural pest and disease detection models even in low-resource (CPU) environments, facilitating their dissemination and use in agriculture, and possesses significant and far-reaching implications.

[0177] It should be understood that the examples and implementation methods described herein are for illustrative purposes only and are not intended to limit the present invention. Those skilled in the art may make various modifications or changes based on them. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a large agricultural pest and disease detection model using multimodal data distillation, characterized by: The following steps are involved: Step 1: Collect and organize the agricultural pest and disease image-text paired datasets, and pre-process them to obtain the reserved datasets S1, S11, and S12; Step 2: Use the base large model to distinguish the data RS1 that can be answered correctly from the data IS1 that cannot be answered correctly from S11. Finally, the pre-reserved dataset S3 is obtained by scoring the large model. Then, the DPO algorithm is fine-tuned on the pre-reserved dataset S3 using the base large model to obtain the honest large model with binary classification judgment. At the same time, the datasets S1 and S12 with long thought chains are merged to obtain the pre-reserved dataset S2. LoRA is fine-tuned based on the honest large model to obtain the data distillation large model. Step 3: Determine the quantization parameters and quantization algorithm of the data distillation large model, and perform quantization-aware training optimization and accuracy calibration to obtain the quantized large model; Step 4: Standardize the user input information or the images automatically captured by the camera, use the quantitative large model for preliminary reasoning, and then perform binary classification to obtain the final answer; The Step 2 further includes: Step 21: Question the pre-reserved dataset S11 via the large base model to ensure the quality of the dataset; Step 22: Save the dataset corresponding to the questions answered incorrectly by the base model as IS1 and add the label "label": 0 to the dataset. Save the dataset corresponding to the questions answered correctly by the base model as RS1 and add the label "label": 1 to the dataset. This yields a dataset S3 with binary classification labels. Using the DPO algorithm, train an honest large model for binary classification to distinguish between knowledge that the base model can answer and knowledge that it cannot answer. Step 23: Merge the pre-reserved datasets S1 and S12 to form the complete agricultural pest and disease image-text pairing dataset S2 with long thought chains for the second stage of LoRA fine-tuning. Step 24: Adopt the improved LoRA low-rank adapter fine-tuning method. By adding a multi-head low-rank weight matrix to the attention layer of the frozen honest large model's Transformer, we adapt it to the downstream task of the agricultural pest and disease image-text pairing dataset. The low-rank matrix is ​​decomposed into multiple heads, with each head focusing on learning features of different dimensions in the agricultural pest and disease image-text data, to further improve the low-rank matrix's ability to extract information granularity. In Step 22, the loss function expression of the DPO algorithm is: In the above formula, x is the problem input of the pre-reserved dataset S3, exp is the exponential function, log is the logarithmic function, and y is the preferred It is RS1,y rejected is IS1, β is the average score obtained by using the large model to score the pre-reserved dataset S11, which is between 0 and 10. The RS1 data score is above 5 points, and the IS1 data score is below 5 points.

2. The method for constructing a large model for agricultural pest and disease detection based on multimodal data distillation according to claim 1, characterized in that: The Step 1 further includes: Step 11. Data Collection and Organization: Because text is easier to obtain comprehensively, we first obtained the text names of agricultural pests and diseases through various channels. We then organized the dataset using a two-layer labeling system. The bottom layer was based on the biological classification system, while the top layer was based on practical agricultural production classification standards. This enabled multi-dimensional, standardized data classification and storage. Entity similarity filtering and manual review were used to remove duplicates and obtain unambiguous text names of agricultural pests and diseases. Step 12: Image-text association expansion: The collected agricultural pest and disease text names are used as entities to search for keywords in professional image databases and internet resources for corresponding images. The large-scale model is first used to determine whether the pest and disease name and image match each other, and the image-text datasets with successful matching are selected. The large-scale model is then used to score the image-text datasets that fall below a threshold of 5. Finally, through manual screening, the image-text matching datasets that are considered to be correct are selected, and the high-quality image-text datasets are retained. Step 13. Large model-guided completion: Design a standardized information template, which is used to clarify the key fields that need to be supplemented in the graphic data of agricultural pests and diseases, and input the graphic data into the large model with deep thinking. The standardized information template guides the large model to carry out targeted completion of the missing or incomplete content in the graphic data, and generate the corresponding reasoning process data set as the long thinking chain data, which together constitute and are saved as the pre-reserved data set S1. At the same time, in order to obtain a diverse and high-quality data set, S1 is used as the main data set, and no more than 15% of the dictionary-like data pairs are selected as question-answer pairs to be generated again, and generated twice respectively to obtain the pre-reserved data sets S11 and S12.

3. The method for constructing a large model for agricultural pest and disease detection based on multimodal data distillation according to claim 1, characterized in that: In Step 24, the expression of the multi-head low-rank weight matrix is: The forward propagation method of the multi-head low-rank weight matrix to change the model is: In the above two formulas, A j and B i Represent multi-head low-rank matrices, i and j represent the number of multi-head low-rank matrices, h represents the maximum number of multi-head low-rank matrices, x represents the input features of the large model with parameters > 500B after removing data with scores below 5, W represents the frozen honest large model, ΔW represents the updated multi-head low-rank weight matrix, and f is the output feature of the large agricultural pest and disease model after LoRA fine-tuning; Perform image preprocessing and normalization on the user input feature x that is input to the quantized large model for inference to increase the generalization of the model. The steps are as follows: Image preprocessing operations include resizing and denoising; Resizing: The input image is uniformly scaled to the input size H×W specified by the model. Using the bilinear interpolation algorithm, for the pixel at the coordinate (x, y) in the input image, the pixel value I(u, v) at the corresponding position (u, v) in the target image is calculated as follows: In the above three formulas, W src 、H src 、W dst and H dst Represent the width and height of the input large model image and the width and height of the target image during inference, respectively. (x i ,y j ) are the four neighboring pixels around (u,v), w i and w j are the linear weights in the horizontal and vertical directions calculated based on the pixel coordinate distance; Denoising: To address the noise that may be generated during the shooting process, a median filter algorithm is used. For each pixel point (x, y) in the image, the median of all pixel values ​​within an n×n neighborhood is taken as the output value of the point, effectively removing impulse noise. Normalization operation: Normalize the preprocessed image and map the pixel value range to the interval required by the model. In order to map to [-1, 1], the linear normalization formula is used: In the above two formulas, I (x, y) is the pixel value of the pixel point in the resized image, min (I) is the minimum pixel value of the resized image, max (I) is the maximum pixel value of the resized image, and I norm (x, y) is the pixel value finally normalized to the interval [-1, 1]; Through the above standardization processing, the differences in size and noise between images taken by different devices are eliminated, and the processed information is used as the input feature x input information of the base large model.

4. The method for constructing a large model for agricultural pest and disease detection using multimodal data distillation according to claim 1, wherein: The Step 3 further includes: Step 31. Determine quantization parameters: Quantization parameters determine the numerical accuracy of model weights and activation values, directly affecting model size, inference speed, and accuracy loss. Step 32. Determine the quantization granularity: The quantization granularity determines the granularity of grouping and quantizing the model weights, affecting the accuracy loss and compression efficiency. Step 33. Determine the quantization algorithm. To alleviate the accuracy loss caused by quantization, adopt the algorithm of layer-by-layer quantization + quantization-aware training optimization + accuracy calibration.

5. The method for constructing a large-scale model for agricultural pest and disease detection using multimodal data distillation according to claim 1, characterized in that: The Step 4 further includes: Step 41: Standardize input data such as text or image information input by the user, or real-time images of the current crop status captured by the camera, to ensure that the input data meets the input requirements of the quantitative large-scale model; Step 42: Input normalization is performed on the normalized data. When the quantitative large model receives the input processed data and generates reasoning and correct answers, it first performs a binary classification judgment to determine whether the question can be answered correctly by itself. If the model can answer correctly, it directly infers the corresponding correct answer to guide the user. For questions that the quantitative large model answers incorrectly, it is necessary to call an external API or provide suggested measures rather than direct methods to guide the user.

6. A multimodal data distillation agricultural pest and disease detection large model system, applicable to the multimodal data distillation agricultural pest and disease detection large model construction method according to any one of claims 1 to 5, characterized in that: include: Large model data distillation module M1, which is used to collect data sets, build an agricultural pest and disease database, and conduct image-text matching questions on the large model to construct pre-reserved data sets S1, S11 and S12; The data set processing module M2 is connected to the large model data distillation module M1 and inputs data to the data set processing module M2. The data set processing module M2 uses the base large model to perform binary classification judgment on the pre-reserved data set S11, obtains the data sets IS1 and RS1 with long thought chains, and uses IS1 and RS1 for DPO fine-tuning of the honest large model of the large model data distillation module M1; The large model quantization module M3 is connected to the data set processing module M2 and inputs data to the large model quantization module M3. The large model quantization module M3 is used to determine the quantization parameters and quantization algorithm, and perform quantization perception training optimization and accuracy calibration to obtain a quantized large model; The large model reasoning module M4 is connected to the data set processing module M2 and the large model quantization module M3, and is used to standardize the data input into the quantitative large model, and then introduce a binary classification judgment mechanism to complete the reasoning and answer of the quantitative large model.

7. The multimodal data distillation agricultural pest detection large model system according to claim 6, characterized in that: In the large model reasoning module M4, the quantitative large model first performs classification judgment based on the binary classification judgment mechanism introduced by the preference answer to determine whether the large model can correctly solve the problem, and then selects the answer method. For problems that the quantitative large model can solve using its own knowledge, direct reasoning is used to obtain the correct answer. For problems that the quantitative large model cannot solve using its own knowledge, external APIs are called or suggested measures are given instead of direct methods to assist in reasoning and obtain the best answer.

8. The multimodal data distillation agricultural pest detection large model system according to claim 6, characterized in that: The large model uses a general large model with a parameter size greater than 500B, and the base large model uses an open source pre-trained model with a parameter size less than 32B.

Images