Agricultural pest control method fusing computer vision and language model

Abstract

The invention belongs to the technical field of crop disease and insect pest control, and particularly relates to an agricultural disease and insect pest control method fusing computer vision and a language model. Based on a vision model and an optimized language model, after the vision model and the optimized language model are integrated, dynamic scene information can be combined; the method can identify various crop diseases in real time and give more professional and detailed suggestions in the field of crop disease control, has good disease identification ability and control suggestion generation ability, further significantly improves the efficiency of disease control in agricultural production, reduces the labor cost, and is beneficial to improving the yield and quality of crops.

Description

Technical Field

[0001] The present invention belongs to the technical field of crop disease and pest control, and specifically relates to an agricultural disease and pest control method integrating computer vision and language models. Background Art

[0002] In the past, the judgment of pests and diseases mainly relied on manual experience and expert knowledge, but this method has the limitations of strong subjectivity, low accuracy, high cost and low efficiency. Manual methods often require a lot of time and money resources, and it is difficult to make effective responses in a timely manner. The best time for control is often missed, which may eventually aggravate the spread and harm of pests and diseases. With the rapid development of artificial intelligence technology, pest and disease detection based on image recognition has gradually become an efficient and accurate solution. With the help of computer vision and deep learning technology, the types of pests and diseases can be quickly and accurately identified, greatly improving the efficiency and accuracy of detection. At the same time, large language models (such as the GPT series) have demonstrated excellent capabilities in general fields because they can answer questions instantly.

[0003] However, on the one hand, the existing pest and disease identification methods have good detection effects under laboratory conditions or relatively simple backgrounds, but under complex field conditions (such as different lighting, complex backgrounds, etc.), the recognition accuracy of diseases is significantly reduced. At the same time, such methods lack the ability to provide real-time, targeted disease prevention and control recommendations that match the identification results. On the other hand, although large language models can answer questions instantly, their application in the field of agricultural disease prevention and control is still insufficient due to the lack of multi-field information interaction capabilities combined with domestic actual needs (including climate change, crop varieties, soil conditions, and pest and disease characteristics). The answers of existing models are often not accurate and professional enough, and the solutions provided lack detail and operability, making it difficult to meet actual application needs. Summary of the invention

[0004] In order to solve the problems raised by the background technology, the present invention provides an agricultural pest control method integrating computer vision and language models.

[0005] The technical solution of the present invention is as follows: The present invention provides an agricultural pest control method integrating computer vision and language model, comprising the following steps: S1: Obtain image data of crop pests and diseases leaves, and obtain preprocessed image data sets after cleaning and labeling; S2: After the preprocessed image data is subjected to preliminary convolution, several deep convolution processes are performed to obtain a high-level feature map; the high-level feature map is sequentially subjected to global average pooling and full connection processing to obtain the health status and severity of the crop; the health status and severity of the crop are processed by the classification function to obtain the first pest category and the first visual model; the first visual model is trained using the image data in the image data set until the weighted focal loss function of the dynamic weight adjustment mechanism reaches a preset condition, thereby obtaining a second visual model; The deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed; S3: The publicly available multi-source text data in the agricultural field is used as a data set to guide the language model for unsupervised pre-training to obtain a pre-trained language model; S4: Obtain crop pest and disease control data, divide the text data of crop varieties, pest and disease categories, and pesticide types, and construct a question-answering text dataset with crop varieties and pest and disease categories as questions and pesticide types as answers; S5: The question-answer text data is imported into the pre-trained language model. In the attention layer and feed-forward layer, the question-answer text is converted into the main component and the noise component. The main component is inserted into the low-rank matrix by the LoRA method. The low-rank matrix is ​​decomposed by efficient parameter decomposition and kernel decomposition. The low-rank update is performed by combining the distribution modeling technology. After training the pre-trained language model, the target language model is obtained. S6: Obtain images of leaves of crop pests to be controlled, and guide them to the second visual model to obtain the pest category; According to the categories of pests and diseases, the similarity retrieval mechanism is used to obtain external related data, and the multi-source text data, crop pest and disease control data and external related data are processed by the attention mechanism to obtain fused data; The fused data is processed by the target language model to generate recommendations for agricultural pest control.

[0006] In step S2, the deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed, and the operation is as follows: After the feature to be processed is subjected to depthwise separable convolution, a first feature map is obtained; Based on the HSV color space, the first feature map is processed by channel attention to obtain a channel attention map, and the channel attention map is multiplied by the first feature map to obtain a second feature map; After the second feature map is axially divided, spatial attention processing is performed to obtain a spatial attention map, and the spatial attention map is multiplied with the second feature map to obtain a third feature map; After the third feature map is added to the feature to be processed, it is processed by the reverse attention mechanism to obtain the fourth feature map; The fourth feature map is connected to the feature to be processed via a skip residual gating to obtain a processed feature map.

[0007] In step S2, the first visual model is trained using the image data in the image data set until the weighted focus loss function of the dynamic weight adjustment mechanism reaches a preset condition, thereby obtaining a second visual model, specifically: After the image data in the image data set is processed by the first visual model, the probability distribution is obtained according to the formula: , calculate the category weight; In the formula, is the category weight, For Category The number of samples of crops, Used to smooth out excessive weights of small categories; In each training batch, the sample weight of each sample is calculated according to the dynamic weight adjustment mechanism; According to the formula: , calculate the loss value; In the formula, is the loss value, For sample The sample weights of is the category weight, is the probability of a positive sample, is the focus factor, which is used to control the degree of attention paid to difficult samples; Until the loss value reaches the preset condition, the second visual model is obtained.

[0008] In step S2, the high-level feature graph is processed by global average pooling and full connection in sequence to obtain the health status and severity of the crop. The health status and severity of the crop are processed by the classification function to obtain the category of pests and diseases, specifically: The feature map of each channel is reduced to a single average value through global average pooling, and the feature map is converted into a global feature vector representation. The global feature vector is fully connected to obtain a vector with the same number of categories, which is recorded as the target vector. According to the length of the target vector, the health status and severity of the crop are obtained; The target vector is processed by the classification function and converted into a probability distribution, and the category corresponding to the highest probability is taken as the pest category.

[0009] In step S5, after training, the target language model is obtained, specifically: Calculate the BLEU score and ROUGE score of the trained language model, and use the language model with a BLEU score greater than the BLEU score threshold and a ROUGE score greater than the ROUGE score threshold as the target language model.

[0010] In step S3, the multi-source text data disclosed in the agricultural field includes: Data on pest and disease knowledge base, crop growth patterns, climate impact analysis, and soil nutrient management.

[0011] In step S4, the crop pest control data includes: Professional text data on pest and disease knowledge, crop management, and pesticide use guidance.

[0012] In step S6, the external source related data includes: Meteorological data, field sensor data, farmer consultation records, and data from public agricultural databases.

[0013] The present invention also provides an agricultural pest control system integrating computer vision and language models, comprising: Image dataset construction module: used to obtain image data of crop pests and diseases leaves, and obtain pre-processed image datasets after cleaning and labeling; Pest and disease identification module: After the preprocessed image data is subjected to preliminary convolution, it is subjected to several deep convolution processes to obtain a high-level feature map; the high-level feature map is sequentially subjected to global average pooling and full connection processing to obtain the health status and severity of the crop; the health status and severity of the crop are processed by the classification function to obtain the first pest and disease category and the first visual model; the image data in the image data set is used to train the first visual model until the weighted focal loss function of the dynamic weight adjustment mechanism reaches the preset condition, thereby obtaining the second visual model; The deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed; Language model pre-training module: It uses the publicly available multi-source text data in the agricultural field as a data set, imports it into a language model, performs unsupervised pre-training, and obtains a pre-trained language model; Question-answering text dataset construction module: used to obtain crop pest control data, divide the text data of crop varieties, pest categories, and pesticide types, and construct a question-answering text dataset with crop varieties and pest categories as questions and pesticide types as answers; Language model training module: used to import question and answer text data into the pre-trained language model. In the attention layer and feedforward layer, the question and answer text is converted into main components and noise components. The main components are inserted into the low-rank matrix by the LoRA method. The low-rank matrix is ​​decomposed by efficient parameter decomposition and kernel decomposition. The low-rank update is performed in combination with the distribution modeling technology. After training the pre-trained language model, the target language model is obtained. Prevention and control suggestion generation module: used to obtain images of the leaves of the crop pests to be controlled, and guide them to the second visual model to obtain the pest category; According to the categories of pests and diseases, the similarity retrieval mechanism is used to obtain external related data, and the multi-source text data, crop pest and disease control data and external related data are processed by the attention mechanism to obtain fused data; The fused data is processed by the target language model to generate recommendations for agricultural pest control.

[0014] Beneficial effects: The present invention is based on a visual model and an optimized language model. After integrating the two, it can combine dynamic scene information, identify a variety of crop diseases in real time and give more professional and detailed suggestions in the field of crop disease prevention and control. It has good disease recognition capabilities and prevention and control suggestion generation capabilities, thereby significantly improving the efficiency of disease prevention and control in agricultural production, reducing labor costs, and helping to increase crop yield and quality.

[0015] During the recognition and classification process, the present invention not only uses a multi-dimensional attention mechanism to focus on detailed features such as morphology, texture, and color, and a reverse attention mechanism to perform feature enhancement to enhance recognition accuracy; it also introduces a weighted focal loss function with a dynamic weight adjustment mechanism, which can not only give higher weights to samples that are difficult to classify, but also automatically adjust the weight distribution according to the dynamic changes in the category distribution, thereby more effectively processing rare disease categories with a small number of samples in the data set.

[0016] The present invention integrates multi-source information training language models and the target language model obtained after BLEU and ROUGE score evaluation, which can better understand and generate professional terms and content in the field of agricultural disease prevention and control, thereby enhancing the practicality and accuracy of the generated text.

[0017] The present invention provides rich contextual support for the generation of target language models by real-time integration of multiple sources of agriculture-related information, thereby optimizing the adaptability and responsiveness of the model in dynamic scenarios; weighted combination of exogenous real-time data and internal knowledge data of the target language model enables more comprehensive consideration of different factors when generating recommendations, thereby improving the scientific nature and operability of the recommendations and generating customized prevention and control plans for specific scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of a method for controlling agricultural pests and diseases by integrating computer vision and language models in an embodiment.

[0019] Figure 2 This is a schematic diagram of the detection results of the second visual model of this application for agricultural pests and diseases categories.

[0020] Figure 3The performance comparison chart of the target language model of this application and other language models, where (a) is the BLEU score and (b) is the ROUGE score. DETAILED DESCRIPTION

[0021] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0022] The present invention provides an agricultural pest control method integrating computer vision and language models, such as Figure 1 As shown, the following steps are included: S1: Obtain image data of crop pests and diseases leaves, and obtain a preprocessed image dataset after cleaning and labeling.

[0023] During the data acquisition phase, we systematically collected image data of leaves and pests of various crop diseases in response to diverse conditions in complex environments. The data sources cover a variety of actual scenarios. Specifically, they include: shooting of complex field conditions such as leaf overlap and uneven lighting, high-precision shooting in the laboratory, field scene collection, and supplementation from public data sets on the Internet, ensuring the breadth and representativeness of the data. The collected data also covers images of different pixel resolutions, thereby improving the adaptability of the visual model to diverse inputs.

[0024] In order to further enrich the diversity and feature distribution of the images, a series of image enhancement techniques were used, such as flipping, translation, rotation and color adjustment. After systematic sorting, a total of 48 disease images of 13 crops and images of healthy leaves of 13 crops were collected. In addition, 30 insect pest images of 13 crops were also collected. These include: Healthy apple leaves, apple leaves with ring rot, apple leaves with black rot, apple leaves with brown spot, apple leaves with gray spot, apple leaves with mosaic, apple leaves with powdery mildew, and apple leaves with rust; Healthy cherry leaves, cherry leaves with powdery mildew; Healthy citrus leaves and citrus leaves infected with Huanglongbing; Healthy corn leaves, corn leaves with common rust, corn leaves with gray leaf spot, and corn leaves with northern leaf blight; Healthy grape leaves, grape leaves with leaf spot disease, grape leaves with mosaic virus disease, grape leaves with yellowing disease, grape leaves with black rot disease, grape leaves with downy mildew disease, grape leaves with powdery mildew disease, and grape leaves with black spot disease; Healthy peach leaves, peach leaves with bacterial spot disease; Healthy pepper leaves, pepper leaves with scab disease, and pepper leaves with bacterial spot disease; Healthy potato leaves, potato leaves with late blight; Healthy rice leaves, rice leaves with brown spot disease, rice leaves with bacterial leaf streak disease, rice leaves with thrips disease, rice leaves with rice blast disease, and rice leaves with yellow dwarf disease; Healthy soybean leaves, soybean leaves with angular leaf spot disease, soybean leaves with bacterial leaf streak disease, soybean leaves with leaf spot disease, soybean leaves with downy mildew, soybean leaves with brown spot disease, soybean leaves with potassium deficiency, soybean leaves with rust disease, and soybean leaves with target leaf spot disease; Healthy strawberry leaves, strawberry leaves with leaf burn; Healthy tomato leaves, tomato leaves with bacterial spot disease, tomato leaves with early blight disease, tomato leaves with late blight disease, tomato leaves with leaf mold disease, tomato leaves with mosaic virus disease, tomato leaves with brown spot disease, and tomato leaves with yellow leaf curl virus disease; Healthy wheat leaves, wheat leaves with leaf rust, wheat leaves with leaf stripe disease, wheat leaves with stripe rust, and wheat leaves with loose smut.

[0025] Apple aphid, apple psyllid; Cherry fruit fly; Citrus aphids; Corn borer, corn weevil, corn leafhopper; Grape fruit borer, grape spider mite; Peach leaf roller, peach aphid, peach fruit fly; pepper aphids, pepper fruit flies, pepper mosquitoes; potato beetles, potato aphids, potato cutworms; Rice planthoppers and rice blast insects; soybean aphids, soybean grubs, soybean spider mites; Strawberry aphids, strawberry fruit flies, strawberry leafhoppers; Tomato aphids, tomato fruit flies; Wheat aphids and wheat mites.

[0026] Finally, a multi-source dataset containing more than 200,000 high-quality images was constructed, providing strong data support for model training and evaluation. This multi-dimensional, high-quality dataset laid the foundation for the robustness and generalization ability of the model in complex field environments.

[0027] Furthermore, the acquired image data is cleaned to remove blurry, incomplete or unrecognizable images. Image quality is tested through automated scripts and supplemented by manual review to ensure the accuracy and reliability of the data set.

[0028] In addition, the images are uniformly converted to the same size and pixel values ​​are normalized to keep the pixel value range of all images consistent. In addition, the images are accurately labeled to ensure the effectiveness of subsequent model training.

[0029] S2: After the preprocessed image data undergoes preliminary convolution, it undergoes several deep convolution processes to obtain a high-level feature map; the high-level feature map undergoes global average pooling and full connection processing in turn to obtain the health status and severity of the crop. The health status and severity of the crop are processed by the classification function to obtain the pest and disease category and the first visual model. The first visual model is trained using the image data in the image data set until the weighted focal loss function of the dynamic weight adjustment mechanism reaches the preset condition to obtain the second visual model.

[0030] Since the classification results of agricultural diseases and pests are directly related to the implementation of prevention and control measures, the visual model is required to have a high ability to perceive complex feature information, the ability to learn rare disease samples with fewer samples, and the ability to utilize tiny local and global information in complex field environments.

[0031] First, the preprocessed image is subjected to preliminary convolution to extract low-level features such as edges, textures, etc. At this point, the number of channels of the image will increase and the spatial dimension of the feature map will decrease.

[0032] Secondly, several times of deep convolution processing can extract more complex and advanced features, such as disease patterns, lesion shapes, insect body shapes, etc. In the feature extraction process, the spatial resolution of the image gradually decreases, while the number of feature channels gradually increases, and meaningful feature representations are gradually condensed and recorded as high-level feature maps.

[0033] Considering that the characteristics of pests and diseases are complex and diverse, for example, the morphology, texture and color characteristics of fruit diseases and leaf diseases are significantly different, and the characteristic differences between pests and diseases (such as appearance structure and color pattern) are even greater. In order to more effectively extract and distinguish these diverse features, preferably, the deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the processed features, and the operation is as follows: Step 1: After the features to be processed are subjected to depth-wise separable convolution, the first feature map is obtained.

[0034] Step 2: Based on the HSV (hue, saturation, and value) color space, the first feature map is processed by channel attention to obtain a channel attention map. The channel attention map is multiplied by the first feature map to obtain the second feature map.

[0035] In the channel dimension, the present invention introduces the HSV color space, which can more sensitively capture the saturation and brightness differences of colors, especially in the color changes of diseases (such as spots and yellowing). At the same time, the feature pyramid network (FPN) is combined to fuse channel information at different resolution levels, further enhancing the expression ability of channel features.

[0036] Step 3: After the second feature map is axially divided, spatial attention processing is performed to obtain a spatial attention map. The spatial attention map is multiplied by the second feature map to obtain the third feature map.

[0037] In the spatial dimension, the present invention focuses on the morphological detail features and processes the spatial features in a more fine-grained manner by subdividing the morphological information of different axes (such as aspect ratio and regional distribution), thereby capturing the morphological details of different diseases, such as the edge features of fruit rot or the distribution characteristics of leaf wormholes.

[0038] In addition, in order to more efficiently handle the huge feature differences between pests and diseases, a dynamic weight allocation strategy is also adopted in channel attention processing and spatial attention processing. By introducing dynamic convolution, the attention weight is automatically adjusted according to the features to be processed, and the texture features of leaf diseases or the morphological features of pests are adaptively focused, thereby improving the versatility and accuracy of the model.

[0039] Step 4: After the third feature map is added to the feature to be processed, it is processed by the reverse attention mechanism to obtain the fourth feature map.

[0040] In order to reduce the interference of complex background on feature extraction, the present invention introduces the reverse attention mechanism for feature enhancement. By suppressing the interference of the background area, the feature expression of the foreground area is enhanced, which is suitable for scenes with uneven field lighting or complex background.

[0041] Step 5: The fourth feature map is connected to the feature to be processed via skip residual gating to obtain the processed feature map.

[0042] In order to further improve the ability to capture local details and deep semantic information, the present invention introduces jump residual gating connections, which makes full use of the fine texture information contained in the low-level features (i.e., the features to be processed) and the semantic information in the high-level features (i.e., the fourth feature map), thereby realizing the effective integration of feature information. By introducing a gating mechanism in the jump residual connection, the fusion ratio of features at different layers is adaptively adjusted to make the information flow smoother, while avoiding the gradient vanishing problem that may occur in traditional deep networks. This structure not only improves the training depth of the network, but also significantly enhances the richness and accuracy of feature expression, which helps to capture subtle local changes and global semantic associations in pest and disease identification. In addition, the introduction of jump residual gating connections optimizes the training efficiency of the model. By reducing the interference of low-level redundant information, the network can focus more on key feature areas, thereby showing stronger robustness and stability in complex field scenes.

[0043] The present invention uses a multi-dimensional attention mechanism to focus on detailed features such as morphology, texture, and color, and a reverse attention mechanism to perform feature enhancement, thereby enhancing the recognition accuracy of the visual model.

[0044] Next, the high-level feature graph is processed by global average pooling and full connection in turn to obtain the health status and severity of the crops. The health status and severity of the crops are processed by the classification function to obtain the pest and disease categories, specifically: The feature map of each channel is reduced to a single average value through global average pooling, and the feature map is converted into a global feature vector representation. The global feature vector is fully connected to obtain a vector with the same number of categories, which is recorded as the target vector. The health status and severity of the crop are obtained based on the length of the target vector. For example, in leaf disease identification, the length of the target vector is 129, including the health status of 12 crops and the mild, moderate and severe severity of 39 diseases.

[0045] The target vector is processed by the classification function and converted into a probability distribution, and the category corresponding to the highest probability is taken as the pest category.

[0046] For example, using the Softmax activation function, the target vector is converted into a probability distribution. The output value of each category represents the probability that the image is predicted to be of that category. In the probability distribution, the category corresponding to the highest probability is the final prediction result. This indicates that the visual model believes that the input leaf image is most likely to belong to this category.

[0047] Finally, during the model training process, although the cross entropy loss function performs well in classification tasks, it is easy to lead to insufficient learning of rare categories when dealing with multi-classification problems with unbalanced category distribution, thereby affecting the overall accuracy. Preferably, the first visual model is trained using image data in the image dataset until the weighted focal loss function of the dynamic weight adjustment mechanism reaches a preset condition, and the second visual model is obtained, specifically: After the image data in the image data set is processed by the first visual model, the probability distribution is obtained according to the formula: , calculate the category weight; In the formula, is the category weight, For Category The number of samples of crops, Used to smooth out excessive weights of small categories; In each training batch, the sample weight of each sample is calculated according to the dynamic weight adjustment mechanism; According to the formula: , calculate the loss value; In the formula, is the loss value, For sample The sample weights of is the category weight, is the probability of a positive sample, is the focus factor, which is used to control the degree of attention paid to difficult samples; Until the loss value reaches the preset condition, the second visual model is obtained.

[0048] The present invention introduces a weighted focal loss function with a dynamic weight adjustment mechanism, which can not only assign higher weights to samples that are difficult to classify, but also automatically adjust the weight distribution according to the dynamic changes of the category distribution, thereby more effectively processing rare disease categories with a small number of samples in the data set and enhancing the stability and convergence speed during training.

[0049] In addition, during the training process, the present invention uses flooding technology. During the training process, when the loss is no longer decreasing, the loss function is allowed to oscillate at a set value. This can prevent the model from becoming confident due to the loss no longer decreasing, thereby further improving the recognition and classification capabilities of the model.

[0050] During the recognition and classification process, the present invention not only uses a multi-dimensional attention mechanism to focus on detailed features such as morphology, texture, and color, and a reverse attention mechanism to perform feature enhancement to enhance recognition accuracy; it also introduces a weighted focal loss function with a dynamic weight adjustment mechanism, which can not only give higher weights to samples that are difficult to classify, but also automatically adjust the weight distribution according to the dynamic changes in the category distribution, thereby more effectively processing rare disease categories with a small number of samples in the data set.

[0051] S3: The publicly available multi-source text data in the agricultural field is used as a data set, which is introduced into the language model for unsupervised pre-training to obtain a pre-trained language model.

[0052] Regarding the language model, it can be based on the ChatGLM3-6B model.

[0053] Furthermore, the multi-source text data disclosed in the agricultural field includes: Data on pest and disease knowledge base, crop growth patterns, climate impact analysis, and soil nutrient management.

[0054] The present invention uniformly preprocesses and encodes multi-source, heterogeneous data, and combines it with a dynamic context learning mechanism designed specifically for agricultural scenarios, so that the language model can more deeply understand the professional terms, field characteristics, and multi-scenario application requirements in agricultural disease prevention and control, thereby greatly improving the language model's ability to analyze and respond to complex agricultural problems. This method effectively breaks through the limitations of traditional unsupervised pre-training in professional field adaptability and has significant innovation and practical value.

[0055] S4: Obtain crop pest and disease control data, divide the text data of crop varieties, pest and disease categories, and pesticide types, and construct a question-answering text dataset with crop varieties and pest and disease categories as questions and pesticide types as answers.

[0056] First, the crop pest control data include: Professional text data on pest and disease knowledge, crop management, and pesticide use guidance can be obtained from authoritative resources in the agricultural field such as professional books such as "High-efficiency Grape Cultivation and Pest and Disease Control with Pictures (Second Edition)".

[0057] Then, after careful cleaning, the data is divided according to crop varieties, pest and disease categories, and pesticide types to ensure the high quality and pertinence of the pre-trained language model training corpus. The ChatGLM3-6B model can then be used to generate a Q&A format dataset that meets the requirements of the target language model, using a JSON structure to standardize the question-answer pair format, including instruction (question instructions), input (input content), and output (standard answer) to ensure the structured and scalable nature of the question-answer data.

[0058] S5: The question and answer text data is imported into the pre-trained language model. In the attention layer and the feed-forward layer, the question and answer text is converted into principal components and noise components. The principal components are inserted into the low-rank matrix by the LoRA method. The low-rank matrix is ​​decomposed by efficient parameter decomposition and kernel decomposition, and the low-rank update is performed in combination with the distribution modeling technology. After training the pre-trained language model, the target language model is obtained.

[0059] For example, based on the ChatGLM3-6B model, if the distribution characteristics of the question-answer text are not fully considered during low-rank updates, noise interference may be introduced. Therefore, this application introduces the LoRA fine-tuning method (Efficient Decomposition and Distribution Modeling LoRA) improved by efficient parameter decomposition and distribution modeling technology in the attention layer and feedforward layer of the pre-trained ChatGLM3-6B model, converting the question-answer text (i.e., input features) into main components and noise components, and only performing low-rank updates on the main components, thereby significantly reducing the impact of noise and improving the model's computational efficiency, storage efficiency, and adaptability to task-specific features. Regarding distribution modeling technology, LoRA's parameter fine-tuning can change dynamically according to the input distribution, thereby improving the generalization ability for different tasks and features.

[0060] At the same time, the traditional matrix decomposition in the LoRA method is replaced by the kernel decomposition method. For example, in the attention layer, the low-rank matrix is ​​decomposed into multiple smaller kernel matrices, which reduces parameter redundancy and retains the interaction information between different dimensions. In the feedforward layer, the low-rank matrix of the fully connected layer in the feedforward network is kernel decomposed to achieve parameter compression and improve the model's ability to capture complex patterns of input features.

[0061] The present invention fine-tunes the language model, which not only improves the computational efficiency, but also optimizes the quality of feature representation, so that the model exhibits higher convergence speed and stability during the fine-tuning process, and enhances the adaptability of the target language model to feature distribution.

[0062] After training, the target language model can accurately generate recommendations containing key information such as pesticide name, dosage and time of use, while ensuring that the output content is logical and fluent in language, fully meeting the professional requirements of the agricultural field.

[0063] Furthermore, in order to ensure that the target language model exhibits excellent professionalism and details when generating disease prevention and control suggestions, in step S5, after training, the target language model is obtained, specifically: Calculate the BLEU score and ROUGE score of the trained language model, and use the language model with a BLEU score greater than the BLEU score threshold and a ROUGE score greater than the ROUGE score threshold as the target language model.

[0064] Among them, the BLEU score is used to measure the similarity between the suggestions generated by the target language model and the reference text, focusing on evaluating the accuracy of key details such as pesticide names and dosages; the ROUGE score focuses on evaluating the coverage and diversity of the text generated by the target language model to ensure that the generated suggestions cover the core information and key points in the reference text.

[0065] By combining unsupervised pre-training and fine-tuning methods with high-quality data, the target language model obtained by the present invention not only significantly improves the professionalism and practicality of disease prevention and control suggestion generation, but also demonstrates superior generation capability and domain adaptability.

[0066] S6: Obtain images of leaves of crop pests to be controlled, and guide them to the second visual model to obtain the pest category; According to the categories of pests and diseases, the similarity retrieval mechanism is used to obtain external related data, and the multi-source text data, crop pest and disease control data and external related data are processed by the attention mechanism to obtain fused data; The fused data is processed by the target language model to generate recommendations for agricultural pest control.

[0067] In order to improve the model's adaptability to dynamic scenarios and enhance its practicality in actual agricultural scenarios, the specific operations for generating prevention and control recommendations are as follows: First, the second vision model is used to recognize images of leaves of crop pests and diseases to be controlled, accurately identify the type of crop disease or pest and its severity, and generate structured text information, including detailed descriptions of the disease type, distribution range, and degree of damage.

[0068] Secondly, in order to further improve the intelligence and professional performance of the target language model in the field of agricultural disease prevention and control, dynamic retrieval is performed to extract relevant information from exogenous data in real time, the retrieved exogenous data is integrated with the model's internal knowledge data, and the attention mechanism is used to dynamically weight the importance of different information sources to ensure the completeness and accuracy of the generated content.

[0069] In step S6, the external source related data includes: Meteorological data, field sensor data, farmer consultation records, and data from public agricultural databases.

[0070] Among them, meteorological data, such as real-time weather information (such as temperature and humidity, rainfall, wind speed, etc.) and historical climate records, help the model generate prevention and control recommendations that match current environmental conditions; Field sensor data, such as crop canopy reflectivity, soil moisture, nitrogen content, etc., provide accurate environmental data support for pest and disease identification and control strategies.

[0071] Data from farmer consultation records and public agricultural databases, such as photos and descriptions of pests and diseases submitted by farmers, and public data related to agriculture (such as pesticide registration information, crop growth cycle parameters, etc.), enhance the practicality of the model in answering specific questions.

[0072] The present invention provides rich contextual support for the generation of target language models by real-time integration of multiple sources of agriculture-related information, thereby optimizing the adaptability and responsiveness of the model in dynamic scenarios; weighted combination of exogenous real-time data and internal knowledge data of the target language model enables more comprehensive consideration of different factors when generating recommendations, thereby improving the scientific nature and operability of the recommendations and generating customized prevention and control plans for specific scenarios.

[0073] Then, using the target language model that integrates multi-source data, the specific operations for generating prevention and control recommendations are as follows: The model is contextualized through instruction, positioning it as a professional consultant in the agricultural field. In the input, enter the text information generated by the disease classification module (such as "My crop currently suffers from [disease type / pest type]"), and can also attach user-supplied planting environment information, such as climate conditions, soil type, or historical disease records. The model will convert semantic information into numerical vectors, perform deep semantic analysis on the input content, and infer and generate prevention and control recommendations.

[0074] Prevention recommendations include: (1) Provide basic prevention and control recommendations including pesticide name, specific usage method and dosage according to the disease type and severity.

[0075] (2) Combined with environmental variables input by the user (such as current climate conditions, planting density, etc.), a more practical control plan is generated, such as recommending biological control methods or climate-adaptive pesticides suitable for the current environment.

[0076] (3) The suggestions are not limited to the use of pesticides, but also include specific agricultural management strategies, such as how to adjust the frequency of irrigation, improve ventilation conditions or apply appropriate amounts of fertilizers to inhibit the spread of diseases and create an ecological environment that is not conducive to the growth of diseases.

[0077] The control method provided by the present invention can update the recommended content in real time according to the dynamic information provided (such as climate change, new farmland disease conditions, etc.), ensuring that the generated plan is always scientific and targeted. For example, emergency control measures will be generated for the risk of disease outbreaks after continuous rainfall; and methods of early deployment are recommended for seasonal high incidence of insect pests.

[0078] By using the method provided by the present invention, the results are displayed through an interactive interface, and users can view the identification results and prevention and control suggestions in real time to obtain convenient decision support. Users can also supplement or modify input information in the interface, such as updating the disease status or providing more detailed environmental data, and regenerate suggestions based on the new information to achieve intelligent interaction. The prevention and control suggestions can be saved in text or table form, which is convenient for users to record and use later. It can also further suggest historical management, support users to browse and retrieve historical records, and facilitate long-term planting management and decision-making.

[0079] The present invention also provides an agricultural pest control system integrating computer vision and language models, comprising: Image dataset construction module: used to obtain image data of crop pests and diseases leaves, and obtain pre-processed image datasets after cleaning and labeling; Pest and disease identification module: After the preprocessed image data is subjected to preliminary convolution, it is subjected to several deep convolution processes to obtain a high-level feature map; the high-level feature map is sequentially subjected to global average pooling and full connection processing to obtain the health status and severity of the crop; the health status and severity of the crop are processed by the classification function to obtain the first pest and disease category and the first visual model; the image data in the image data set is used to train the first visual model until the weighted focal loss function of the dynamic weight adjustment mechanism reaches the preset condition, thereby obtaining the second visual model; The deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed; Language model pre-training module: It uses the publicly available multi-source text data in the agricultural field as a data set, imports it into a language model, performs unsupervised pre-training, and obtains a pre-trained language model; Question-answering text dataset construction module: used to obtain crop pest control data, divide the text data of crop varieties, pest categories, and pesticide types, and construct a question-answering text dataset with crop varieties and pest categories as questions and pesticide types as answers; Language model training module: used to import question and answer text data into the pre-trained language model. In the attention layer and feedforward layer, the question and answer text is converted into main components and noise components. The main components are inserted into the low-rank matrix by the LoRA method. The low-rank matrix is ​​decomposed by efficient parameter decomposition and kernel decomposition. The low-rank update is performed in combination with the distribution modeling technology. After training the pre-trained language model, the target language model is obtained. Prevention and control suggestion generation module: used to obtain images of the leaves of the crop pests to be controlled, and guide them to the second visual model to obtain the pest category; According to the categories of pests and diseases, the similarity retrieval mechanism is used to obtain external related data, and the multi-source text data, crop pest and disease control data and external related data are processed by the attention mechanism to obtain fused data; The fused data is processed by the target language model to generate recommendations for agricultural pest control.

[0080] Performance Testing 1. Identification performance of agricultural pests and diseases The second visual model of the present application is used to detect agricultural pests and diseases. Figure 2 It can be seen that the second visual model of the present application can not only distinguish between healthy and diseased crops, but also further identify the types of pests and diseases of different crops with high accuracy.

[0081] 2. Generate suggestions for agricultural pest control The target language model of this application (denoted as CDIP-ChatGLM3) is used to generate agricultural pest control suggestions, and its performance is compared with the language models in the prior art (ChatGLM3-6B, Qwen-max, Llama-3.1-405B-Instruct, GPT-4o), as shown in Figure 2. Figure 3 As shown, it can be seen that compared with the language model in the prior art, whether it is BLEU score or ROUGE score, the target language model of this application (CDIP-ChatGLM3) has excellent performance.

Claims

1. A method for controlling agricultural pests and diseases by integrating computer vision and language models, characterized in that: The following steps are involved: S1: Obtain image data of crop pests and diseases leaves, and obtain preprocessed image data sets after cleaning and labeling; S2: After the preprocessed image data is subjected to preliminary convolution, several deep convolution processes are performed to obtain a high-level feature map; the high-level feature map is sequentially subjected to global average pooling and full connection processing to obtain the health status and severity of the crop; the health status and severity of the crop are processed by the classification function to obtain the first pest category and the first visual model; the first visual model is trained using the image data in the image data set until the weighted focal loss function of the dynamic weight adjustment mechanism reaches a preset condition, thereby obtaining a second visual model; The deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed; S3: The publicly available multi-source text data in the agricultural field is used as a data set to guide the language model for unsupervised pre-training to obtain a pre-trained language model; S4: Obtain crop pest and disease control data, divide the text data of crop varieties, pest and disease categories, and pesticide types, and construct a question-answering text dataset with crop varieties and pest and disease categories as questions and pesticide types as answers; S5: The question-answer text data is imported into the pre-trained language model. In the attention layer and feed-forward layer, the question-answer text is converted into the main component and the noise component. The main component is inserted into the low-rank matrix by the LoRA method. The low-rank matrix is ​​decomposed by efficient parameter decomposition and kernel decomposition. The low-rank update is performed by combining the distribution modeling technology. After training the pre-trained language model, the target language model is obtained. S6: Obtain images of leaves of crop pests to be controlled, and guide them to the second visual model to obtain the pest category; According to the categories of pests and diseases, the similarity retrieval mechanism is used to obtain external related data, and the multi-source text data, crop pest and disease control data and external related data are processed by the attention mechanism to obtain fused data; The fused data is processed by the target language model to generate recommendations for agricultural pest control.

2. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S2, the deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed, and the operation is as follows: After the feature to be processed is subjected to depthwise separable convolution, a first feature map is obtained; Based on the HSV color space, the first feature map is processed by channel attention to obtain a channel attention map, and the channel attention map is multiplied by the first feature map to obtain a second feature map; After the second feature map is axially divided, spatial attention processing is performed to obtain a spatial attention map, and the spatial attention map is multiplied with the second feature map to obtain a third feature map; After the third feature map is added to the feature to be processed, it is processed by the reverse attention mechanism to obtain the fourth feature map; The fourth feature map is connected to the feature to be processed via a skip residual gating to obtain a processed feature map.

3. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S2, the first visual model is trained using the image data in the image data set until the weighted focus loss function of the dynamic weight adjustment mechanism reaches a preset condition, thereby obtaining a second visual model, specifically: After the image data in the image data set is processed by the first visual model, the probability distribution is obtained according to the formula: , calculate the category weight; In the formula, is the category weight, For Category The number of samples of crops, Used to smooth out excessive weights of small categories; In each training batch, the sample weight of each sample is calculated according to the dynamic weight adjustment mechanism; According to the formula: , calculate the loss value; In the formula, is the loss value, For sample The sample weights of is the category weight, is the probability of a positive sample, is the focus factor, which is used to control the degree of attention paid to difficult samples; Until the loss value reaches the preset condition, the second visual model is obtained.

4. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S2, the high-level feature graph is processed by global average pooling and full connection in sequence to obtain the health status and severity of the crop. The health status and severity of the crop are processed by the classification function to obtain the category of pests and diseases, specifically: The feature map of each channel is reduced to a single average value through global average pooling, and the feature map is converted into a global feature vector representation. The global feature vector is fully connected to obtain a vector with the same number of categories, which is recorded as the target vector. According to the length of the target vector, the health status and severity of the crop are obtained; The target vector is processed by the classification function and converted into a probability distribution, and the category corresponding to the highest probability is taken as the pest category.

5. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S5, after training, the target language model is obtained, specifically: Calculate the BLEU score and ROUGE score of the trained language model, and use the language model with a BLEU score greater than the BLEU score threshold and a ROUGE score greater than the ROUGE score threshold as the target language model.

6. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S3, the multi-source text data disclosed in the agricultural field includes: Data on pest and disease knowledge base, crop growth patterns, climate impact analysis, and soil nutrient management.

7. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S4, the crop pest control data includes: Professional text data on pest and disease knowledge, crop management, and pesticide use guidance.

8. The agricultural pest control method integrating computer vision and language model according to claim 1, characterized in that: In step S6, the external source related data includes: Meteorological data, field sensor data, farmer consultation records, and data from public agricultural databases.

9. An agricultural pest control system integrating computer vision and language model, characterized in that: include: Image dataset construction module: used to obtain image data of crop pests and diseases leaves, and obtain pre-processed image datasets after cleaning and labeling; Pest and disease identification module: After the preprocessed image data is subjected to preliminary convolution, it is subjected to several deep convolution processes to obtain a high-level feature map; the high-level feature map is sequentially subjected to global average pooling and full connection processing to obtain the health status and severity of the crop; the health status and severity of the crop are processed by the classification function to obtain the first pest and disease category and the first visual model; the image data in the image data set is used to train the first visual model until the weighted focal loss function of the dynamic weight adjustment mechanism reaches the preset condition, thereby obtaining the second visual model; The deep convolution processing includes sequentially performing channel space attention processing, reverse attention mechanism processing and skip residual gating connection on the features to be processed; Language model pre-training module: It uses the publicly available multi-source text data in the agricultural field as a data set, imports it into a language model, performs unsupervised pre-training, and obtains a pre-trained language model; Question-answering text dataset construction module: used to obtain crop pest control data, divide the text data of crop varieties, pest categories, and pesticide types, and construct a question-answering text dataset with crop varieties and pest categories as questions and pesticide types as answers; Language model training module: used to import question and answer text data into the pre-trained language model. In the attention layer and feedforward layer, the question and answer text is converted into main components and noise components. The main components are inserted into the low-rank matrix by the LoRA method. The low-rank matrix is ​​decomposed by efficient parameter decomposition and kernel decomposition. The low-rank update is performed in combination with the distribution modeling technology. After training the pre-trained language model, the target language model is obtained. Prevention and control suggestion generation module: used to obtain images of the leaves of the crop pests to be controlled, and guide them to the second visual model to obtain the pest category; According to the categories of pests and diseases, the similarity retrieval mechanism is used to obtain external related data, and the multi-source text data, crop pest and disease control data and external related data are processed by the attention mechanism to obtain fused data; The fused data is processed by the target language model to generate recommendations for agricultural pest control.

Images