Agricultural pest diagnosis and control method, device, electronic equipment and storage medium

Abstract

The application provides an agricultural pest diagnosis and control method and device, electronic equipment and storage medium, and relates to the technical field of agriculture.The method comprises the following steps: extracting image features, shooting time and shooting positions from crop pest images of a target area; inputting the image features into a multi-modal recognition model to obtain a first diagnosis result output by the multi-modal recognition model; inputting the shooting time and territorial environment data corresponding to the shooting positions into a multi-source data mining model to obtain a second diagnosis result output by the multi-source data mining model; and inputting the first diagnosis result and the second diagnosis result into a pest knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions.The application realizes the precision, interpretability of pest diagnosis and the individualization of control schemes by means of multi-modal information fusion, step-by-step reasoning technology and in combination with the pest knowledge graph.

Description

Technical Field

[0001] This invention relates to the field of agricultural technology, and in particular to a method, apparatus, electronic device, and storage medium for the diagnosis and control of agricultural pests and diseases. Background Technology

[0002] Currently, the diagnosis of agricultural pests and diseases mainly relies on farmers' experience or on-site inspections by agricultural technicians, resulting in low diagnostic efficiency, strong subjectivity, and a high rate of misjudgment. Existing intelligent diagnostic systems mostly output diagnostic results directly based on traditional machine learning models, lacking interpretability of the reasoning process and failing to fully integrate dynamic factors such as crop growth environment and agricultural operations, leading to a disconnect between prevention and control decisions and actual needs. Therefore, how to achieve accurate and interpretable diagnosis and personalized prevention and control has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, and storage medium for the diagnosis and control of agricultural pests and diseases, which solves the problems of low accuracy and uninterpretability of agricultural pest and disease diagnosis and lack of personalization of control schemes in the prior art, and realizes accurate and interpretable diagnosis and personalized control of agricultural pests and diseases.

[0004] This invention provides a method for diagnosing and controlling agricultural pests and diseases, comprising the following steps: Extract image features, shooting time, and shooting location from crop disease and pest images of the target area; The image features are input into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on the type of pest and disease and its disease pattern. The shooting time and the local environmental data corresponding to the shooting location are input into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information. The first and second diagnostic results are input into the pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions.

[0005] According to the present invention, a method for diagnosing and controlling agricultural pests and diseases includes inputting the first diagnostic result and the second diagnostic result into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management recommendations, comprising: Based on the shooting location and shooting time, the geographical area to be diagnosed, the crop growth period, and future weather forecast are determined to form a spatiotemporal anchor point; Using the aforementioned spatiotemporal anchor points as constraints, a search is performed in the pest and disease knowledge graph to obtain pest and disease entities, environmental condition entities, and prevention and control scheme entities related to the current spatiotemporal context, forming a dynamic subgraph; Match the pest and disease type in the first diagnostic result with the pest and disease entity in the dynamic subgraph to confirm the target pest and disease. Based on the crop status information and pest and disease status information in the second diagnostic result, the environmental condition entities and control scheme entities related to the target pest and disease in the dynamic sub-graph are screened and adapted. In the dynamic subgraph, the causal relationship chain connected to the target pest is traversed; Based on the screened and adapted environmental condition entities and prevention and control scheme entities, and combined with the causal relationship chain, the disease pattern information in the first diagnostic result and the pest and disease risk level in the second diagnostic result, plant protection and pesticide management suggestions adapted to the current spatiotemporal context and crop status are generated.

[0006] According to the method for diagnosing and controlling agricultural pests and diseases provided by the present invention, the multimodal recognition model is trained based on the following method: Construct a training dataset; the training dataset includes multiple data pairs, each data pair including an image sample and a text sample; the image sample is a crop pest and disease image containing a natural field background; the text sample is a structured descriptive text corresponding to the image sample, containing information on pest and disease type, disease characteristics, and disease development patterns; A supervised fine-tuning method is used to train the pre-trained multimodal large language model using the training dataset; During training, low-rank adaptive LoRA technology is used to optimize the parameters of the query, key, and value projection matrices of the attention layer in the multimodal large language model, thereby obtaining the multimodal recognition model.

[0007] According to the present invention, a method for diagnosing and controlling agricultural pests and diseases includes inputting the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model, comprising: The image features are fused with the various multimodal feature vectors in the local environmental data to generate a joint representation vector; the multimodal feature vectors include meteorological feature vectors, soil feature vectors, and remote sensing feature vectors. Based on the joint representation vector and the shooting time, the second diagnostic result is output through the classifier and regression model in the multi-source data mining model.

[0008] According to the present invention, a method for diagnosing and controlling agricultural pests and diseases includes fusing the image features with various multimodal feature vectors in the local environmental data to generate a joint representation vector, comprising: The image features and each of the multimodal feature vectors are linearly transformed through their respective fully connected layers and mapped to a unified shared semantic space to obtain aligned feature vectors. Each aligned feature vector is input into the cross-modal attention mechanism module to calculate the attention weight of each aligned feature vector. Based on the attention weights, the aligned feature vectors are weighted and summed to generate the joint representation vector.

[0009] According to the present invention, an agricultural pest and disease diagnosis and control method is provided, wherein the second diagnostic result is output based on the joint representation vector and the shooting time, through the classifier and regression model in the multi-source data mining model, including: The joint representation vector and the shooting time are input into the classifier to obtain the classification results and confidence scores of crop growth stage and growth status, which are used as the crop status information. The joint representation vector and the shooting time are input into the regression model to obtain a predicted value of the severity of the disease, and the predicted value is mapped to a predefined severity level. Based on the joint representation vector, the contribution of the image features and each of the multimodal feature vectors to the severity level is calculated; the pest and disease status information includes the severity level and the contribution.

[0010] According to the present invention, a method for diagnosing and controlling agricultural pests and diseases further includes: The large language model is guided by predefined thought chain prompts, and agricultural pest and disease diagnosis and control operations are performed based on the multimodal recognition model, the multi-source data mining model, and the pest and disease knowledge graph.

[0011] This invention also provides an agricultural pest and disease diagnosis and control device, comprising the following modules: The extraction module is used to extract image features, shooting time, and shooting location from crop disease and pest images of the target area; The first diagnostic module is used to input the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on the type of pest and disease and its disease pattern. The second diagnostic module is used to input the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information. The reasoning module is used to input the first diagnostic result and the second diagnostic result into the pest and disease knowledge graph for fusion reasoning and to generate plant protection pesticide management suggestions.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the agricultural pest and disease diagnosis and control method described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the agricultural pest and disease diagnosis and control method as described above.

[0014] The present invention provides a method, apparatus, electronic device, and storage medium for the diagnosis and control of agricultural pests and diseases. This method extracts image features, shooting time, and shooting location from crop pest and disease images of a target area; inputs the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on pest and disease type and disease development patterns; inputs the shooting time and corresponding local environmental data into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information; and inputs the first and second diagnostic results into a pest and disease knowledge graph for fusion and reasoning to generate plant protection and pesticide management recommendations. This invention achieves precise and interpretable pest and disease diagnosis and personalized control solutions through multimodal information fusion and step-by-step reasoning techniques combined with a pest and disease knowledge graph. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts illustrating the agricultural pest and disease diagnosis and control method provided by the present invention.

[0017] Figure 2 This is the second flowchart of the agricultural pest and disease diagnosis and control method provided by the present invention.

[0018] Figure 3 This is a schematic diagram of the pest and disease knowledge graph provided by the present invention.

[0019] Figure 4 This is a schematic diagram of the agricultural pest and disease diagnosis and control device provided by the present invention.

[0020] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] Large-scale model thinking chain technology improves model interpretability by explicitly breaking down reasoning steps, and large-scale models have unique advantages in general knowledge reasoning, but their application in the agricultural field is relatively simple. Traditional diagnostic models have the following shortcomings: (1) Single perception dimension: relying solely on visual symptoms and ignoring key environmental stress factors such as climate, soil, and crop growth period, resulting in a high misdiagnosis rate (e.g., unable to distinguish between physiological diseases and infectious diseases).

[0023] (2) The reasoning process is black box: the model directly outputs the diagnostic results, lacks interpretable reasoning logic, and farmers find it difficult to verify and trust.

[0024] (3) Decision-making is static and general: the prevention and control recommendations are not dynamically adapted to real-time weather, regional policies and crop growth stages, and are not practical enough.

[0025] Based on this, this invention proposes a method for the diagnosis and control of agricultural pests and diseases. It constructs a pest and disease knowledge graph and, through multimodal information fusion and step-by-step reasoning techniques, achieves precise and interpretable pest and disease diagnosis and personalized control solutions. By mining the underlying conditions of an image, including its location, disease characteristics, affected parts, disease status, and the future development trend of the disease in the local area, it provides a basis for accurate diagnosis and control. The main objectives of this invention include: Achieve accurate and interpretable diagnosis: not only identify lesions, but also integrate multi-source environmental data to reveal the causal mechanism of disease occurrence (why the disease occurs) and predict its development trend.

[0026] Generate localized dynamic prevention and control plans: Based on the knowledge graph of pests and diseases, provide users with personalized agronomic measures and pesticide recommendations that are closely matched with their local area, current season, crop growth stage and environmental conditions.

[0027] The following is combined Figures 1-5 The present invention describes a method, apparatus, electronic device, and storage medium for the diagnosis and control of agricultural pests and diseases.

[0028] Figure 1 This is one of the flowcharts illustrating the agricultural pest and disease diagnosis and control method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Extract image features, shooting time, and shooting location from crop disease and pest images of the target area.

[0029] The target area refers to farmland plots that require disease and pest diagnosis.

[0030] Image features refer to high-dimensional, numerical vector representations automatically extracted from original images of diseases and pests. They are mathematical abstractions that computers can understand and that can characterize the symptoms of diseases or the morphology of pests in the image. These features contain key information about the color, shape, texture, and distribution of lesions.

[0031] The capture time is the exact moment when the image was captured, automatically recorded from the image file metadata. It is mainly used to determine the current growth stage of the crop (such as tillering stage and heading stage), and to correlate with concurrent and future meteorological data for environmental stress analysis.

[0032] The shooting location is the geographic coordinates (latitude and longitude) of the image metadata recorded by the positioning module of the shooting device (such as a smartphone). It is mainly used to accurately call the local environmental data (such as meteorology, soil, remote sensing images) corresponding to the coordinate point, as well as to associate the regional pest and disease occurrence patterns and local plant protection policies of the region.

[0033] Acquire crop disease and pest images of the target area, for example, by receiving crop images with natural field backgrounds taken and uploaded by a user terminal device; or by retrieving images that meet the criteria from a pre-established crop disease and pest image database.

[0034] Extract image features, shooting time, and shooting location from crop disease and pest images of the target area. For example, decode the crop disease and pest images into a pixel matrix and perform standardization preprocessing operations, including at least size normalization and pixel value normalization; input the preprocessed pixel matrix into a pre-trained convolutional neural network model, extract a high-dimensional feature vector from a specified layer of the network as the image feature; parse the header information of the crop disease and pest images, and extract the geographic coordinates of the shooting time and shooting location from the EXIF ​​exchange file.

[0035] Step 102: Input the image features into the multimodal recognition model to obtain the first diagnostic result output by the multimodal recognition model.

[0036] The multimodal recognition model is a specialized model based on a multimodal large language model, fine-tuned with domain data, specifically designed for interpreting agricultural images and generating professional diagnostic reports. The primary diagnostic result is a visual evidence-based diagnostic conclusion output by the model, which may include information on pest and disease types and disease progression patterns. It should be understood that pest and disease types include disease types and pest types. Disease types are caused by pathogens or environmental factors and mainly include infectious and non-infectious diseases; pest types are caused by harmful organisms such as insects or mites. Disease progression information refers to the entire process and internal logic of when, where, and under what conditions a specific pest or disease occurs, develops, spreads, and disappears.

[0037] Image features are input into a supervised, fine-tuned multimodal recognition model. Based on its pre-trained visual and linguistic knowledge, the model decodes the image features to generate a natural language text containing disease and pest diagnosis conclusions. From the natural language text generated by the model, structured diagnostic information is extracted through rule matching or keyword recognition technology, and the structured diagnostic information is organized into a first diagnostic result.

[0038] Step 103: Input the shooting time and the local environmental data corresponding to the shooting location into the multi-source data mining model to obtain the second diagnostic result output by the multi-source data mining model.

[0039] Local environmental data refers to a collection of multimodal, time-series external parameters related to crop growth environment, obtained through data interface calls or IoT devices based on the geographic coordinates of the shooting location. Local environmental data can include: meteorological data, such as real-time and historical time-series data of temperature, humidity, precipitation, wind speed, and sunshine duration; soil data, such as soil moisture, temperature, pH value, and nitrogen, phosphorus, and potassium content; and remote sensing data, such as vegetation indices reflecting the macroscopic growth of crops, obtained by satellites or drones.

[0040] The multi-source data mining model is a specialized analytical model built on neural networks, used to fuse multimodal environmental features and perform multi-task learning. This model can map environmental data (meteorological, soil, remote sensing) from different sources and with different scales (meteorological, soil, remote sensing) to a unified semantic space and perform deep fusion through feature alignment, attention mechanisms, and other methods, thereby mining potential patterns that indicate crop health status and disease risk.

[0041] Optionally, the output layer of the multi-source data mining model is connected to multiple parallel prediction heads to solve different sub-tasks simultaneously. For example, one classifier determines crop growth, another classifier predicts disease type, and a regression model assesses severity.

[0042] The second diagnostic result is a structured assessment conclusion regarding crop physiological status and disease stress risk, derived from environmental data by a multi-source data mining model. This second diagnostic result can include crop status information and pest and disease status information. It should be understood that crop status information refers to an assessment of the crop's own physiological and growth status, which may include growth stages such as seedling stage, tillering stage, jointing stage, and heading stage (inferred from the time of image capture); growth condition such as healthy, mildly stressed, severely stressed, or wilted (inferred from environmental and remote sensing data). Pest and disease status information refers to the probability and severity of disease occurrence inferred from environmental conditions, which may include predicted pest and disease types; risk levels, used to quantitatively assess the probability of disease occurrence or potential severity (e.g., low, medium, high); and environmental contribution, used to quantify the contribution of various environmental characteristics (e.g., humidity, temperature) to the current risk prediction, enhancing interpretability.

[0043] The shooting time and corresponding local environmental data are input into a multi-source data mining model to obtain a second diagnostic result output by the model. For example, multimodal feature vectors from the local environmental data and the shooting time are input into multiple parallel task modules within the multi-source data mining model for diagnosis. These parallel task modules include at least: a classifier for assessing crop growth stage and vigor, a classifier for identifying potential pest and disease types, and a regression model for assessing disease severity. The outputs of these parallel task modules are integrated to generate a second diagnostic result, which includes at least crop status information output by the classifier, pest and disease status information output by the classifier and regression model, and the contribution of each environmental feature calculated by the cross-modal attention mechanism module.

[0044] Step 104: Input the first diagnostic result and the second diagnostic result into the pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions.

[0045] The pest and disease knowledge graph is a structured semantic network storing knowledge in the field of agricultural pests and diseases in graph form. The types of node entities in the knowledge graph can include pest and disease entities, crop entities, environmental condition entities, control measure entities, geographical region entities, and symptom entities, etc.; the types of relationships between edges in the knowledge graph can include causal relationships, temporal relationships, treatment relationships, attribution relationships, and spatial relationships, etc. It should be understood that the pest and disease knowledge graph is spatiotemporally aware, meaning that some of its entities and relationships are subject to valid temporal or spatial constraints.

[0046] Plant protection pesticide management recommendations refer to a set of operable, sequential instructions generated based on a fusion reasoning process and dynamically adapted to the current diagnostic context.

[0047] Using a pest and disease knowledge graph as the logical carrier, the first diagnostic result of the multimodal recognition model and the second diagnostic result of the multi-source data mining model are aligned, conflict-checked, and supplemented to ultimately derive accurate and practical plant protection pesticide management recommendations.

[0048] In one embodiment, reference Figure 2 First, images of pests and diseases are captured in the field to obtain image features, occurrence time, and location attributes. Then, the image features are processed by a multimodal recognition model to identify and diagnose the types of pests and diseases and their disease patterns. Simultaneously, based on the coordinate information in the images, local meteorological, soil, and remote sensing information is accessed. Through a multimodal data mining model, the crop status and pest and disease status at that location are diagnosed. Finally, combined with a pest and disease knowledge graph, plant protection pesticide application and management recommendations are provided.

[0049] The agricultural pest and disease diagnosis and control method provided in this invention extracts image features, shooting time, and shooting location from crop pest and disease images of a target area; inputs the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on pest and disease type and disease development pattern; inputs the shooting time and local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information; inputs the first and second diagnostic results into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions. This invention achieves precise and interpretable pest and disease diagnosis and personalized control solutions by using multimodal information fusion and step-by-step reasoning technology combined with a pest and disease knowledge graph.

[0050] Based on the above embodiments, the step of inputting the first diagnostic result and the second diagnostic result into the pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions includes: Based on the shooting location and shooting time, the geographical area to be diagnosed, the crop growth period, and future weather forecast are determined to form a spatiotemporal anchor point; Using the aforementioned spatiotemporal anchor points as constraints, a search is performed in the pest and disease knowledge graph to obtain pest and disease entities, environmental condition entities, and prevention and control scheme entities related to the current spatiotemporal context, forming a dynamic subgraph; Match the pest and disease type in the first diagnostic result with the pest and disease entity in the dynamic subgraph to confirm the target pest and disease. Based on the crop status information and pest and disease status information in the second diagnostic result, the environmental condition entities and control scheme entities related to the target pest and disease in the dynamic sub-graph are screened and adapted. In the dynamic subgraph, the causal relationship chain connected to the target pest is traversed; Based on the screened and adapted environmental condition entities and prevention and control scheme entities, and combined with the causal relationship chain, the disease pattern information in the first diagnostic result and the pest and disease risk level in the second diagnostic result, plant protection and pesticide management suggestions adapted to the current spatiotemporal context and crop status are generated.

[0051] It should be understood that a spatiotemporal anchor is a structured data object composed of geographic location coordinates, timestamps, and the derived agronomic context, which serves as an index for retrieving knowledge graphs.

[0052] Based on the shooting location and time, the geographic area to be diagnosed, the crop growth stage, and future weather forecasts are determined to form spatiotemporal anchor points. For example, the geographic coordinates of the shooting location are analyzed, and the administrative division and agricultural planting area to which the coordinates belong are determined through the inverse geographic information system (GIS) service, which serves as the geographic area to be diagnosed. Based on the shooting time, geographic area, and target crop type, the local crop phenological calendar model is queried to infer the current growth stage of the crop. Based on the geographic coordinates, a meteorological data interface is called to obtain historical meteorological records and future weather forecast data for a preset period of time. The geographic area, crop growth stage, and future weather forecast data are associated and encapsulated to generate spatiotemporal anchor points.

[0053] It should be understood that the current spatiotemporal context refers to the collection of all environmental and background information related to this diagnostic session, activated by the spatiotemporal anchor point. A pest / disease entity refers to a node in the knowledge graph that represents a specific pest / disease. An environmental condition entity refers to a node in the knowledge graph that represents an environmental factor or state that can induce or influence the occurrence and development of pests / diseases. A control plan entity refers to a node in the knowledge graph that represents a specific pest / disease management measure.

[0054] The geographical region, crop growth period, and future weather forecast in the spatiotemporal anchor points are converted into one or more query conditions suitable for graph databases. A graph query is then executed to retrieve entities in the knowledge graph that meet at least one of the following conditions: pest and disease entities whose active area matches the geographical region; pest and disease entities whose peak incidence period is associated with the crop growth period; and environmental condition entities whose triggering conditions match the future weather forecast. Using the retrieved pest and disease entities as the core, the system traverses their directly connected control scheme entities and causal relationships, encapsulating all relevant entities and their relationships to form an independent, dynamic subgraph for the current diagnostic session.

[0055] It should be understood that the target pests and diseases refer to the objects that are finally confirmed in this diagnosis after preliminary diagnosis by the multimodal recognition model and matching and verification in the dynamic subgraph.

[0056] The textual semantic similarity between the pest type name in the first diagnostic result and the names and aliases of each pest entity in the dynamic sub-image is calculated. The cosine similarity between the feature vector of the user-uploaded image and the feature vector of the symptom image associated with each pest entity in the dynamic sub-image is also calculated. For each pest entity, its textual semantic similarity score and image feature similarity score are fused to generate a comprehensive matching score. All entities are then sorted in descending order based on this score. The pest entity with the highest comprehensive matching score, or multiple pest entities with scores exceeding a preset threshold, are identified as the target pests for this diagnosis.

[0057] The real-time environmental data in the second diagnostic result is compared with the environmental condition entities in the dynamic subgraph. Environmental condition entities that match the current measured environmental data are selected to verify the environmental rationality of the target pest and disease occurrence. Based on the crop growth stage in the second diagnostic result, control schemes suitable for that growth stage and safe for the crop are selected from the control scheme entities in the dynamic subgraph, excluding prohibited schemes or those that may cause phytotoxicity. Based on the pest and disease risk level in the second diagnostic result, the selected control schemes are parameter-adapted. For high-risk levels, high-dose or potent pesticide combinations are adapted; for low-risk levels, low-dose or protective prevention schemes are adapted. The set of environmental condition entities and control scheme entities after consistency verification, safety screening, and efficacy adaptation is output as the direct basis for generating subsequent management recommendations.

[0058] Starting from the target pest / disease entity, traverse backwards along the relationship edges to retrieve all upstream causal entities that directly or indirectly cause its occurrence. Upstream causal entities include at least the pathogenic entity, the inducing environmental condition entity, and the susceptible host stage entity. Starting from the target pest / disease entity, traverse forwards along the relationship edges to retrieve all downstream consequence entities that it may cause. Downstream consequence entities include at least the entities that further aggravate the disease, and the entities that cause crop yield and quality loss. In the causal chain obtained by the traversal, locate and mark all nodes connected to the control plan entity. These nodes are the selectable intervention points. Integrate the upstream causal entities, the target pest / disease entity, the downstream consequence entities, and the control intervention points to form one or more complete causal chains from cause to consequence to control.

[0059] Based on the pest and disease risk level in the second diagnostic results, and combined with the severity of downstream consequences in the causal relationship chain, the urgency of the control action is determined; based on the disease pattern information in the first diagnostic results and future weather forecasts, specific application time windows are planned, avoiding adverse weather such as rain and strong winds; based on the selected and adapted control scheme entities, and combined with the growth stage in the crop status information, the dosage, application frequency, and safety interval of the pesticide are finely adjusted; integrating the outputs of all the above steps, a structured plant protection pesticide management recommendation is generated, which clearly includes at least: target pests and diseases, recommended control schemes, application timing, dosage and operation guidelines, and an explanation of the decision-making basis based on the causal relationship chain.

[0060] In one specific embodiment, the pest and disease knowledge graph incorporates spatiotemporal anchors into the RAG (Retrieval-Augmented Generation), breaking through the traditional two-dimensional knowledge model of "pest and disease-control" and constructing a three-dimensional knowledge fusion RAG of "spatiotemporal-entity-behavior". The specific implementation method is as follows: 1) Data Source Acquisition and Processing: This includes pest and disease literature, agricultural manuals, pesticide registration information, agricultural research papers, field records, and local plant protection pesticide application plans. Information extraction and knowledge fusion are performed on the data. For example, a Large Language Model (LLM) is used as a domain expert extractor, and few-shot prompts are designed for the agricultural field to extract triples from the text. For instance, extract entities and relations from the following text and output them in the form of (head entity, relation, tail entity). Text: "Rice blast is highly susceptible to outbreaks during the peak tillering stage when there is continuous rainy weather. It can be controlled using isoprothiolane or tricyclazole." Expected Output: (Rice blast, high incidence during, peak tillering stage), (continuous rainy weather, is, inducing condition), (isoprothiolane, can control, rice blast).

[0061] 2) Spatiotemporal Entity and Relationship Extraction: Extract spatiotemporally related entities and relationships, such as (County A, located in the Central China double-cropping rice region), (June, is the early rice booting stage), (rice leaf roller, peak migration period is in early July). Multimodal feature extraction: Use ViT (Vision Transformer) to extract features from pest and disease images, obtain high-dimensional vectors, store them in a vector database, and associate them with symptom entities in the atlas, discretizing continuous environmental data into environmental condition nodes in the atlas.

[0062] The design incorporates a spatiotemporal graph schema, using a graph database to store entities and relationships, and a vector database to store image and text embedding vectors. A joint index is created for geographic and reproductive period nodes. When a query occurs, the relevant subgraph range is quickly located using spatiotemporal anchors, significantly reducing the retrieval space and improving efficiency.

[0063] 3) Spatiotemporal Awareness Retrieval and Enhancement: When a user submits a query (e.g., uploading a picture of a diseased rice leaf, along with the location "County A" and the time "June 20th"), the system operates according to the following process: The system extracts geographic coordinates (or parses "County A" into coordinates) and a time point ("June 20th") from the user input. Based on the coordinates and time, the system determines the crop type (e.g., single-season rice), the current growth stage (e.g., peak tillering stage), and obtains the recent weather forecast for the region (weather for the next week). All of this information constitutes the spatiotemporal anchor point for this query.

[0064] The system converts queries (such as image feature vectors + text "What disease is this?") into graph query statements. The retrieval process is not performed on the entire graph, but rather on a subgraph defined by spatiotemporal anchor points: it displays all pests and diseases that are prevalent in County A in June and whose symptoms are similar to those in the image, and identifies their required environmental conditions and recommended control measures. This retrieval process can directly exclude diseases that occur in northern regions or only during the heading stage.

[0065] The similarity between the feature vector of the user-uploaded image and the vectors associated with all symptom entities in the sub-image is calculated. Combining text description similarity and image similarity, the most matching symptom node is found, thus locating the pest / disease entity. After retrieving the target pest / disease (e.g., rice blast), its causal relationships are further traversed, such as... Figure 3 As shown, this cause-effect subgraph contains all the elements of diagnosis, attribution, prediction, and decision-making.

[0066] In the spatiotemporal awareness retrieval process, the similarity between the input information and entities in the pest and disease knowledge graph is calculated: Let the vector of the user-uploaded image after feature extraction be... A symptom entity in a pest and disease knowledge graph The associated image feature vector is Their cosine similarity is: ; The comprehensive search score is a combined assessment of semantic similarity and other metadata filtering. The scoring function is designed as follows: ; in, It refers to the status of crops and diseases. With entity The similarity between text descriptions (calculated using cosine similarity). It is an adjustable hyperparameter used to balance the importance of text and image modalities, and the system returns the set of entities with the highest scores.

[0067] This invention utilizes spatiotemporal anchors to activate a localized knowledge graph, ensuring deep coupling between diagnostic conclusions and specific farmland environments, thus solving the problem of static decision-making in traditional models. Secondly, through cross-validation and screening adaptation of multi-source diagnostic results and dynamic subgraphs, image features are fused with environmental stress factors, improving the ability to distinguish between physiological and infectious diseases. By traversing causal chains to generate plant protection recommendations, prevention and control plans are dynamically linked with crop growth stages, real-time weather, and disease occurrence patterns, forming personalized management strategies that are both interpretable and timely, solving the problems of single perception dimensions, black-box reasoning, and generalized decision-making in traditional systems.

[0068] Based on the above embodiments, the multimodal recognition model is trained in the following manner: Construct a training dataset; the training dataset includes multiple data pairs, each data pair including an image sample and a text sample; the image sample is a crop pest and disease image containing a natural field background; the text sample is a structured descriptive text corresponding to the image sample, containing information on pest and disease type, disease characteristics, and disease development patterns; A supervised fine-tuning method is used to train the pre-trained multimodal large language model using the training dataset; During training, low-rank adaptive LoRA technology is used to optimize the parameters of the query, key, and value projection matrices of the attention layer in the multimodal large language model, thereby obtaining the multimodal recognition model.

[0069] It should be understood that the multimodal recognition model is trained using the Qwen-VL multimodal language model to analyze images and the types and patterns of pests and diseases. For example, the training and inference process of the multimodal recognition model may include: A training dataset is constructed, comprising multiple data pairs. Each data pair includes an image sample and a text sample (i.e., an image-text pair). The image sample is a crop pest and disease image with a natural field background. The text sample is a structured descriptive text corresponding to the image sample, containing information on pest and disease type, disease characteristics, and disease progression. For example, a (Image, Text) format training dataset is constructed. All images and labels need to be reviewed, corrected, and enriched by experts to ensure consistent and accurate annotation. The Image represents field images of pests and diseases, covering different crops, different pests and diseases, different disease stages, different severity levels, different shooting angles, and different lighting conditions. The Image also includes a natural field background to improve the model's generalization ability. The Text data consists of text data; each image needs to be accompanied by a description file and a structured JSON tag, simulating natural language input by the user, containing information such as disease characteristics and disease progression. For example, this is a photo of rice in a field. The rice in the picture is infected with rice blast. Rice blast usually occurs from the tillering stage to the jointing stage of rice. Symptoms include the base of the leaves turning grayish-black and the upper part of the leaves gradually turning brown. When the humidity in the field is high, a grayish-black mold layer can be seen on the surface of the diseased parts. In severe cases, it can lead to the death of seedlings.

[0070] A supervised fine-tuning approach is employed, using the training dataset to train a pre-trained multimodal large language model. Simultaneously, during training, low-rank adaptive LoRA (LoR) is used to optimize the parameters of the query, key, and value projection matrices in the attention layer of the multimodal large language model, resulting in a multimodal recognition model. It should be understood that Qwen-VL is a general-purpose multimodal large language model, fine-tuned using supervised fine-tuning (SFT). The low-rank adaptive LoRA technique fine-tunes only a small portion of the model's parameters to reduce computational cost and resource consumption, effectively preventing overfitting. The fine-tuned parameters may include low-quality adaptation matrices of the query, key, and value projection matrices in the attention layer. A standard autoregressive language modeling loss is used, where the model predicts the next token based on previous tokens. Training employs the AdamW optimizer, with a learning rate set between (1e-5, 2e-4), for 30 epochs.

[0071] After the multimodal recognition model is trained, the user-uploaded image is input into the trained and fine-tuned Qwen-VL model. The model generates structured diagnostic text, which includes information such as disease name, disease pattern, and disease characteristics.

[0072] This invention constructs a training dataset and uses LoRA technology to fine-tune a pre-trained multimodal large language model, thereby transforming a general visual language model into a multimodal recognition model. This enables the model to not only identify pests and diseases but also generate interpretable conclusions containing diagnostic criteria and disease patterns. This effectively solves the problem that traditional models can only output classification labels and lack professional knowledge and reasoning credibility. At the same time, LoRA technology significantly reduces training costs and avoids the risk of overfitting.

[0073] Based on the above embodiments, the step of inputting the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain the second diagnostic result output by the multi-source data mining model includes: The image features are fused with the various multimodal feature vectors in the local environmental data to generate a joint representation vector; the multimodal feature vectors include meteorological feature vectors, soil feature vectors, and remote sensing feature vectors. Based on the joint representation vector and the shooting time, the second diagnostic result is output through the classifier and regression model in the multi-source data mining model.

[0074] It should be understood that images of pests and diseases contain visual features of crop diseases, which can be extracted using CNNs (Convolutional Neural Networks) to extract deep visual feature vectors, i.e., image features. Based on the coordinates and time, recent (past 7-15 days) meteorological data for that location is retrieved online, including temperature (daily average temperature, maximum / low temperature, accumulated temperature), humidity, and precipitation. This meteorological data is then converted into meteorological feature vectors. Based on the coordinates, local static soil data is retrieved online, including soil type (sandy, clay, loam) and chemical properties (pH, organic matter, nitrogen, phosphorus, potassium), forming a soil feature vector. Recent multispectral satellite images of the site were obtained from a remote sensing platform, and the Normalized Difference Vegetation Index (NDVI) was calculated to reflect crop growth, and the Normalized Difference Water Index (NDWI) was calculated to reflect crop water stress. A time-series vegetation curve was constructed, and features were extracted to form a remote sensing feature vector. .

[0075] Optionally, local environmental data may also include data on historical risks and disasters, planting habits, etc.

[0076] Image features are fused with various multimodal feature vectors from local environmental data to generate a joint representation vector. For example, [the following is an example of a fusion process]. , , and By merging, we obtain Based on the joint representation vector and the shooting time, a second diagnostic result is output through the classifier and regression model in the multi-source data mining model.

[0077] This invention deeply integrates image features with environmental feature vectors from multiple sources, including meteorology, soil, and remote sensing, to construct a joint representation vector that characterizes the crop growth environment. This overcomes the perceptual limitations of traditional diagnostic models that rely solely on visual information. Based on this joint vector and environmental time-series information, the model can accurately assess the crop's physiological state and the risk of pest and disease stress, thereby improving the ability to distinguish between physiological and infectious diseases and providing a reliable basis for early warning and dynamic decision-making.

[0078] Based on the above embodiments, the step of fusing the image features with the various multimodal feature vectors in the local environmental data to generate a joint representation vector includes: The image features and each of the multimodal feature vectors are linearly transformed through their respective fully connected layers and mapped to a unified shared semantic space to obtain aligned feature vectors. Each aligned feature vector is input into the cross-modal attention mechanism module to calculate the attention weight of each aligned feature vector. Based on the attention weights, the aligned feature vectors are weighted and summed to generate the joint representation vector.

[0079] Each modality feature vector is mapped to a unified shared semantic space through its own fully connected layer (FC) encoding, ensuring that features from different sources and with different physical meanings can be directly compared and computed. Specifically, image feature vectors, meteorological feature vectors, soil feature vectors, and remote sensing feature vectors are input into their respective dedicated fully connected layers and linearly transformed using the following formula: ; in, The aligned feature vectors, For the first Input feature vectors of each modality and These are the learnable weight matrix and bias vector for the corresponding modality, respectively.

[0080] Through mapping using a dedicated fully connected layer, feature vectors from different modalities with varying origins, physical meanings, and dimensions are mapped. The features are projected onto a shared feature space with uniform dimensions and semantics, forming aligned feature vectors that can be directly compared and computed. .

[0081] Simultaneously, a cross-modal attention mechanism module is added to dynamically calculate the importance of features from different modalities. For example, if the model observes image features suggesting mold, while meteorological features indicate recent high humidity, the attention weight of the meteorological features will automatically increase because it provides crucial corroboration in the current diagnosis. Specifically, for each aligned feature vector... The initial attention score is calculated using a shared feedforward neural network and a non-linear activation function, as shown in the following formula: ; in, For the first Attention score for each modality and These are the learnable weight matrix and bias vector, respectively. These are learnable context vectors.

[0082] The initial attention scores of all modalities are input into the Softmax function for normalization to obtain the final attention weights for each modality. Output the final attention weights Each The numerical value represents the relative importance of the feature vector of the corresponding modality in the current specific diagnostic task. The formula for calculating the weight is: ; in, For the first Attention score for each modality and The calculation method is the same.

[0083] Scalar attention weights corresponding to each modality The corresponding aligned feature vector Perform scalar-vector multiplication to obtain a weighted eigenvector. The weighted feature vectors of all modalities are summed element-wise to generate a preliminary fusion vector, which is then used as the final joint representation vector. Its mathematical expression is: ; This joint representation vector integrates information from all modalities, and the weights... This reflects the relative importance of different modalities in the current specific diagnostic task.

[0084] This invention addresses the problems of inconsistent dimensions and semantic barriers in multimodal data by mapping features with different physical meanings, such as those from images, meteorology, soil, and remote sensing, to a unified semantic space through their respective fully connected layers. It utilizes a cross-modal attention mechanism to dynamically assess the relative importance of each environmental factor in the current diagnostic context, achieving intelligent feature-level filtering. Furthermore, by generating a joint representation vector rich in contextual information through weighted summation, the model can simulate the thought process of agricultural experts in comprehensive judgment, thereby improving the ability to analyze the causes of complex pests and diseases and the accuracy of diagnostic decisions.

[0085] Based on the above embodiments, the step of outputting the second diagnostic result based on the joint representation vector and the shooting time, through the classifier and regression model in the multi-source data mining model, includes: The joint representation vector and the shooting time are input into the classifier to obtain the classification results and confidence scores of crop growth stage and growth status, which are used as the crop status information. The joint representation vector and the shooting time are input into the regression model to obtain a predicted value of the severity of the disease, and the predicted value is mapped to a predefined severity level. Based on the joint representation vector, the contribution of the image features and each of the multimodal feature vectors to the severity level is calculated; the pest and disease status information includes the severity level and the contribution.

[0086] The shooting time is converted into a periodic temporal coding feature and concatenated with the joint representation vector to form an enhanced feature vector. The enhanced feature vector is input into the first Softmax classifier to obtain the classification result of the crop's growth stage, and its predicted probability is output as the first confidence level. The enhanced feature vector is input into the second Softmax classifier to obtain the classification result of the crop's growth status, and its predicted probability is output as the second confidence level. The classification result of the growth stage and its first confidence level, together with the classification result of the growth status and its second confidence level, are combined to form the crop status information.

[0087] For example, to assess crop status, a classifier is built based on... The model assesses the overall condition of the crop, including its growth stage and vigor. For example, even if image patches are not obvious, remote sensing data shows a sharp drop in NDVI, and meteorological data indicates sustained high temperatures; the model might determine that the crop is under water stress. Based on a multi-classification task, the model predicts its probability distribution. ; in, This represents the total number of disease categories. and It is a category With the corresponding weights and biases, the model output confidence score is the maximum probability value. .

[0088] The joint representation vector and the encoded shooting time features are input into a linear regression model, and the calculation is performed... Obtain a continuous predictive value representing the severity of the disease. ,in and The weights and biases learned by the regression model are used; the continuous predicted values ​​are compared with a set of predefined threshold intervals, and based on the comparison results, the predicted values ​​are mapped to the corresponding predefined severity levels, which include at least three ordered categories: mild, moderate, and severe; the severity levels obtained after mapping are output as an important component of the pest and disease status information.

[0089] The attention weights calculated based on the cross-modal attention mechanism module are directly used as the initial contribution of the corresponding modal feature vector pairs to form the joint representation vector. The initial contribution is then weighted and adjusted according to the magnitude of the corresponding dimension weights in the weight vector of the regression model to reflect the actual impact of different features in the final regression prediction. All modal-adjusted contributions are normalized so that their sum is 1, resulting in a standardized final contribution. The final contribution is then correlated with the severity level and used together as the output of pest and disease status information.

[0090] This invention, through collaborative reasoning of joint representation vectors and temporal information, not only synchronously outputs key growth states such as crop growth period and growth status, but also accurately quantifies the severity level of diseases. By calculating the contribution of multimodal features to the severity, the black-box model is transformed into an interpretable decision-making system, enabling farmers to clearly identify major pathogenic environmental factors (such as high humidity and low temperature), providing a direct scientific basis for formulating targeted prevention and control strategies, and solving the problem that traditional models can only output disease types but cannot quantify the severity and analyze the causes.

[0091] Based on the above embodiments, the method for diagnosing and controlling agricultural pests and diseases further includes: The large language model is guided by predefined thought chain prompts, and agricultural pest and disease diagnosis and control operations are performed based on the multimodal recognition model, the multi-source data mining model, and the pest and disease knowledge graph.

[0092] Constructing a thought chain for monitoring, diagnosing, and controlling agricultural pests and diseases enables the large language model to perceive, analyze, and reason from a more professional perspective. Specifically, prompts are established for the entire process, guiding the large model to execute according to the set thought process. First, define the role and task. For example, you are a top agricultural plant protection expert with decades of field experience. Your task is to complete a comprehensive analysis from diagnosis to control, based on user-provided crop disease images and associated spatiotemporal metadata (geographic location, time), following a rigorous reasoning process. Your final output must be clearly structured, supported by solid evidence, and offer feasible recommendations. Second, define the thought process and output structure. For example, you must strictly adhere to the following thought chain: "Perception-Retrieval-Analysis-Decision." Your final answer should be in Markdown format and include all of the following parts: Step 1: Multimodal Information Perception and Confirmation Instructions: First, confirm all the input information you have received (including location and time). This establishes the basic context for the analysis.

[0093] Output example: "Your diagnostic request has been received. We are analyzing a paddy field located in County A (latitude and longitude: 113.08°E, 28.25°N), with the photo taken on June 20th (crop growth stage: peak tillering stage). We are currently performing a comprehensive diagnosis by combining the field photos with the above spatiotemporal environmental information." Step 2: Preliminary Diagnosis and Image Analysis Instructions: Utilize your multimodal understanding to describe the symptom features in the image in detail. Considering the spatiotemporal context, provide 1-2 most likely preliminary diagnostic conclusions and indicate their confidence levels.

[0094] Output Example: Preliminary diagnosis and identification, including image analysis: The image shows typical spindle-shaped lesions on rice leaves. The lesions are grayish-white in the center, brown at the edge, and surrounded by a yellow halo, consistent with typical symptoms of rice blast. Diagnostic Conclusion: Preliminary diagnosis is leaf blast, with a confidence level of approximately 85%. Further confirmation and exclusion of other similar diseases are needed, based on environmental data.

[0095] Step 3: In-depth source tracing and causal analysis: Instructions: Retrieve and utilize the inherent agricultural knowledge graph to answer "What disease is this?" and "Why is it occurring here now?" Provide a detailed description of the disease and focus on constructing a causal chain between the "environment-pathogen-host" relationship to explain the root cause of the current disease outbreak.

[0096] Output Example: Disease Introduction and Cause Analysis, including disease type: brief introduction of pathogen and damage; symptom details: detailed description of symptoms, echoing the observation in the first step; occurrence pattern: this disease is highly prevalent from the tillering stage to the jointing stage. The pathogen prefers a high temperature and high humidity environment, with an optimal temperature of 25-28 degrees Celsius. Spore germination requires humidity above 90% and a continuous water film for 6-8 hours.

[0097] Causal attribution: The main causes of this outbreak are as follows: host sensitivity period: your rice is in the peak tillering stage where it is susceptible to disease; meteorological factors: it was found that there was three consecutive days of rainfall in County A recently, the humidity in the field remained saturated, and the temperature was maintained at 26-30 degrees Celsius, which perfectly matched the ideal conditions for the outbreak of pathogens; regional factors: this area is a perennial occurrence area of ​​rice blast, and the source of pathogens is widely present.

[0098] Step 4: Forecasting and Risk Assessment Instructions: Based on the above causal analysis, infer the future development trend of the disease. Combine this with future weather forecasts to issue quantitative or qualitative risk warnings.

[0099] Output Example: Future Forecast and Risk Assessment, including Weather Forecast: The next three days are expected to be mainly cloudy and rainy, with temperature and humidity conditions continuing to favor the pathogen; Development Trend: Without immediate intervention, lesions are expected to rapidly expand and connect within 3-5 days, leading to extensive leaf death; Risk Level: High Risk. The current disease is highly likely to spread rapidly and develop from leaf blast into the more serious neck blast, resulting in significant yield reduction.

[0100] Step 5: Generate prevention and control decision prescriptions: Instructions: Provide specific and actionable solutions. Solutions must be closely tied to the spatiotemporal context (e.g., recommending locally compliant pesticides, considering the current growth stage) and embody the concept of integrated pest management.

[0101] Example output: Integrated pest management prescription, including emergency agricultural operations: immediately dig ditches to drain water, reduce field humidity, and disrupt the pathogen's breeding environment. This is currently the most urgent and effective measure; chemical control plan: spray fungicide immediately after rain when the weather clears; preferred pesticide: 75% tricyclazole wettable powder (30g per mu), this pesticide has strong systemic properties and good preventive effect, and is a highly effective pesticide for controlling rice blast; usage precautions: the safety interval is 21 days. Please be sure to rotate pesticides to delay resistance development. After one week, you can consider combining it with isoprothiolane or kasugamycin; compliance check: all the above recommended pesticides meet the requirements of green agricultural production in Hunan Province.

[0102] Step 6: Recommendations for Long-Term Management and Monitoring Directive: Provide follow-up action guidelines to transform a one-time diagnosis into a long-term health management strategy.

[0103] Output Example: Subsequent monitoring and management recommendations, including regular checks: revisit on the 3rd and 7th day after application to check whether lesion expansion has been inhibited; full-process protection: spray again at the heading and panicle stages to prevent neck blast; long-term strategy: after harvesting this crop, it is recommended to deeply till the soil and dispose of diseased straw. For the next season, select disease-resistant varieties and pay attention to balanced fertilization (increasing the application of silicon and potassium fertilizers) to enhance plant resistance.

[0104] In one specific embodiment, given an input image of a diseased area, the output of the large model is: "Based on the images and location information you provided, your rice is most likely infected with leaf blast (confidence level 85%)."

[0105] Introduction: Leaf blast is a type of rice blast disease that occurs on the leaves of rice and is a serious and prevalent disease in rice production. Initially, it appears as small, dark green spots, which gradually enlarge into typical spindle-shaped or rhomboid (fusiform) spots. These spots are approximately 1-1.5 cm long and 0.3-0.5 cm wide. The center of the spot is grayish-white, with a reddish-brown to brown edge and a distinct pale yellow halo around the periphery. Brown necrotic lines often extend towards the veins at both ends of the spot. Under humid conditions, a gray mold layer (i.e., conidia) will develop on the underside of the spot.

[0106] Occurrence Pattern: Rice blast can occur throughout the entire rice growth period, but the peak period for disease occurrence and damage is from the tillering stage to the jointing stage. The pathogen mainly overwinters as conidia and mycelium on diseased grains (seeds) and straw. The optimal temperature for mycelial growth and spore production is 25-28 degrees Celsius. Spore germination requires a relative humidity of over 90%, and ideally, the presence of water droplets or a film of water for 6-8 hours is required. Therefore, rainy, foggy, and dewy weather conditions are highly conducive to the occurrence and spread of the disease.

[0107] Cause of the incident: The photo was taken in plot B of County A. County A is currently in the peak tillering stage and has experienced three consecutive days of light rain. The high temperature and humidity environment is extremely conducive to the germination of rice blast spores.

[0108] Future forecast: The next three days will be mainly cloudy. If not controlled in time, the lesions will expand and connect rapidly, causing the leaves to wither and die, and may develop into a more serious neck blast.

[0109] Prevention and control measures: 1) Agricultural operations: Immediately dig ditches to drain water and reduce field humidity. 2) Chemical control: Immediately after rain, spray 75% tricyclazole wettable powder (30g per mu). Rotate pesticides as needed, with a safe interval of 21 days. Check again after one week. If the disease has spread, consider combining with isoprothiolane.

[0110] This invention, through predefined thought chain prompts, transforms a large language model into an expert in the agricultural field, enabling it to systematically schedule and coordinate three professional modules: a multimodal recognition model, a multi-source data mining model, and a pest and disease knowledge graph. This integrates the traditionally fragmented perception, analysis, and decision-making processes into a coherent "perception-verification-tracing-decision" reasoning flow, ensuring that diagnostic conclusions integrate visual evidence and environmental stress factors, and are deeply verified and causally explained through the knowledge graph. This generates dynamic prevention and control plans that are closely adapted to the real-time spatiotemporal context, achieving precision, interpretability, and personalization of pest and disease diagnosis and control plans.

[0111] The following describes the agricultural pest and disease diagnosis and control device provided by the present invention. The agricultural pest and disease diagnosis and control device described below can be referred to in correspondence with the agricultural pest and disease diagnosis and control method described above.

[0112] refer to Figure 4 The agricultural pest and disease diagnosis and control device provided by the present invention includes an extraction module 401, a first diagnosis module 402, a second diagnosis module 403, and a reasoning module 404.

[0113] Extraction module 401 is used to extract image features, shooting time and shooting location from crop disease and pest images of the target area; The first diagnostic module 402 is used to input the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on the type of pest and disease and its disease pattern. The second diagnostic module 403 is used to input the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information. The reasoning module 404 is used to input the first diagnostic result and the second diagnostic result into the pest and disease knowledge graph for fusion reasoning and to generate plant protection pesticide management suggestions.

[0114] The agricultural pest and disease diagnosis and control device provided in this invention extracts image features, shooting time, and shooting location from crop pest and disease images of a target area; inputs the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on pest and disease type and disease development pattern; inputs the shooting time and local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information; inputs the first and second diagnostic results into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions. This invention achieves accurate and interpretable pest and disease diagnosis and personalized control solutions by using multimodal information fusion and step-by-step reasoning technology combined with a pest and disease knowledge graph.

[0115] In one embodiment, the inference module 404 is further configured to: Based on the shooting location and shooting time, the geographical area to be diagnosed, the crop growth period, and future weather forecast are determined to form a spatiotemporal anchor point; Using the aforementioned spatiotemporal anchor points as constraints, a search is performed in the pest and disease knowledge graph to obtain pest and disease entities, environmental condition entities, and prevention and control scheme entities related to the current spatiotemporal context, forming a dynamic subgraph; Match the pest and disease type in the first diagnostic result with the pest and disease entity in the dynamic subgraph to confirm the target pest and disease. Based on the crop status information and pest and disease status information in the second diagnostic result, the environmental condition entities and control scheme entities related to the target pest and disease in the dynamic sub-graph are screened and adapted. In the dynamic subgraph, the causal relationship chain connected to the target pest is traversed; Based on the screened and adapted environmental condition entities and prevention and control scheme entities, and combined with the causal relationship chain, the disease pattern information in the first diagnostic result and the pest and disease risk level in the second diagnostic result, plant protection and pesticide management suggestions adapted to the current spatiotemporal context and crop status are generated.

[0116] In one embodiment, the agricultural pest and disease diagnosis and control device further includes a training module for: Construct a training dataset; the training dataset includes multiple data pairs, each data pair including an image sample and a text sample; the image sample is a crop pest and disease image containing a natural field background; the text sample is a structured descriptive text corresponding to the image sample, containing information on pest and disease type, disease characteristics, and disease development patterns; A supervised fine-tuning method is used to train the pre-trained multimodal large language model using the training dataset; During training, low-rank adaptive LoRA technology is used to optimize the parameters of the query, key, and value projection matrices of the attention layer in the multimodal large language model, thereby obtaining the multimodal recognition model.

[0117] In one embodiment, the second diagnostic module 403 is further configured to: The image features are fused with the various multimodal feature vectors in the local environmental data to generate a joint representation vector; the multimodal feature vectors include meteorological feature vectors, soil feature vectors, and remote sensing feature vectors. Based on the joint representation vector and the shooting time, the second diagnostic result is output through the classifier and regression model in the multi-source data mining model.

[0118] In one embodiment, the second diagnostic module 403 is further configured to: The image features and each of the multimodal feature vectors are linearly transformed through their respective fully connected layers and mapped to a unified shared semantic space to obtain aligned feature vectors. Each aligned feature vector is input into the cross-modal attention mechanism module to calculate the attention weight of each aligned feature vector. Based on the attention weights, the aligned feature vectors are weighted and summed to generate the joint representation vector.

[0119] In one embodiment, the second diagnostic module 403 is further configured to: The joint representation vector and the shooting time are input into the classifier to obtain the classification results and confidence scores of crop growth stage and growth status, which are used as the crop status information. The joint representation vector and the shooting time are input into the regression model to obtain a predicted value of the severity of the disease, and the predicted value is mapped to a predefined severity level. Based on the joint representation vector, the contribution of the image features and each of the multimodal feature vectors to the severity level is calculated; the pest and disease status information includes the severity level and the contribution.

[0120] In one embodiment, the agricultural pest and disease diagnosis and control device further includes a diagnosis and control module, used for: The large language model is guided by predefined thought chain prompts, and agricultural pest and disease diagnosis and control operations are performed based on the multimodal recognition model, the multi-source data mining model, and the pest and disease knowledge graph.

[0121] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an agricultural pest and disease diagnosis and control method. The method includes: extracting image features, shooting time, and shooting location from crop pest and disease images of a target area; inputting the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes pest and disease type and disease occurrence pattern information; inputting the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information; and inputting the first diagnostic result and the second diagnostic result into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions.

[0122] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the agricultural pest and disease diagnosis and control methods provided by the above methods. The method includes: extracting image features, shooting time, and shooting location from crop pest and disease images of a target area; inputting the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes pest and disease type and disease occurrence pattern information; inputting the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information; and inputting the first diagnostic result and the second diagnostic result into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the agricultural pest and disease diagnosis and control methods provided by the above methods. The method includes: extracting image features, shooting time, and shooting location from crop pest and disease images of a target area; inputting the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes pest and disease type and disease occurrence pattern information; inputting the shooting time and local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information; and inputting the first diagnostic result and the second diagnostic result into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for diagnosing and controlling agricultural pests and diseases, characterized in that, include: Extract image features, shooting time, and shooting location from crop disease and pest images of the target area; The image features are input into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on the type of pest and disease and its disease pattern. The shooting time and the local environmental data corresponding to the shooting location are input into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information. The first and second diagnostic results are input into the pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management suggestions. The step of inputting the first diagnostic result and the second diagnostic result into a pest and disease knowledge graph for fusion reasoning to generate plant protection pesticide management recommendations includes: Based on the shooting location and shooting time, the geographical area to be diagnosed, the crop growth period, and future weather forecast are determined to form a spatiotemporal anchor point; Using the aforementioned spatiotemporal anchor points as constraints, a search is performed in the pest and disease knowledge graph to obtain pest and disease entities, environmental condition entities, and prevention and control scheme entities related to the current spatiotemporal context, forming a dynamic subgraph; Match the pest and disease type in the first diagnostic result with the pest and disease entity in the dynamic subgraph to confirm the target pest and disease. Based on the crop status information and pest and disease status information in the second diagnostic result, the environmental condition entities and control scheme entities related to the target pest and disease in the dynamic sub-graph are screened and adapted. In the dynamic subgraph, the causal relationship chain connected to the target pest is traversed; Based on the screened and adapted environmental condition entities and prevention and control scheme entities, and combined with the causal relationship chain, the disease pattern information in the first diagnostic result and the pest and disease risk level in the second diagnostic result, plant protection and pesticide management suggestions adapted to the current spatiotemporal context and crop status are generated.

2. The method for diagnosing and controlling agricultural pests and diseases according to claim 1, characterized in that, The multimodal recognition model was trained using the following method: Construct a training dataset; the training dataset includes multiple data pairs, each data pair including an image sample and a text sample; the image sample is a crop pest and disease image containing a natural field background; the text sample is a structured descriptive text corresponding to the image sample, containing information on pest and disease type, disease characteristics, and disease development patterns; A supervised fine-tuning method is used to train the pre-trained multimodal large language model using the training dataset; During training, low-rank adaptive LoRA technology is used to optimize the parameters of the query, key, and value projection matrices of the attention layer in the multimodal large language model, thereby obtaining the multimodal recognition model.

3. The method for diagnosing and controlling agricultural pests and diseases according to claim 1, characterized in that, The step of inputting the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain the second diagnostic result output by the multi-source data mining model includes: The image features are fused with the various multimodal feature vectors in the local environmental data to generate a joint representation vector; the multimodal feature vectors include meteorological feature vectors, soil feature vectors, and remote sensing feature vectors. Based on the joint representation vector and the shooting time, the second diagnostic result is output through the classifier and regression model in the multi-source data mining model.

4. The method for diagnosing and controlling agricultural pests and diseases according to claim 3, characterized in that, The step of fusing the image features with the various multimodal feature vectors in the local environmental data to generate a joint representation vector includes: The image features and each of the multimodal feature vectors are linearly transformed through their respective fully connected layers and mapped to a unified shared semantic space to obtain aligned feature vectors. Each aligned feature vector is input into the cross-modal attention mechanism module to calculate the attention weight of each aligned feature vector. Based on the attention weights, the aligned feature vectors are weighted and summed to generate the joint representation vector.

5. The method for diagnosing and controlling agricultural pests and diseases according to claim 3, characterized in that, Based on the joint representation vector and the shooting time, the second diagnostic result is output through the classifier and regression model in the multi-source data mining model, including: The joint representation vector and the shooting time are input into the classifier to obtain the classification results and confidence scores of crop growth stage and growth status, which are used as the crop status information. The joint representation vector and the shooting time are input into the regression model to obtain a predicted value of the severity of the disease, and the predicted value is mapped to a predefined severity level. Based on the joint representation vector, the contribution of the image features and each of the multimodal feature vectors to the severity level is calculated; the pest and disease status information includes the severity level and the contribution.

6. The method for diagnosing and controlling agricultural pests and diseases according to claim 1, characterized in that, The methods for diagnosing and controlling agricultural pests and diseases also include: The large language model is guided by predefined thought chain prompts, and agricultural pest and disease diagnosis and control operations are performed based on the multimodal recognition model, the multi-source data mining model, and the pest and disease knowledge graph.

7. An agricultural pest and disease diagnosis and control device, characterized in that, include: The extraction module is used to extract image features, shooting time, and shooting location from crop disease and pest images of the target area; The first diagnostic module is used to input the image features into a multimodal recognition model to obtain a first diagnostic result output by the multimodal recognition model; the first diagnostic result includes information on the type of pest and disease and its disease pattern. The second diagnostic module is used to input the shooting time and the local environmental data corresponding to the shooting location into a multi-source data mining model to obtain a second diagnostic result output by the multi-source data mining model; the second diagnostic result includes crop status information and pest and disease status information. The reasoning module is used to input the first diagnostic result and the second diagnostic result into the pest and disease knowledge graph for fusion reasoning and to generate plant protection pesticide management suggestions. The reasoning module is also used to determine the geographical area to be diagnosed, the crop growth period and the future weather forecast based on the shooting location and the shooting time, so as to form a spatiotemporal anchor point; and to search in the pest and disease knowledge graph with the spatiotemporal anchor point as a constraint to obtain pest and disease entities, environmental condition entities and prevention and control scheme entities related to the current spatiotemporal context, forming a dynamic subgraph. Match the pest and disease type in the first diagnostic result with the pest and disease entity in the dynamic subgraph to confirm the target pest and disease. Based on the crop status information and pest and disease status information in the second diagnostic result, the environmental condition entities and control scheme entities related to the target pest and disease in the dynamic subgraph are screened and adapted; in the dynamic subgraph, the causal relationship chain connected to the target pest and disease is traversed. Based on the screened and adapted environmental condition entities and prevention and control scheme entities, and combined with the causal relationship chain, the disease pattern information in the first diagnostic result and the pest and disease risk level in the second diagnostic result, plant protection and pesticide management suggestions adapted to the current spatiotemporal context and crop status are generated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the agricultural pest and disease diagnosis and control method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the agricultural pest and disease diagnosis and control method as described in any one of claims 1 to 6.

Images