Pest-disease causal relationship identification method and system and pest-disease causal map construction method and system

Abstract

This invention discloses a method and system for identifying causal relationships in pests and diseases, as well as a method and system for constructing causal graphs for pests and diseases. The method includes: acquiring multimodal data; preprocessing the multimodal data and dividing the preprocessed multimodal data into sample-level time-series window data; performing semantic alignment on each time-series window data and constructing event candidate triples; generating personalized question prompt templates based on the contextual information of the event candidate triples; calling a large-scale speech model for causal relationship determination to perform causal relationship determination based on the question prompt templates, obtaining causal determination results; wherein, the causal determination results include causal labels and confidence levels; and constructing a causal graph for pests and diseases based on the event candidate triples and the causal determination results. This invention is beneficial for constructing structurally sound and well-defined causal graphs for pests and diseases, and is suitable for mining and identifying causal relationships in pests and diseases in complex agricultural scenarios.

Description

Technical Field

[0001] This invention relates to the field of agricultural intelligent analysis and artificial intelligence technology, and in particular to a method and system for constructing a causal map of pests and diseases based on multimodality, as well as the method and system for constructing a causal map of pests and diseases. Background Technology

[0002] Crop diseases and pests are among the main factors affecting agricultural yield and food security. Traditional methods for analyzing diseases and pests rely on expert experience or rule-based modeling, depending on agricultural experts manually summarizing causal rules or drawing causal maps. While these methods offer some interpretability, they heavily depend on human effort and subjective knowledge accumulation, making them difficult to extend to large-scale, high-frequency data analysis tasks, especially failing to meet the needs of real-time early warning and automated modeling. While statistical causal inference (such as Granger causality and structural equation modeling) has emerged in recent years, it has somewhat compensated for these shortcomings, but it still struggles to handle causal information in unstructured data (such as agricultural condition records, expert logs, and weather reports), and its adaptability is poor when facing complex and ever-changing agricultural ecosystems. Summary of the Invention

[0003] To address the technical problems existing in the background art, this invention proposes a method and system for constructing causal maps of pests and diseases, as well as the method and system for constructing causal maps of pests and diseases.

[0004] This invention proposes a method and system for constructing a multimodal causal map of pests and diseases, comprising:

[0005] Acquire multimodal data;

[0006] The multimodal data is preprocessed, and the preprocessed multimodal data is divided into sample-level time-series window data;

[0007] Perform semantic alignment on the data for each time-series window and construct event candidate triples;

[0008] Generate personalized question prompt templates based on the contextual information of event candidate triples;

[0009] The causal relationship determination large speech model is invoked to perform causal relationship determination based on the question prompt template, and the causal determination result is obtained;

[0010] Based on the candidate triples of events and the causal determination results, a causal map of pests and diseases is constructed.

[0011] Preferably, the multimodal data includes structured data, unstructured text, and unstructured images.

[0012] Preferably, the multimodal data is preprocessed, specifically including: normalizing the structured data; generating text semantic embedding vectors based on the unstructured text; and performing feature extraction and structure enhancement on the unstructured image to obtain image semantic feature embedding vectors and image region entities.

[0013] Preferably, generating a text semantic embedding vector based on unstructured text specifically includes: generating a text semantic embedding vector based on unstructured text using a text encoding large language model.

[0014] Preferably, feature extraction and structure enhancement are performed on the unstructured image to obtain image semantic feature embedding vectors and image region entities, specifically including:

[0015] The CLIP model is used to extract initial semantic features from unstructured images, forming a high-dimensional feature representation of the images;

[0016] The SLIC superpixel algorithm is used to perform superpixel segmentation on the high-dimensional feature representation of the image to obtain multiple semantic regions, and each semantic region is used as an image region node.

[0017] Node connections are constructed based on the pixel contact relationships between nodes in the image region to obtain a spatial adjacency graph;

[0018] Construct a non-local similarity graph based on image region nodes with high cosine similarity but not spatially adjacent semantic feature vectors;

[0019] Construct a fused graph structure based on the spatial adjacency graph and the nonlocal similarity graph;

[0020] Semantic information is propagated on the fused graph structure using graph convolutional networks or graph attention mechanisms to obtain image semantic feature embedding vectors.

[0021] Semantic classification is performed on each image region node based on the image semantic feature embedding vector, and high-confidence agricultural entities are extracted as image region entities.

[0022] Preferably, each time-series window data includes structured data, text semantic embedding vectors, image semantic embedding vectors, and image region entities within the corresponding time-series window.

[0023] Preferably, semantic alignment is performed on each time-series window of data, and event candidate triples are constructed. Specifically, this includes: analyzing the text semantic embedding vector, identifying verbal or resultative expressions with event semantics, and forming text candidate event expressions.

[0024] Based on the image semantic embedding vector, image region entities are obtained; natural language generation is performed on the image region entities to expand or supplement the semantic boundaries of the image region entities and form image region-level candidate event expressions.

[0025] Align textual candidate event representations with image region-level candidate event representations to a unified semantic space to form a unified event representation;

[0026] Within a unified semantic space, event matching and reasoning are performed on the unified event representation by combining the context information of the current time sequence window, and event candidate triples are constructed.

[0027] Preferably, the causal determination result includes causal label and confidence level.

[0028] Preferably, a causal graph of pests and diseases is constructed based on the candidate triplets of events and the causal determination results. Specifically, this includes: initially constructing a causal graph of pests and diseases based on the candidate triplets of events; optimizing the causal graph of pests and diseases; and defining the causal strength of each causal edge in the optimized causal graph of pests and diseases based on the causal labels and confidence levels, the embedding similarity of events in the candidate triplets of events, and the preset prior weights of the agricultural knowledge graph.

[0029] Preferably, the causal map of pests and diseases is optimized, specifically by using Relational-GCN to optimize the expanded causal map of pests and diseases.

[0030] Preferably, a personalized question prompt template is generated based on the contextual information of the event candidate triples, specifically including:

[0031] Based on the contextual information of the event candidate triples, a temporal enhanced context vector is formed;

[0032] Based on the temporal augmentation context vector, personalized question prompt templates are learned from historical training tasks using the meta-prompt mechanism based on meta-learning.

[0033] Preferably, the large speech model for determining causal relationships is a finely tuned GPT-4 model.

[0034] Preferably, after constructing the causal map of pests and diseases based on the candidate triplet of events and the causal determination results, the method further includes: visualizing the causal map of pests and diseases.

[0035] Preferably, after visualizing the causal map of pests and diseases, the method further includes: obtaining feedback information; and adjusting and updating the causal map of pests and diseases based on the feedback information.

[0036] Secondly, this invention also proposes a multimodal pest and disease causal map construction system, comprising:

[0037] The acquisition module is used to acquire multimodal data;

[0038] The preprocessing module is used to preprocess the multimodal data and divide the preprocessed multimodal data into sample-level time-series window data.

[0039] The semantic alignment and event construction module is used to perform semantic alignment on each time window data and construct event candidate triples;

[0040] The question prompt template generation module is used to generate personalized question prompt templates based on the context information of the event candidate triplet.

[0041] The causal relationship determination module is used to call the causal relationship determination large speech model to perform causal relationship determination based on the question prompt template and obtain the causal relationship determination result;

[0042] The causal graph construction module constructs a causal graph of pests and diseases based on candidate event triples and causal determination results.

[0043] Thirdly, this invention also proposes a multimodal method for identifying causal relationships in pests and diseases, including:

[0044] The pest causal map is constructed using the multimodal pest causal map construction method described in any one of the first aspects;

[0045] Obtain target pest and disease events:

[0046] Counterfactual intervention simulations are performed on target pest and disease events on a pest and disease causal graph, and graph neural reasoning networks are used to propagate causal paths and extract the embedding features of multiple causal paths related to the target pest and disease events. The embedding features of multiple causal paths are then combined into combined features through a fusion mechanism.

[0047] By inputting the combined features into a pre-built multimodal joint risk prediction model, the causal relationship identification results of the target pest and disease events are obtained.

[0048] Fourthly, the present invention also proposes a multimodal pest and disease causal relationship identification system, comprising: a processor and a memory, wherein the memory is used to store one or more programs; when one or more programs are executed by the processor, the steps of the multimodal pest and disease causal relationship identification method as described in any one of the third aspects are implemented.

[0049] The proposed method and system for identifying causal relationships in pests and diseases, as well as the method and system for constructing causal maps of pests and diseases, preprocess the acquired multimodal data and divide the preprocessed multimodal data into sample-level time-series window data to uniformly represent causal information, opening up semantic fusion paths between different data sources and providing a foundation for modeling complex agricultural environments. Semantic alignment is performed on each time-series window data, and event candidate triples are constructed. Then, a large-scale speech model for causal relationship determination is called to perform causal relationship determination based on the question prompt template to obtain the causal determination result. Based on the event candidate triples, causal labels, and confidence scores, a structurally sound causal map of pests and diseases with distinct edge classes is constructed, which is suitable for the mining and identification of causal relationships in pests and diseases in complex agricultural scenarios. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a method for constructing a multimodal causal map of pests and diseases according to an embodiment of the present invention. Detailed Implementation

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] Firstly, referring to Figure 1 This invention proposes a method and system for constructing a multimodal causal map of pests and diseases, comprising:

[0053] Acquire multimodal data;

[0054] Preprocess the multimodal data;

[0055] The preprocessed multimodal data is divided into sample-level time-series window data;

[0056] Perform semantic alignment on the data for each time-series window and construct event candidate triples;

[0057] Generate personalized question prompt templates based on the contextual information of event candidate triples;

[0058] The causal relationship determination large speech model is invoked to perform causal relationship determination based on the question prompt template, and the causal determination result is obtained; the causal determination result includes causal label and confidence level;

[0059] Based on the candidate triples of events and the causal determination results, a causal map of pests and diseases with temporal sequence, semantic interpretability and causal strength is constructed.

[0060] This invention preprocesses the acquired multimodal data and divides the preprocessed multimodal data into sample-level time-series window data to uniformly represent causal information, opening up semantic fusion paths between different data sources and providing a foundation for modeling complex agricultural environments. Semantic alignment is performed on each time-series window data, and event candidate triples are constructed. Then, a large-scale speech model for causal relationship determination is invoked to perform causal relationship determination based on question prompt templates, obtaining causal determination results. These results include causal labels and confidence levels. Based on the event candidate triples and the causal determination results, a well-structured, edge-class distinguishable pest and disease causal map with temporal sequence, semantic interpretability, and causal strength is constructed. This significantly improves the accuracy and interpretability of causal relationship modeling, enabling a systematic representation of pest and disease transmission mechanisms, environmental impact paths, and complex causal interactions among agricultural entities. It also provides reliable support for subsequent causal inference, risk prediction, intelligent intervention, and decision-making, exhibiting good versatility, scalability, and application potential for practical deployment, especially suitable for pest and disease causal relationship mining and identification in complex agricultural scenarios.

[0061] In this embodiment, multimodal data includes structured data, unstructured text, and unstructured images. Structured data includes meteorological data (temperature, humidity, precipitation), remote sensing index (NDVI), and planting records (sowing date, fertilization, etc.), in CSV / JSON / SQL formats. Unstructured text includes agricultural technology reports, expert logs, and group chat records, in TXT, PDF, or Doc formats. Unstructured images include images of pests and diseases, and field monitoring screenshots, in PNG, JPG, or other image formats.

[0062] It is important to understand that, in this embodiment, the time series window refers to dividing multimodal data into sample-level continuous segments according to the time dimension, used to model the temporal context dependencies of events. Each time series window contains structured data (such as temperature and humidity), text data (such as agricultural technology logs), and image data (such as field photos) within a fixed time range to achieve causal chain extraction and alignment between events.

[0063] In this embodiment, the preprocessing of multimodal data specifically includes: normalizing structured data; generating text semantic embedding vectors based on unstructured text; and performing feature extraction and structure enhancement on unstructured images to obtain image semantic feature embedding vectors and image region entities.

[0064] This embodiment uses the Z-score method for normalization. The normalization formula is x' = (x - μ) / σ; where x' is the normalized structured data, x is the structured data, μ is the mean, and σ is the standard deviation.

[0065] The text semantic embedding vector is embedding_text = LLM_encoder(text).

[0066] Specifically, generating text semantic embedding vectors from unstructured text includes: using a text encoding big language model (LLM) to generate text semantic embedding vectors from unstructured text, thereby leveraging the language understanding and generation capabilities of the text encoding big language model (LLM) to automatically mine causal event chains from complex agricultural texts.

[0067] Specifically, in this embodiment, the text encoding large language model is ChatGLM-6B. ChatGLM-6B is called to perform contextual understanding on unstructured text, generate text semantic embeddings, and retain potential causal clues.

[0068] In a further embodiment, feature extraction and structure enhancement are performed on the unstructured image to obtain image semantic feature embeddings and image region entities, specifically including:

[0069] The CLIP model is used to extract initial semantic features from unstructured images, forming a high-dimensional feature representation of the images;

[0070] The SLIC superpixel algorithm is used to perform superpixel segmentation on the high-dimensional feature representation of the image to obtain multiple semantic regions, and each semantic region is used as an image region node.

[0071] Node connections are constructed based on the pixel contact relationships between nodes in the image region to obtain a spatial adjacency graph;

[0072] Construct a non-local similarity graph based on image region nodes with high cosine similarity but not spatially adjacent semantic feature vectors;

[0073] Construct a fused graph structure based on the spatial adjacency graph and the nonlocal similarity graph;

[0074] Semantic information is propagated on the fused graph structure using graph convolutional networks or graph attention mechanisms to obtain image semantic feature embedding vectors.

[0075] Semantic classification is performed on each image region node based on the image semantic feature embedding vector, and high-confidence agricultural entities are extracted as image region entities; among them, agricultural entities include “insects”, “withered leaves”, “spots”, etc.

[0076] The structure-enhanced image semantic feature embedding is: embedding_image = GNN(CNN(SLIC(image))).

[0077] In this embodiment, each time-series window data includes structured data, text semantic embedding vectors, image semantic embedding vectors, and image region entities within the corresponding time-series window, i.e., {T_i:[X_struct,embedding_text,embedding_image,image_objects]}, that is, each time-series window T i This corresponds to an information unit containing four types of features, namely structured data features X. struct Text semantic embedding vector `embedding_text`, image semantic embedding vector `embedding_image`, and image region entities `image_objects`.

[0078] In order to construct event candidate triples, in this embodiment, semantic alignment is performed on each time window data and event candidate triples are constructed. Specifically, this includes: analyzing the text semantic embedding vector, identifying verbal or resultative expressions with event semantics, and forming text candidate event expressions.

[0079] Based on the image semantic embedding vector, image region entities are obtained; natural language generation is performed on the image region entities to expand or supplement the semantic boundaries of the image region entities and form image region-level candidate event expressions.

[0080] Align text-based candidate event representations with image region-level candidate event representations to a unified semantic space to form a unified event representation (unified_embedding), ensuring semantic consistency and contextual synergy across modal events;

[0081] Within a unified semantic space, event matching and reasoning are performed on the unified event representation by combining the context information of the current time window (such as crop stage, climate background, agricultural operations, etc.) to construct event candidate triples.

[0082] The event candidate triple takes the form of (event A, relation type, event B), where event A and event B are event candidates from text or image sources, and the relation type is a neutral expression, to be determined by a large language model.

[0083] Where, unified_embedding = Align(embedding_text, embedding_image); where unified_embedding is the unified event representation, Align() represents semantic alignment, embedding_text represents the text semantic embedding vector, and embedding_image represents the image semantic embedding vector.

[0084] In constructing event candidate triples, event elements are first extracted from both the text and image modalities, followed by information fusion to ultimately build structured event candidate triples. The construction process of these event candidate triples includes:

[0085] Text modal event representation extraction; image modal event source extraction; region-level event representation generation; cross-modal event semantic fusion; event triple construction.

[0086] In the process of extracting text modal event expressions, the text semantic embedding vector `embedding_text` is analyzed to identify verbal or resultative expressions with event semantics, forming preliminary text event candidates. For example, phrases expressing motive or resultative events such as "temperature rise" or "planthopper outbreak" are extracted from the text. This step is based on the domain prior vocabulary and dependency syntax relations, and can also be assisted by calling the event extraction large language model (LLM_EventExtraction) to perform imperative extraction through prompt templates, such as: Prompt: "Extract the agricultural events described in the following text: {text}" → ["continuous high temperature", "planthopper surge"].

[0087] In the process of extracting image modal event sources, the image region is first divided using the SLIC superpixel algorithm, and then a graph neural inference network (such as GCN or GAT) is used to generate an image semantic embedding vector (embedding_image) to complete the classification of image region nodes. The classification label of each high-confidence region node is used as an interpretable event source in the image, which is called the image region entity label (such as "insects", "mold", "leaf discoloration" etc.).

[0088] In the process of generating region-level event representations, for the identified image region nodes, an image description model (such as BLIP2) is further invoked to generate natural language for the local regions to assist in identifying the agricultural events they may represent. For example, the region-level description generated for a certain insect region is: "Dense insect infestation appears on the leaves," thereby expanding or supplementing the semantic boundaries of the image region entities and forming image region-level candidate event representations.

[0089] In the process of cross-modal event semantic fusion, semantic alignment models (such as CLIP and BLIP) are used to align text event representations with image region entity embeddings to a unified semantic space, forming a unified event representation (unified_embedding), ensuring that cross-modal events have semantic consistency and contextual collaboration.

[0090] In the process of constructing event candidate triples, under a unified semantic space, event matching and reasoning are performed in combination with the contextual information of the current time window (such as crop stage, climate background, agricultural operations, etc.) to construct event candidate triples in the form of (event A, relation type, event B). Event A and event B are respectively from event candidates from the text or image end, and the relation type is a neutral expression, which will be determined by the large language model later.

[0091] For example, combining the text "Recent high temperatures have exacerbated pest infestations in the fields" with image region recognition results:

[0092] Candidate text events: ["High temperatures persist", "Insect infestation worsens"];

[0093] Image region labels: ["Insects", "Leaf discoloration"];

[0094] BLIP2 output area events: ["Insects are concentrated in certain areas", "Leaves show signs of wilting"];

[0095] The candidate triplet structures for the events are ("persistent high temperature", "possibly related", "concentrated distribution of insects"), ("insects", "possibly related", "leaf wilting symptoms"), and ("persistent high temperature", "possibly related", "intensified insect infestation").

[0096] In this embodiment, a personalized question prompt template is generated based on the contextual information of the event candidate triples, specifically including:

[0097] Based on the contextual information of the event candidate triples, a temporal enhanced context vector is formed;

[0098] Based on the temporal augmentation context vector, personalized question prompt templates are learned from historical training tasks using the meta-prompt mechanism based on meta-learning.

[0099] Where Prompt = MetaPrompt(context_vector); Prompt represents the question prompt template, MetaPrompt() represents the Meta-Prompt mechanism, and context_vector represents context information.

[0100] The context information in this embodiment integrates information such as crop type, growth stage, time information, and meteorological background.

[0101] This embodiment generates prompt templates that conform to agricultural conditions through a meta-prompt mechanism based on meta-learning. This enables the subsequent large language model used for recognition to adaptively optimize causal identification for different periods, different crops, and different weather conditions, effectively improving the accuracy and adaptability of subsequent causal relationship extraction.

[0102] In this embodiment, result = LLM(Prompt + (eventA, eventB)) → {label, confidence}. Here, the causal label includes promoting, inhibiting, and neutral, and LLM represents a large speech model for determining causal relationships.

[0103] In this embodiment, the large speech model for determining causal relationships is a GPT-4 model fine-tuned based on agricultural data. For example, the output is "promote", with a confidence level of 0.92.

[0104] It should be noted that the causal relationship determination large speech model in this embodiment is based on

[0105] The pre-trained model based on the GPT-4 architecture was further fine-tuned on an agricultural dataset.

[0106] To optimize its language understanding and semantics in agricultural event recognition and causal relationship determination tasks.

[0107] Adaptability. Fine-tuning employs a dual-task training method combining instruction learning and event relationship classification.

[0108] The formula takes a prompt template (Prompt) and event candidate triples as input and outputs a causal label.

[0109] The labels for the training data, along with their corresponding confidence levels, are derived from expert-annotated datasets and agricultural causality studies.

[0110] Knowledge graph.

[0111] In this embodiment, the fine-tuned GPT-4 model not only returns causal labels (such as "promote"), but also...

[0112] It outputs a confidence score ("progressive", "inhibitory", "neutral") to quantify the credibility of causal relationships and participates in the construction of causal edge strength in subsequent causal graphs. Compared to existing methods...

[0113] Using a large language model that generates results solely based on open-ended question-and-answer formats, this embodiment employs a finely tuned GPT-4 model that combines agricultural-specific expressions, temporal context, and expert prior knowledge to perform causal analysis.

[0114] Inference optimization enables highly adaptable causal relationship identification in the agricultural field, particularly suitable for...

[0115] It is suitable for typical agricultural situations such as sparse text, ambiguous expression, and multiple intertwined events.

[0116] In this embodiment, a causal graph of pests and diseases is constructed based on candidate event triples and causal determination results, specifically including:

[0117] Based on the candidate triplet of events, a preliminary causal map of pests and diseases was constructed.

[0118] Optimize the causal map of pests and diseases;

[0119] Based on causal labels and confidence levels, the embedding similarity of events in event candidate triples, and the pre-defined prior weights of the agricultural knowledge graph, the causal strength of each causal edge in the optimized pest and disease causal graph is defined.

[0120] This embodiment is configured in such a way that it can construct a causal graph of pests and diseases with multimodal nodes, heterogeneous edge types, and causal intensity quantification capabilities.

[0121] Wherein, G = (V, E, R); where G represents the causal graph of pests and diseases, V is the event node, and the source includes structured data, text content and image region entities; E is the causal edge, and R is the semantic type of the causal edge.

[0122] The optimized pest and disease causal graph includes the following node types: structured event nodes, image region nodes, and text event nodes. The topology of the image region nodes is provided by the generated fused graph structure, including spatial adjacency edges and semantic similarity edges. Spatial adjacency edges refer to the edges between adjacent region units in the image, while semantic similarity edges refer to the connections established between non-adjacent regions with high cosine similarity in the feature space. Together, they constitute the topology of the image region graph, supporting local and non-local semantic propagation on the graph in graph neural inference networks.

[0123] The edge types of the optimized pest and disease causal graph are r∈R, including: natural influences, human behavior, and image observations.

[0124] Among these are natural influences, such as "precipitation → increased humidity".

[0125] Human actions, such as "spraying pesticides → reducing pests".

[0126] Image observation, such as "spotted image region → disease representation".

[0127] In this embodiment, the embedding similarity originates from the semantic vectors (text / image / event triples) generated from multimodal data. The semantic correlation between events is quantified by cosine similarity, and is ultimately used to optimize the weights of edges in the causal graph, thereby improving the semantic interpretability of the graph.

[0128] In a further embodiment, the causal map of pests and diseases is optimized, specifically including:

[0129] The extended causal graph of pests and diseases is optimized using Relational-GCN to perform graph embedding propagation and improve structural coherence and semantic consistency of nodes.

[0130] In the optimization process of Relational-GCN, for a certain node v in the graph i Its representation at layer l+1 The update formula is:

[0131]

[0132] In the formula, Let i be the representation of node i at level l+1. N represents the set of all relation types in a pest and disease causal graph; r (i) represents the set of neighbors of node i under relation type r; Let j be the representation of node j at layer l; c represents the weight matrix associated with relation type r; i,r Represents the normalization constant; Let represent the self-connection weight matrix, and σ represent the activation function.

[0133] In the causal graph of pests and diseases, the causal strength of each causal edge is:

[0134]

[0135] In the formula, e ij This represents a directed causal edge from event node i to event node j; weight() represents the causal strength, s llm s represents the confidence level of the output of a large language model. prior This represents prior knowledge derived from verified causal chain weights in agricultural knowledge graphs (such as the Chinese Agricultural Technology Encyclopedia and CNKI). Let represent the cosine similarity between the embeddings of event node i and event node j. Here, α, β, and γ are weighting coefficients used to fuse the confidence of the large language model, the prior causal score of the agricultural knowledge graph, and the semantic similarity of the event semantic embedding vectors, satisfying α + β + γ = 1.

[0136] This embodiment explicitly models the causal mechanisms such as "climate-driven", "human intervention", and "policy influence" in agricultural scenarios. It combines the Relational-GCN graph neural inference network to optimize the graph structure of the pest and disease causal graph and dynamically adjusts the parameters to improve the fitting ability and generalization performance of the causal edge strength, thereby enhancing path coherence and semantic interpretability.

[0137] In this embodiment, after constructing the pest and disease causal map based on the event candidate triples and the causal determination results, the method further includes: visualizing the pest and disease causal map.

[0138] This includes visualizing the cause-and-effect graph of pests and diseases, specifically by using graph visualization tools such as D3.js or ECharts to draw and display the cause-and-effect graph of pests and diseases.

[0139] In this embodiment, after visualizing the causal map of pests and diseases, the method further includes: obtaining feedback information; and adjusting and updating the causal map of pests and diseases based on the feedback information.

[0140] This embodiment supports an expert feedback mechanism to facilitate adjustments to the graph structure and causal edge labels, using feedback information to update event embeddings, optimize problem prompt templates, and iteratively optimize the graph structure.

[0141] Secondly, this invention also proposes a multimodal pest and disease causal map construction system, comprising:

[0142] The acquisition module is used to acquire multimodal data;

[0143] The preprocessing module is used to preprocess the multimodal data and divide the preprocessed multimodal data into sample-level time-series window data.

[0144] The semantic alignment and event construction module is used to perform semantic alignment on each time window data and construct event candidate triples;

[0145] The question prompt template generation module is used to generate personalized question prompt templates based on the context information of the event candidate triplet.

[0146] The causal relationship determination module is used to call the causal relationship determination large speech model to perform causal relationship determination based on the question prompt template and obtain the causal relationship determination result;

[0147] The causal graph construction module constructs a causal graph of pests and diseases based on candidate event triples and causal determination results.

[0148] It should be understood that the multimodal pest and disease causal map construction system proposed in this invention can realize the multimodal pest and disease causal map construction method proposed in the first aspect.

[0149] In this embodiment, the construction system further includes a visualization module, which is used to visualize and display the cause-and-effect diagram of pests and diseases.

[0150] In this embodiment, the construction system further includes a feedback and adjustment module, which is used to acquire feedback information and adjust and update the causal map of pests and diseases based on the feedback information.

[0151] Specifically, the feedback and adjustment module has a built-in visual editing and expert feedback channel for the graph, supports the addition and deletion of causal paths and the correction of causal edges, forming a knowledge enhancement closed loop of "human-machine co-construction" that enables the system to continuously evolve and adapt.

[0152] Thirdly, this invention also proposes a multimodal method for identifying causal relationships in pests and diseases, including:

[0153] The pest causal map is constructed using any of the multimodal pest causal map construction methods in the first aspect;

[0154] Obtain the target pest and disease event Y:

[0155] Counterfactual intervention simulation is performed on the target pest event Y on the pest causal graph, and graph neural inference network is used to propagate causal paths to extract the embedding features of multiple causal paths related to the target pest event; and the embedding features of multiple causal paths are combined into combined features through a fusion mechanism.

[0156] By inputting the combined features into a pre-built multimodal joint risk prediction model, the causal relationship identification results of the target pest and disease events are obtained.

[0157] The causal relationship identification results include multiple event paths, corresponding causal reasoning logic, predicted risk values, and explanatory text, supporting both complexity and robustness.

[0158] This invention constructs a causal graph of pests and diseases, and based on user-defined variable intervention conditions, employs a graph neural network for causal path propagation and risk state estimation to generate probability changes of target pest and disease events under different intervention scenarios. Simultaneously, through causal path tracing and interpretive text generation, it outputs causal reasoning chains and human-readable diagnostic language results, effectively improving the accuracy, stability, and interpretability of pest and disease causal relationship identification. This can also support tasks such as pest early warning and timing decision-making in precision agriculture scenarios.

[0159] The multimodal joint risk prediction model in this embodiment includes MLP, GAT, and Transformer.

[0160] Among them, the three architectures—MLP, GAT, and Transformer—are suitable for feature fusion and graph structure modeling tasks at different levels. Specifically, the GAT model is used for node state propagation and intervention reasoning in graph structures, the MLP model is used for multimodal feature fusion and risk scoring in a unified vector space, and the Transformer model is used for contextual modeling and logical sequence analysis of intervention paths or event chains.

[0161] This invention improves the GAT structure by combining the graph structure characteristics of agricultural scenarios and adds a causal edge type attention mechanism and an edge strength weighting mechanism to enhance the path discrimination ability and result stability during intervention modeling.

[0162] Specifically, the combined features are input into a pre-constructed multimodal joint risk prediction model to obtain the diagnostic results of the target pest and disease event, including:

[0163] Construct a multimodal joint risk prediction model;

[0164] The combined features are input into a pre-built multimodal joint risk prediction model to obtain the event path, risk prediction value and causal reasoning logic of the target pest and disease event.

[0165] Based on the event path, risk prediction value, and causal reasoning logic of the target pest and disease event, a corresponding explanatory text is generated.

[0166] During the generation of the corresponding explanatory text, a natural language causal chain explanation is automatically generated based on the event path, risk prediction value, and causal reasoning logic of the target pest or disease event. For example: "Due to the influence of event A, through the mediation of events B and C, event D is ultimately caused, with a predicted risk of 0.83." Another example: "Due to continuous high temperatures, through pest hatching and plant stress mechanisms, a planthopper outbreak occurs, with a predicted risk value of 0.83."

[0167] In this embodiment, after obtaining the diagnostic results of the target pest and disease event, the method further includes:

[0168] While displaying the causal map of pests and diseases, it dynamically highlights high-risk event paths and inference nodes in the causal relationship identification results, supports the display of node attributes and semantic tags, emphasizes interpretability, facilitates interaction with users, and effectively improves the user experience.

[0169] For example, when displaying a causal graph of pests and diseases, nodes represent events, edges represent relationships, and color / thickness indicates the strength of causality.

[0170] In this embodiment, after obtaining the diagnostic results of the target pest and disease event, the method further includes:

[0171] Support agricultural experts in revising the marginal weights and causal paths of target pest and disease events;

[0172] Based on the edge weights and causal paths of the modified target pest and disease events, update the structure of the problem prompt template and the pest and disease causal graph to achieve continuous optimization and knowledge evolution through human-machine collaboration.

[0173] This embodiment supports interactive revisions and incorporates expert feedback to continuously optimize the graph structure and causal extraction problem prompt templates, thereby enhancing the "human-machine co-construction" capability.

[0174] In this embodiment, a counterfactual intervention simulation is performed on the target pest event on the pest causal graph to obtain the probability of the target pest event Y occurring under the intervention conditions, specifically including:

[0175] Obtain the intervention conditions set by the user for the target pest and disease event;

[0176] The graph neural inference network (GAT) is used to traverse the graph starting from the intervention node, propagate causal signals along the path, aggregate upstream and downstream event information, update the state of each node, and output the risk estimate of the target pest and disease event under the current intervention conditions.

[0177] For example, the user sets the intervention condition as do(high temperature = 3 days) to see if it "leads to a surge in planthoppers", thus triggering a causal path simulation; do(X = "continuous high temperature") → triggers a causal path simulation. Starting from node x, the simulation propagates along the causal edges in the graph; it traverses adjacent event nodes and uses the GNN inference module to aggregate upstream causal signals; it simulates the impact strength of the intervention on the target pest event Y (such as "surge in planthoppers").

[0178] Under this intervention condition, the probability of change of the target pest event Y (such as a planthopper outbreak) is P(Y|do(X=x))≈inference network(G,X=x).

[0179] In this embodiment, the graph neural reasoning network is a structure-aware graph neural reasoning network (GAT). After the intervention node is activated, the state of all nodes on the path is updated; weights are assigned to each candidate causal chain to determine the dominant path under intervention; and finally, the probability of the target pest event Y occurring under the intervention condition is output.

[0180] This embodiment can also select appropriate prevention and control strategies based on the causal relationship identification results of the target pest and disease event.

[0181] For example, if the intervention condition for "planthopper outbreak" is "pre-watering", and the causal relationship identification result of the target pest event under the intervention condition is "if pre-watering is carried out, the risk of planthopper outbreak induced by high temperature will decrease from 0.78 to 0.41", then pre-watering can be carried out between high temperature time points.

[0182] Fourthly, the present invention also proposes a multimodal pest and disease causal relationship identification system, comprising: a processor and a memory, wherein the memory is used to store one or more programs; when one or more programs are executed by the processor, the steps of the multimodal pest and disease causal relationship identification method as described in any one of the third aspects are implemented.

[0183] The present invention will now be described in conjunction with specific embodiments.

[0184] Example 1

[0185] This invention proposes a multimodal method for identifying causal relationships in pests and diseases, comprising:

[0186] S1. Acquire multimodal data; Specifically, in a typical rice-growing area of ​​Anhui Province, multimodal data related to rice planthoppers were collected from 2020 to 2022, covering 12 data types. Structured data included: meteorological data (temperature, humidity, precipitation), planting information (sowing period, variety, fertilization frequency), and pest and disease observation data (rice planthopper numbers and density); unstructured images included: remote sensing images (NDVI, thermal infrared); unstructured text included agricultural expert reports, agricultural logs, WeChat group communication texts, etc. All data were accessed through the acquisition interface to the system bus and uniformly mapped to the agricultural season timeline based on timestamps to ensure temporal consistency and semantic alignment.

[0187] S2. Preprocess the multimodal data and divide the preprocessed multimodal data into sample-level time-series window data;

[0188] S21. Structured data standardization: Use the Z-score method to uniformly normalize numerical features such as temperature and humidity.

[0189] S22. Unstructured text embedding: Call ChatGLM-6B to perform contextual understanding on agricultural technology texts, generate semantic embeddings, and retain potential causal clues;

[0190] S23, Image Embedding Extraction and Structural Enhancement;

[0191] S4. Perform semantic alignment on the data of each time window and construct event candidate triples;

[0192] S5. Generate a personalized question prompt template based on the context information of the event candidate triplet;

[0193] S6. Call the causal relationship determination large speech model to perform causal relationship determination based on the question prompt template, and obtain the causal determination result; among which, the causal determination result includes causal label and confidence level;

[0194] S7. Construct a causal graph of pests and diseases based on the candidate triples of events and the causal determination results;

[0195] S8. Obtain target pest and disease events;

[0196] S9. Perform counterfactual intervention simulation on the target pest and disease event on the pest and disease causal graph, and use graph neural reasoning network to propagate causal paths, extract the embedding features of multiple causal paths related to the target pest and disease event; and form combined features from the embedding features of multiple causal paths through a fusion mechanism.

[0197] S10. Input the combined features into the pre-constructed multimodal joint risk prediction model to obtain the causal relationship identification results of the target pest and disease event;

[0198] S11. Visualize the causal relationship identification results of the pest and disease causal map and / or the target pest and disease event;

[0199] S12. Obtain feedback from experts and adjust and update the causal map of pests and diseases.

[0200] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal causal map of pests and diseases, characterized in that, include: Acquire multimodal data; where multimodal data includes structured data, unstructured text, and unstructured images; The multimodal data is preprocessed, including: normalizing the structured data; generating text semantic embedding vectors based on the unstructured text; and performing feature extraction and structure enhancement on the unstructured image to obtain image semantic feature embedding vectors and image region entities. The preprocessed multimodal data is divided into sample-level time-series window data; wherein, the time-series window data includes structured data, text semantic embedding vectors, image semantic embedding vectors, and image region entities within the corresponding time-series window; The text semantic embedding vectors are analyzed to identify verbal or resultative expressions with event semantics, forming text candidate event expressions. Image region entities are obtained based on image semantic embedding vectors. Natural language generation is performed on the image region entities to expand or supplement their semantic boundaries, forming image region-level candidate event expressions. The text candidate event expressions and image region-level candidate event expressions are aligned to a unified semantic space to form a unified event representation. Under the unified semantic space, event matching and reasoning are performed on the unified event representation in combination with the context information of the current time window to construct event candidate triples. Generate personalized question prompt templates based on the contextual information of event candidate triples; The causal relationship determination large speech model is invoked to perform causal relationship determination based on the question prompt template, and the causal determination result is obtained; Based on the candidate triples of events and the causal determination results, a causal map of pests and diseases is constructed.

2. The method for constructing a multimodal causal map of pests and diseases according to claim 1, characterized in that, Generate text semantic embedding vectors from unstructured text, specifically by using a text encoding large language model to generate text semantic embedding vectors from unstructured text.

3. The method for constructing a multimodal causal map of pests and diseases according to claim 1, characterized in that, Feature extraction and structure enhancement are performed on unstructured images to obtain semantic feature embedding vectors and image region entities, specifically including: The CLIP model is used to extract initial semantic features from unstructured images, forming a high-dimensional feature representation of the images; The SLIC superpixel algorithm is used to perform superpixel segmentation on the high-dimensional feature representation of the image to obtain multiple semantic regions, and each semantic region is used as an image region node. Node connections are constructed based on the pixel contact relationships between nodes in the image region to obtain a spatial adjacency graph; Construct a non-local similarity graph based on image region nodes with high cosine similarity but not spatially adjacent semantic feature vectors; Construct a fused graph structure based on the spatial adjacency graph and the nonlocal similarity graph; Semantic information is propagated on the fused graph structure using graph convolutional networks or graph attention mechanisms to obtain image semantic feature embedding vectors. Semantic classification is performed on each image region node based on the image semantic feature embedding vector, and high-confidence agricultural entities are extracted as image region entities.

4. The method for constructing a multimodal causal map of pests and diseases according to claim 1, characterized in that, Personalized question prompt templates are generated based on the contextual information of the event candidate triples, specifically including: Based on the contextual information of the event candidate triples, a temporal enhanced context vector is formed; Based on the temporal augmentation context vector, personalized question prompt templates are learned from historical training tasks using the meta-prompt mechanism based on meta-learning.

5. The method for constructing a multimodal causal map of pests and diseases according to claim 1, characterized in that, The causal relationship determination large speech model is a finely tuned GPT-4 model; The fine-tuning process of the GPT-4 model adopts a dual-task training method of instruction learning and event relationship classification. The input is a question prompt template and event candidate triples, and the output is a causal label and corresponding confidence score. The labels of the training data come from an expert-annotated dataset and an agricultural causal knowledge graph.

6. The method for constructing a multimodal causal map of pests and diseases according to claim 1, characterized in that, The causal determination results include causal labels and confidence levels; Among them, based on the candidate triples of events and the causal determination results, a causal map of pests and diseases is constructed, specifically including: Based on the candidate triplet of events, a preliminary causal map of pests and diseases was constructed. Optimize the causal map of pests and diseases; Based on causal labels and confidence levels, the embedding similarity of events in event candidate triples, and the pre-defined prior weights of the agricultural knowledge graph, the causal strength of each causal edge in the optimized pest and disease causal graph is defined.

7. The method for constructing a multimodal causal map of pests and diseases according to claim 6, characterized in that, Optimize the causal map of pests and diseases, specifically including: The expanded causal map of pests and diseases was optimized using Relational-GCN.

8. The method for constructing a multimodal causal map of pests and diseases according to claim 1, characterized in that, After constructing the causal map of pests and diseases based on the candidate triplet of events and the causal determination results, the process also includes: visualizing the causal map of pests and diseases.

9. The method for constructing a multimodal causal map of pests and diseases according to claim 8, characterized in that, After visualizing the cause-and-effect diagram of pests and diseases, the process also includes: obtaining feedback information; and adjusting and updating the cause-and-effect diagram of pests and diseases based on the feedback information.

10. A multimodal pest and disease causal map construction system, applied to the multimodal pest and disease causal map construction method according to any one of claims 1-9, characterized in that, include: The acquisition module is used to acquire multimodal data; The preprocessing module is used to preprocess the multimodal data and divide the preprocessed multimodal data into sample-level time-series window data. The semantic alignment and event construction module is used to perform semantic alignment on each time window data and construct event candidate triples; The question prompt template generation module is used to generate personalized question prompt templates based on the context information of the event candidate triplet. The causal relationship determination module is used to call the causal relationship determination large speech model to perform causal relationship determination based on the question prompt template and obtain the causal relationship determination result; The causal graph construction module constructs a causal graph of pests and diseases based on candidate event triples and causal determination results.

11. A method for identifying causal relationships in pests and diseases based on multimodal approaches, characterized in that, include: The pest causal map is constructed using the multimodal pest causal map construction method described in any one of claims 1-9; Obtain target pest and disease events: Counterfactual intervention simulations were performed on target pest and disease events on the pest and disease causal graph, and graph neural reasoning networks were used to propagate causal paths and extract the embedding features of multiple causal paths related to the target pest and disease events. Furthermore, the embedded features of multiple causal paths are combined into combined features through a fusion mechanism; By inputting the combined features into a pre-built multimodal joint risk prediction model, the causal relationship identification results of the target pest and disease events are obtained.

12. A multimodal pest and disease causal relationship identification system, characterized in that, include: Processor and memory; memory is used to store one or more programs. When one or more programs are executed by a processor, the steps of the multimodal pest and disease causal relationship identification method as described in claim 11 are implemented.

Images