Crop knowledge graph question answering method and system based on large model and data assimilation

Abstract

The embodiment of the application provides a crop knowledge graph question and answer method and system based on a large model and data assimilation. In the method, an initial knowledge graph is generated based on a dynamic graph neural network; the initial knowledge graph is updated based on a dynamic incremental graph updating algorithm; knowledge extraction and understanding are performed on a user's question based on a large language model (LLM), the output result of the LLM is processed, and a graph structured query template is generated; a Prompt query template constructed by the LLM is acquired, the Prompt query template is fused with the knowledge graph, an optimal information query is obtained by using a reinforcement learning path optimization method, and graph query results obtained by each Prompt query template are integrated; and a question and answer result is generated based on the fused graph query result of the LLM after optimization and adjustment. The application refines a vertical field large model by fusing multi-modal data, combining historical user query questions q to generate high-precision qa question and answer pairs, and improving the question and answer accuracy by building a multi-layer Prompt strategy.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more particularly to a crop knowledge graph question answering method and system based on large models and data assimilation. Background Technology

[0002] The bulk grain question-and-answer system is a specific scenario within the question-and-answer system framework. Its goal is to build a question-and-answer system through algorithms that allows users to quickly and accurately obtain answers to questions by inputting them. By providing fast and accurate information and analysis, the bulk grain question-and-answer system helps users in various fields improve efficiency, reduce costs, and enhance competitiveness, playing an irreplaceable role in the entire grain industry.

[0003] With the continuous development of question-answering system technology, the technology has also been greatly improved, but it still faces the following problems:

[0004] First, the problem of multimodal data silos: The current question-and-answer system for bulk grains adopts a single-modal question-and-answer system solution. By parsing user questions, the incompleteness of the data leads to the weak cross-modal correlation analysis capability of the current question-and-answer system, making it difficult to support the linkage decision-making of "weather-soil-market".

[0005] Second, the problem of static knowledge graphs: Although current question answering systems integrate data using knowledge graphs, which has improved the results compared to previous technologies, current knowledge graphs are mostly statically constructed based on historical data and cannot adapt to dynamic changes. This leads to a significant lag in the information provided in the answers, resulting in inaccurate responses.

[0006] Third, the answers are monotonous and the user experience is poor: Traditional knowledge graph methods generate answers based on several rule models, resulting in fixed answers and limited wording for users. Although many people are trying to use LLM models to address the diversity of answers, general-purpose models have low accuracy in answering questions in specific domains. At the same time, current answers are single-function dialogues without more reasons or suggestions, ultimately leading to monotonous answers, low accuracy, and poor user experience. Summary of the Invention

[0007] The purpose of this application is to provide a crop knowledge graph question answering method and system based on large models and data assimilation, so as to at least solve the problems of weak cross-modal association analysis capabilities, large lag in the information provided in the answers, and the current content answers being singular, inaccurate, and having poor user experience in question answering systems of related technologies.

[0008] To achieve the above objectives, the embodiments of this application provide the following technical solutions.

[0009] According to one embodiment of this application, a crop knowledge graph question answering method based on large models and data assimilation is provided;

[0010] Includes the following steps:

[0011] Obtain a multimodal dataset of crops, which includes text data and image data;

[0012] Based on the Graph Attention Network (GAT) model, feature vectors from different modalities are fused to generate feature vectors with unified representation, thus constructing a multimodal data assimilation network.

[0013] Based on the assimilation network, an initial knowledge graph is generated using a dynamic graph neural network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model.

[0014] Based on the Large Language Model (LLM), knowledge is extracted and understood from the user's question. The output of the LLM is processed to generate a graph structured query template. The Prompt query template constructed by the LLM is obtained and fused with the knowledge graph. The optimal information query is obtained by using reinforcement learning path optimization methods. The graph query results obtained from each Prompt query template are integrated to generate a graph path query candidate set.

[0015] Question answering results are generated based on the optimized and adjusted LLM fusion graph query results. Specifically, the query question and graph path query candidate set are fused using an attention mechanism to calculate the answer confidence and obtain the answer candidate set. Multi-dimensional output is obtained by constructing a multi-layer Prompt to obtain the final question answering result.

[0016] Furthermore, the steps for constructing a multimodal data assimilation network specifically include:

[0017] Vector extractors are used to extract vectors from text data and image data respectively; for text data, the Word2vec vector extractor is used for vector extraction, and for image data, the ResNet network is used for vector extraction.

[0018] A cross-modal alignment module is used to learn feature weights from feature vectors of different modalities, calculate the inter-modal correlation weight matrix, and obtain the standardized feature vector of each entity.

[0019] Multimodal feature fusion is performed on the standardized feature vectors. The fused feature vectors are then input into the contrastive learning loss function, which is used to optimize the assimilation network. Multimodal features of the same entity are used as positive sample pairs, while negative sample pairs are constructed by randomly selecting and combining different entities. Through contrastive learning optimization, multimodal features of the same entity are brought closer together in the vector space, while multimodal features of different entities are moved further apart in the vector space.

[0020] Furthermore, the steps for generating the initial knowledge graph based on the dynamic graph neural network include:

[0021] Based on the assimilated dataset, a set of generated entities for crops is constructed; each entity includes multiple nodes, and each node contains multiple attributes, represented as follows: ;

[0022] Construct the set of relationships between entities in the entity set, represented as: ;

[0023] The features at each time point are spatially aggregated, and then temporal dependencies are captured through one-dimensional dilated convolution.

[0024] Spatiotemporal features are fused, and the Spatiotemporal Graph Convolutional Network ST-GCN is used to construct an initial graph of the multimodal assimilated dataset, represented as follows: ,in, , Represents an attribute vector. Represents a set of attribute vectors; , Represents relation weights. Represents the set of relation weights.

[0025] Furthermore, the steps for updating the initial knowledge graph based on the dynamic incremental graph update algorithm include:

[0026] Node state updates use a gated recurrent unit (GRU) model, combining the current node state and information from neighboring nodes to update attributes. The node state update is represented as:

[0027]

[0028] Where N(i) represents the neighboring nodes. This is a relation type weight matrix.

[0029] Furthermore, the steps for knowledge extraction and understanding of user questions based on the Large Language Model (LLM) include:

[0030] For image input data, the Qwen2.5-VL model is used to understand the input data;

[0031] For plain text data, Qwen2.5-7B is used for understanding and knowledge extraction;

[0032] The problem comprehension expression is as follows:

[0033]

[0034] In the formula, LLM represents Large Language Model, MLLM represents Multimodal Large Language Model, and Query_result represents the query result;

[0035] Based on the input data format, the corresponding large model is invoked and different prompts are constructed for knowledge extraction and understanding. The prompt is an instruction to the LLM model, and the prompt uses the few-shot method to understand and extract the problem.

[0036] Furthermore, in the step of obtaining the optimal information query using reinforcement learning path optimization methods, the reinforcement learning path optimization methods include:

[0037] Status: Current node ;

[0038] Action: Select the next jump relationship ;

[0039] Reward function: ;

[0040] Iterative selection of the optimal path using Q-learning:

[0041]

[0042] In the formula, Q(s,a) represents the expected reward for taking action a in state s. As a discount factor, It's the learning rate. Indicates the next state All possible actions The maximum Q value in, "" indicates the operation of updating the Q value in Q-learning.

[0043] Furthermore, after obtaining the optimal information query step, the process also includes:

[0044] Paths with similarity greater than the similarity threshold are redundantly filtered to obtain the first path result;

[0045] The formula for calculating the similarity between two nodes is as follows:

[0046]

[0047] In the formula, , Representing node 1 and node 2, , These represent the vector values ​​of node 1 and node 2, respectively.

[0048] Calculate the node similarity value if the following conditions are met: ,and If it is a redundant path, then it is determined to be a redundant path, where, The similarity threshold;

[0049] By pruning redundant paths, a second path is obtained, which is represented as follows:

[0050]

[0051] In the formula, Indicates the reserved path, , These represent the weight values ​​for path 1 and path 2, respectively.

[0052] Based on the results of the first path and the second path, the graph query results obtained from each query template are used to generate a graph path query candidate set.

[0053] Furthermore, the optimized and adjusted LLM is obtained through the following steps:

[0054] Offline collection of a large dataset q of historical search queries from users with different distributions;

[0055] Using the constructed knowledge graph, the collected historical dataset q is traversed to obtain the graph path set A corresponding to q;

[0056] The Prompt module processes the map path data to generate prediction results, management opinions, and response results. The response results data are then corrected and adjusted based on business rules.

[0057] The collected QA question-and-answer pairs are formatted as model input, and represented as follows:

[0058]

[0059] For the QA question-answer pairs, supervised learning is performed using an LLM model. This model maximizes the likelihood function of the target sequence and minimizes the cross-entropy loss. The loss function is as follows:

[0060]

[0061] Where N is the number of question-answer pairs and L is the length of the label. This is the output of the i-th sample in the first t-1 steps; Indicates that in a given input sequence and the output sequence of the previous time step Under the condition that the model parameters are When predicting the output of the i-th sample at the current time step t. The probability of;

[0062] By optimizing the training samples and parameters, a fine-tuned LLM model that performs best on the validation set is obtained.

[0063] Furthermore, the steps to obtain the final question-answering result by constructing multi-layered Prompts and generating multi-dimensional outputs include: simultaneously constructing multiple Prompts for the question in parallel, including a question parsing Prompt, an original construction Prompt, and a suggestion Prompt; then, nesting a result integration Prompt in the outermost layer; and finally, merging the outputs of the first-layer model to obtain the final question-answering result.

[0064] Another embodiment of the present invention provides a crop knowledge graph question answering system based on large models and data assimilation. This crop knowledge graph question answering system includes the following modules:

[0065] The data acquisition module is used to acquire multimodal datasets of crops, which include text data and image data;

[0066] The assimilation network construction module is used to fuse feature vectors from different modalities based on the graph attention network (GAT) model, generate feature vectors with unified representation, and construct a multimodal data assimilation network.

[0067] The knowledge graph construction and update module is used to generate an initial knowledge graph based on a dynamic graph neural network on the basis of the assimilation network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model.

[0068] The candidate set generation module is used to extract and understand user questions based on the Large Language Model (LLM), process the output of the LLM, and generate graph structured query templates; it obtains the Prompt query templates constructed by the LLM, integrates the Prompt query templates with the knowledge graph, uses reinforcement learning path optimization methods to obtain the optimal information query, and integrates the graph query results obtained from each Prompt query template to generate a graph path query candidate set;

[0069] The question-and-answer result generation module is used to generate question-and-answer results based on the optimized and adjusted LLM fusion graph query results. Specifically, it uses an attention mechanism to fuse the query question and the graph path query candidate set, calculates the answer confidence, obtains the answer candidate set, and outputs the final question-and-answer result by constructing a multi-layered Prompt.

[0070] Compared with existing technologies, the beneficial effects of the crop knowledge graph question answering method and system based on large models and data assimilation in this invention are:

[0071] First, this invention utilizes multimodal data fusion technology and uses the GAT algorithm to fuse data from multiple modalities to enrich the construction of the graph, enabling the question-answering system's content library to accurately cover the questions, thus significantly improving the final question-answering accuracy. This solves the problems of current question-answering systems in the bulk grain field being unable to handle image issues and having relatively simple question-answering content.

[0072] Second, this invention utilizes the GRU algorithm to implement a graph update mechanism on the knowledge graph, and proposes a reinforcement learning path optimization method to obtain the optimal information query. This not only enables the knowledge graph to update node information in real time, but also enables the dynamic optimization of query information, ensuring that the content input into the large model is the latest, most accurate, and most complete, thereby improving the timeliness, accuracy, and coverage of the final question answering system.

[0073] Third, this invention employs a multi-layer Prompt mechanism combined with QA collection to train an LLM model for a specific domain, which compensates for the lack of generalization ability of fine-tuning large models. This significantly improves the professionalism and accuracy of LLM's answers to questions in the field of bulk grains. At the same time, the multi-layer Prompt strategy corrects the LLM model results, resulting in a great improvement in user experience. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0075] Figure 1 This is an overall flowchart of the crop knowledge graph question answering method based on large model and data assimilation of the present invention;

[0076] Figure 2 This is a sub-flowchart of the crop knowledge graph question answering method based on large model and data assimilation of the present invention;

[0077] Figure 3 This is a schematic diagram illustrating the multimodal data fusion and knowledge graph construction of the present invention;

[0078] Figure 4 This is a schematic diagram illustrating the LLM model fine-tuning and content generation of this invention;

[0079] Figure 5 This is a structural block diagram of the crop knowledge graph question answering system based on large model and data assimilation of the present invention;

[0080] Figure 6 This is a structural block diagram of a computer device according to the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0082] This invention leverages the content richness capabilities of multimodal data, combines historical user query questions (q) to generate highly accurate QA / QA pairs, fine-tunes large-scale models in vertical domains, and constructs dynamic knowledge graphs. Ultimately, by building a multi-layered Prompt strategy, the overall question-answering accuracy, multifunctionality, and performance of the bulk grain question system are improved.

[0083] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0084] Please refer to Figure 1 In one embodiment of the present invention, a crop knowledge graph question answering method based on large models and data assimilation is provided, comprising the following steps:

[0085] S101. Obtain the multimodal dataset of the crop, which includes text data and image data;

[0086] In step S101 of this invention, which is the process of multimodal agricultural data acquisition and preprocessing, firstly, a multimodal dataset D={d1,d2,...,d...} related to bulk grains is acquired. i ,...,d M}, where d i This represents the i-th type of dataset, which mainly includes the following types of datasets:

[0087] Meteorological data: mainly temperature, precipitation, wind speed, etc.

[0088] Soil sensor data: mainly temperature, pH value, nutrient content, etc.

[0089] Remote sensing image data: mainly crop-related image data;

[0090] Market data: prices, supply and demand, etc.

[0091] Agricultural textual data: data such as policy documents and research reports;

[0092] Specifically, in this embodiment of the invention, data related to weather, market conditions, and agricultural text are crawled by a web crawler. Remote sensing images and soil-related growth data are collected by sensors. The data obtained by the web crawler is updated in real time by setting a timed task. The input is sent to a database for storage. Image data is saved to an address accessible via the intranet for convenient subsequent processing. Data collected by sensors is updated to the database in real time.

[0093] Furthermore, step S101 of the present invention also includes a data preprocessing process, wherein each type of data is standardized, numerical data is normalized to the [0-1] interval, text data is vectorized, and image data is extracted using multi-scale feature vectors. The formulas for standardization and normalization are as follows:

[0094] ;

[0095] ;

[0096] Where x is the original data point, For normalized data, The data is standardized, and u is the mean of the data, where denoted as the variance of the original data; min(x) represents the minimum input value, and max(x) represents the maximum input value.

[0097] Please continue to refer to Figure 1 The implementation steps of the question-answering method provided in this embodiment further include:

[0098] S102. Based on the graph attention network GAT model, feature vectors of different modalities are fused to generate feature vectors with unified representation, and a multimodal data assimilation network is constructed.

[0099] For further details, please refer to Figure 2 The steps of constructing a multimodal data assimilation network according to the present invention specifically include:

[0100] S201. Use a vector extractor to extract vectors from text data and image data respectively; for text data, use the Word2vec vector extractor to extract vectors, and for image data, use the ResNet network to extract vectors.

[0101] S202. The cross-modal alignment module is used to learn the feature weights of the feature vectors of different modalities, calculate the inter-modal correlation weight matrix, and obtain the standardized feature vector of each entity.

[0102] In step S202, for the multimodal vector data composed of the extracted text vectors and image vectors, feature weights are learned, and the intermodal correlation weight matrix is ​​calculated using the following formula, expressed as:

[0103] )

[0104] Where Q and K are the feature matrices for different modalities, and d is the feature dimension. This represents the incidence matrix, and T represents the transpose operation;

[0105] In step S202 of the present invention, the standardized feature vector of each entity, such as a land parcel or a weather station, is finally obtained.

[0106] S203. Perform feature fusion on the standardized feature vectors. Input the fused feature vectors into the contrastive learning loss function, and use the contrastive learning loss function to optimize the assimilation network. Multimodal features of the same entity are used as positive sample pairs, while negative sample pairs are constructed by randomly selecting combinations of different entities. Through contrastive learning optimization, multimodal features of the same entity are brought closer together in the vector space, while multimodal features of different entities are moved further apart. Specifically, the GAT model is used to fuse the normalized and standardized feature vectors of different modalities. The fused feature vector is represented as follows: ,in, Represents the sample set, Representing dimension, Represent each sample; the generated feature vector E i The input is fed into the contrastive learning loss function, which is used to optimize the assimilation network. Multimodal features of the same entity are used as positive sample pairs, and negative sample pairs are constructed by randomly combining different entities.

[0107] Through contrastive learning optimization, it is ensured that multimodal features of the same entity are close together in the vector space, while features of different entities are far apart; for example, multimodal features of the same entity could be sensor data and satellite imagery of a land parcel; the loss function of contrastive learning is expressed as:

[0108] in, These are hyperparameter variables used to control the smoothness of the distribution; Cosine similarity is used to calculate the distance between positive sample distribution 'a' and negative sample distribution 'b', where T represents the transpose operation; M is the maximum number of samples; and k represents the sample variable. Represents the input vector. Represents a positive correlation vector. Represents a negative correlation vector;

[0109] In calculating the loss function for contrastive learning At that time, , Substitution In, that is ;

[0110] Similarly, .

[0111] Please continue to refer to Figure 1 The crop knowledge graph question answering method based on large models and data assimilation provided by this invention further includes the following steps:

[0112] S103. Based on the assimilation network, an initial knowledge graph is generated using a dynamic graph neural network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model.

[0113] S104. Based on the Large Language Model (LLM), extract and understand the user's question, process the LLM output, and generate a graph structured query template; obtain the Prompt query template constructed by the LLM, integrate the Prompt query template with the knowledge graph, use reinforcement learning path optimization methods to obtain the optimal information query, integrate the graph query results obtained from each Prompt query template, and generate a graph path query candidate set.

[0114] S105. Question answering results are generated based on the optimized and adjusted LLM fusion graph query results; wherein, the query question and graph path query candidate set are fused using the attention mechanism, the answer confidence is calculated, the answer candidate set is obtained, and the final question answering results are obtained by constructing a multi-layer Prompt for multi-dimensional output.

[0115] like Figures 1-3 As shown, in this embodiment of the invention, step S103, the step of generating an initial knowledge graph based on a dynamic graph neural network, includes:

[0116] Based on the assimilated dataset, a generated entity set for food crops is constructed; each entity includes multiple nodes, and each node contains attributes, represented as follows: ;

[0117] Similarly, defining the related relation set, that is, constructing the relation set between entities in the entity set, is represented as:

[0118] ;

[0119] The features at each time point are spatially aggregated, and then temporal dependencies are captured through one-dimensional dilated convolution, as shown below: ,in, This represents the convolution operation. Representing node characteristics, Indicates spatiotemporal characteristics;

[0120] The spatiotemporal features are fused and represented as follows:

[0121]

[0122] in, Indicates fusion characteristics, This indicates a normalization operation;

[0123] The initial graph is constructed using the spatiotemporal graph convolutional network ST-GCN, represented as follows: ,in, , Represents an attribute vector. Represents a set of attribute vectors; , Represents relation weights. Represents the set of relation weights.

[0124] Furthermore, in step S103 of this invention, when real-time data arrives, the graph update process is initiated. Specifically, the initial knowledge graph is updated based on a dynamic incremental graph update algorithm, including:

[0125] Node state updates use a gated recurrent unit (GRU) model, combining the current node state and information from neighboring nodes to update attributes. The node state update is represented as:

[0126] ;

[0127] Where N(i) is the set of neighboring nodes, This is a relation type weight matrix; Let j represent the state of node i at time t, and j represent node j. This represents the relational weight between nodes i and j at time t. This represents the updated state of node i at time t+1.

[0128] Therefore, this invention utilizes multimodal data fusion technology to address the current problems of question-and-answer systems in the bulk grain field being unable to handle image-based questions and having relatively simple question-and-answer content. By fusing data from multiple modalities to enrich the construction of the graph, the question-and-answer system's content library can accurately cover the questions, resulting in a significant improvement in the final question-and-answer accuracy.

[0129] In addition, this invention utilizes the ability to construct dynamic knowledge graphs, which not only enables the knowledge graph to update node information in real time, but also ensures that the content input into the large model is the latest due to the accelerated data update cycle. This improves the timeliness of the final question-answering system, while the accuracy of the answers also increases as knowledge changes.

[0130] Furthermore, the steps for knowledge extraction and understanding of user questions based on the Large Language Model (LLM) include:

[0131] For multimodal data input from users, the data type is determined via input buttons. Specifically, for image input, the Qwen2.5-VL model is used for knowledge extraction and understanding; for plain text data, Qwen2.5-7B is used for knowledge extraction and understanding. The specific problem-solving expressions are shown below:

[0132]

[0133] In the formula, LLM represents a large language model, MLLM represents a multimodal large language model, and Q... The model recognition result is the final output obtained after the model processes the user input.

[0134] In this embodiment of the invention, Qwen2.5-VL is a model capable of processing multimodal data such as images; Qwen2.5-7B processes text and belongs to a single-modal language model.

[0135] In this embodiment of the invention, the user's input question is understood by performing different modal model processing based on the user's data source identifier; specifically, image data is directly understood using MLLM, and text data is understood using the graph fusion large model of the present invention, enabling the question answering system of the present invention to handle multimodal question input;

[0136] Furthermore, according to the input data format, the present invention calls the corresponding large model and constructs different Prompts for knowledge extraction and understanding. Here, the Prompt is an instruction to the LLM model, used to understand and extract the problem. The Prompt uses the few-shot method to understand and extract the problem.

[0137] Finally, the embodiments of the present invention process the output results of LLM to generate a graph structured query template.

[0138] Furthermore, this invention obtains the Prompt query template constructed by LLM, integrates the Prompt query template with the knowledge graph, and uses a reinforcement learning path optimization method to obtain the optimal information query. The specific reinforcement learning path optimization method includes:

[0139] Status: Current node ;

[0140] Action: Select the next jump relationship Where r represents a relation and R represents a set of relations;

[0141] Reward function: Where Reward represents the reward function. , Represents a weighted system;

[0142] Iterative selection of the optimal path using Q-learning:

[0143]

[0144] In the formula, Q(s,a) represents the expected reward (Q value) for taking action a in state s. As a discount factor, It's the learning rate. Indicates the next state All possible actions The maximum Q value in, " indicates the operation of updating the Q-value in Q-learning;

[0145] Furthermore, after obtaining the optimal information query step, the process also includes:

[0146] Paths with similarity greater than the similarity threshold are redundantly filtered to obtain the first path result;

[0147] The formula for calculating the similarity between two nodes is as follows:

[0148]

[0149] In the formula, , Representing node 1 and node 2, , These represent the vector values ​​of node 1 and node 2, respectively.

[0150] Calculate the node similarity value if the following conditions are met: ,and If it is a redundant path, then it is determined to be a redundant path, where, The similarity threshold;

[0151] Redundant paths are pruned, retaining those with larger weights to obtain the second path result, represented as follows:

[0152]

[0153] In the formula, Indicates the reserved path, , These represent the weight values ​​of path 1 and path 2, respectively. For example: when the weight value of path 1 is greater than the weight value of path 2, path 1 is retained as the second path result; when the weight value of path 1 is not greater than the weight value of path 2, path 2 is retained as the second path result.

[0154] Based on the results of the first path and the second path, the graph query results obtained from each query template are used to generate a graph path query candidate set.

[0155] Furthermore, such as Figure 4 As shown, in this embodiment of the invention, the optimized LLM is obtained through the following steps:

[0156] Offline collection of a large dataset q of historical search queries from users with different distributions;

[0157] Using the constructed knowledge graph, the collected historical dataset q is traversed to obtain the graph path set A corresponding to q;

[0158] The Prompt module processes the map path data, generates prediction results, management opinions, and response results, and corrects and adjusts the response results data in accordance with business rules.

[0159] The collected QA question-and-answer pairs are formatted as model input, and represented as follows:

[0160] ; , , This means that the input sample variable x (input question) is used.

[0161] ; , , This means that the input sample variable y is the output result.

[0162] For the QA question-answer pairs, supervised learning is performed using an LLM model. This model maximizes the likelihood function of the target sequence and minimizes the cross-entropy loss. The loss function is as follows: Where N is the number of question-answer pairs (qa) and L is the length of the label. This represents the input sequence for the i-th sample. This is the output of the i-th sample in the first t-1 steps; Indicates that in a given input sequence and the output sequence of the previous time step Under the condition that the model parameters are When predicting the output of the i-th sample at the current time step t. The probability of.

[0163] By optimizing the training samples and parameters, a fine-tuned LLM model that performs best on the validation set is obtained.

[0164] Furthermore, in this embodiment of the invention, in step S105, the formula for calculating the confidence level of the answer is expressed as:

[0165]

[0166] in, This represents the probability of answer a given question q. This represents the activation function. This represents the weight associated with the answer. q represents the information related to answer a in the knowledge graph, and q represents the question.

[0167] This invention takes the input query question and the high-confidence candidate answer set a, and outputs a comprehensive multi-dimensional result by constructing a multi-layered Prompt.

[0168] The steps involved in constructing multi-layered Prompts to generate multi-dimensional outputs and obtain the final question-and-answer result include: simultaneously constructing multiple Prompts for the question in parallel, including a question parsing Prompt, an original construction Prompt, and a suggestion Prompt; then, nesting a result integration Prompt at the outermost layer; and finally, merging the outputs of the first-layer model to obtain the final question-and-answer result.

[0169] This invention leverages the capabilities of the LLM model, not only by fine-tuning the LLM model for a specific domain through QA (question-answering) pairs, significantly improving the professionalism and accuracy of LLM's responses to questions in the bulk grain field, but also by employing a multi-layered Prompt mechanism strategy to give the final model's output multifunctional capabilities, greatly enhancing the user experience.

[0170] Please refer to Figure 5 In another embodiment of this application, a crop knowledge graph question-answering system based on large models and data assimilation is provided;

[0171] The question-and-answer system includes the following modules:

[0172] The data acquisition module 301 is used to acquire a multimodal dataset of crops, which includes text data and image data;

[0173] The assimilation network construction module 302 is used to perform feature fusion on feature vectors of different modalities based on the graph attention network GAT model, generate feature vectors with unified representation, and construct a multimodal data assimilation network;

[0174] The knowledge graph construction and update module 303 is used to generate an initial knowledge graph based on a dynamic graph neural network on the basis of the assimilation network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model.

[0175] The candidate set generation module 304 is used to extract and understand the user's question based on the large language model LLM, process the output of LLM, and generate a graph structured query template; obtain the Prompt query template constructed by LLM, integrate the Prompt query template with the knowledge graph, use reinforcement learning path optimization method to obtain the optimal information query, integrate the graph query results obtained from each Prompt query template, and generate a graph path query candidate set;

[0176] The question-and-answer result generation module 305 is used to generate question-and-answer results based on the optimized and adjusted LLM fusion graph query results. Specifically, it uses an attention mechanism to fuse the query question and the graph path query candidate set, calculates the answer confidence, obtains the answer candidate set, and outputs the final question-and-answer result by constructing a multi-layer Prompt.

[0177] like Figure 6 As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus.

[0178] The memory includes non-volatile storage media and internal memory. The non-volatile storage media of the computer device stores the operating system and may also store computer programs. When executed by the processor, these computer programs enable the processor to implement a crop knowledge graph question-answering method based on large models and data assimilation.

[0179] The internal memory may also store computer programs that, when executed by the processor, enable the processor to perform a crop knowledge graph question-answering method based on large models and data assimilation.

[0180] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the crop knowledge graph question-answering method based on large model and data assimilation provided in the above embodiment.

[0182] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A crop knowledge graph question-answering method based on large models and data assimilation, characterized in that, Includes the following steps: Obtain a multimodal dataset of crops, which includes text data and image data; Based on the Graph Attention Network (GAT) model, feature vectors from different modalities are fused to generate feature vectors with unified representation, thus constructing a multimodal data assimilation network. Based on the assimilation network, an initial knowledge graph is generated using a dynamic graph neural network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model. The steps for generating an initial knowledge graph based on a dynamic graph neural network include: constructing a set of generated entities for crops based on the assimilated dataset; wherein each entity includes multiple nodes, and each node contains multiple attributes, represented as follows: ； Construct the set of relationships between entities in the entity set, represented as: ； The features of each time point are spatially aggregated, and temporal dependencies are captured by one-dimensional dilated convolution; the spatiotemporal features are fused, and the spatiotemporal graph convolutional network ST-GCN is used to construct an initial graph of the dataset after multimodal assimilation. The steps for updating the initial knowledge graph using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model include: updating node states by using the GRU model to update attributes by combining the current node state and information from neighboring nodes; Based on the Large Language Model (LLM), knowledge is extracted and understood from the user's question. The output of the LLM is processed to generate a graph structured query template. The Prompt query template constructed by the LLM is obtained and fused with the knowledge graph. The optimal information query is obtained by using reinforcement learning path optimization methods. The graph query results obtained from each Prompt query template are integrated to generate a graph path query candidate set. Question answering results are generated based on the optimized and adjusted LLM fusion graph query results. Specifically, the query question and graph path query candidate set are fused using an attention mechanism to calculate the answer confidence and obtain the answer candidate set. Multi-dimensional output is then obtained by constructing a multi-layer Prompt to obtain the final question answering result. The optimized LLM is obtained through the following steps: A large dataset q of historical user search queries with different distributions is collected offline; using the constructed knowledge graph, the collected historical dataset q is traversed to obtain the graph path set corresponding to q; a Prompt is constructed to process the graph path data, generating prediction results, management opinions, and answer results, and the answer result data is corrected and adjusted based on business rules; the collected qa question-answer pairs are formatted as model input; the qa question-answer pairs are subjected to supervised learning using the LLM model, maximizing the likelihood function of the target sequence and minimizing the cross-entropy loss; and the fine-tuned LLM model with the best performance on the validation set is obtained by optimizing the training samples and parameters. The steps to obtain the final question-and-answer result by constructing multiple layers of Prompts and outputting multiple dimensions include: simultaneously constructing multiple Prompts for the question in parallel, including a question parsing Prompt, an original construction Prompt, and a suggestion Prompt; then nesting a result integration Prompt in the outermost layer; and finally merging the outputs of the first layer of the model to obtain the final question-and-answer result.

2. The crop knowledge graph question answering method based on large model and data assimilation according to claim 1, characterized in that, The steps for constructing a multimodal data assimilation network specifically include: Vector extractors are used to extract vectors from text data and image data respectively. For text data, the Word2vec vector extractor is used for vector extraction; for image data, the ResNet network is used for vector extraction. A cross-modal alignment module is used to learn feature weights from feature vectors of different modalities, calculate the inter-modal correlation weight matrix, and obtain the standardized feature vector of each entity. Multimodal feature fusion is performed on the standardized feature vectors. The fused feature vectors are then input into the contrastive learning loss function. The contrastive learning loss function is used to optimize the assimilation network. Multimodal features of the same entity are used as positive sample pairs, and negative sample pairs are constructed by randomly selecting and combining different entities. Through comparative learning optimization, the multimodal features of the same entity are brought closer together in the vector space, while the multimodal features of different entities are moved further apart in the vector space.

3. The crop knowledge graph question answering method based on large model and data assimilation according to claim 2, characterized in that, The node state update uses a gated recurrent unit (GRU) model, combining the current node state and information from neighboring nodes. In the step of updating attributes, the node state update is represented as follows: ; in, For the set of neighboring nodes, This is a relation type weight matrix; Let j represent the state of node i at time t, and j represent node j. This represents the relationship weight between node i and node j at time t. This represents the updated state of node i at time t+1.

4. The crop knowledge graph question answering method based on large model and data assimilation according to claim 3, characterized in that, The steps for knowledge extraction and understanding of user questions based on the Large Language Model (LLM) include: For image input data, the Qwen2.5-VL model is used to extract and understand the knowledge from the input data; For plain text data, knowledge extraction and understanding are performed by calling Qwen2.5-7B; The conditional expression is as follows: ； In the formula, LLM represents a large language model, MLLM represents a multimodal large language model, and Q... The model identifies the results; Based on the input data format, the corresponding large model is invoked and different prompts are constructed for knowledge extraction and understanding. The prompt is an instruction to the LLM model, and the prompt uses the few-shot method to understand and extract the problem.

5. The crop knowledge graph question answering method based on large model and data assimilation according to claim 4, characterized in that, In the step of obtaining the optimal information query using reinforcement learning path optimization methods, the reinforcement learning path optimization methods include: Status: Current node ; Action: Select the next jump relationship Where r represents a relation and R represents a set of relations; Reward function: ;in, Represents the reward function, , Represents a weighted system; Iterative selection of the optimal path using Q-learning: ; In the formula, Q(s,a) represents the expected reward for taking action a in state s. As a discount factor, It's the learning rate. Indicates the next state All possible actions The maximum Q value in, "" indicates the operation of updating the Q value in Q-learning.

6. The crop knowledge graph question answering method based on large model and data assimilation according to claim 5, characterized in that, After obtaining the optimal information query step, the following steps are also included: Paths with similarity greater than the similarity threshold are redundantly filtered to obtain the first path result; The formula for calculating the similarity between two nodes is as follows: ; In the formula, , Representing node 1 and node 2, , These represent the vector values ​​of node 1 and node 2, respectively. Calculate the node similarity value if the following conditions are met: ,and If it is a redundant path, then it is determined to be a redundant path, where, The similarity threshold; By pruning redundant paths, a second path is obtained, which is represented as follows: ; In the formula, Indicates the reserved path, , These represent the weight values ​​for path 1 and path 2, respectively. Based on the results of the first path and the second path, the graph query results obtained from each query template are used to generate a graph path query candidate set.

7. A crop knowledge graph question answering system for implementing the crop knowledge graph question answering method based on large model and data assimilation as described in any one of claims 1 to 6, characterized in that, Includes the following modules: The data acquisition module is used to acquire multimodal datasets of crops, which include text data and image data; The assimilation network construction module is used to fuse feature vectors from different modalities based on the graph attention network (GAT) model, generate feature vectors with unified representation, and construct a multimodal data assimilation network. The knowledge graph construction and update module is used to generate an initial knowledge graph based on a dynamic graph neural network on the basis of the assimilation network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model. The candidate set generation module is used to extract and understand the user's question based on the large language model LLM, process the output of LLM, and generate a graph structured query template. Obtain the Prompt query template constructed by LLM, integrate the Prompt query template with the knowledge graph, use reinforcement learning path optimization method to obtain the optimal information query, integrate the graph query results obtained from each Prompt query template, and generate a graph path query candidate set. The question-and-answer result generation module is used to generate question-and-answer results based on the optimized and adjusted LLM fusion graph query results. Specifically, it uses an attention mechanism to fuse the query question and the graph path query candidate set, calculates the answer confidence, obtains the answer candidate set, and outputs the final question-and-answer result by constructing a multi-layered Prompt.

Images