Crop knowledge graph question answering method and system based on large model and data assimilation

Abstract

The embodiment of the invention provides a crop knowledge graph question answering method and system based on a large model and data assimilation, and the method comprises the steps: generating an initial knowledge graph based on a dynamic graph neural network; updating the initial knowledge graph based on a dynamic incremental graph updating algorithm; performing knowledge extraction and understanding on questions of a user based on a large language model LLM, processing an output result of the LLM, and generating a graph structured query template; obtaining Prompt query templates constructed by LLM, fusing the Prompt query templates with the knowledge graph, obtaining optimal information query by applying a reinforcement learning path optimization method, and integrating graph query results obtained by the Prompt query templates; and performing question and answer result generation based on the optimized and adjusted LLM fusion graph query result. According to the method, multi-modal data are fused, a high-precision qa question-answer pair is generated in combination with historical user query questions q to finely adjust the vertical field large model, and the question-answer accuracy is improved by building a multi-layer Prompt strategy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of natural language processing technology, and more specifically, to a crop knowledge graph question-answering method and system based on a large model and data assimilation. Background Art

[0002] The bulk grain question-and-answer system is a question-and-answer system scenario in a specific field. Its goal is to build a question-and-answer system through algorithms so that users can quickly and accurately get answers to questions by inputting them. By providing fast and accurate information and analysis, the bulk grain question-and-answer system helps users in various fields improve efficiency, reduce costs, and enhance competitiveness, playing an irreplaceable role in the entire grain industry.

[0003] With the continuous development of question-answering system technology, question-answering system technology has also been greatly improved, but it still faces the following problems:

[0004] First, there's the issue of multimodal data silos: The current bulk grain Q&A system uses a single-modal approach, parsing user questions. However, due to data incompleteness, the system's cross-modal correlation analysis capabilities are weak, making it difficult to support integrated decision-making across weather, soil, and market factors.

[0005] Second, the problem of static knowledge graphs: Although current question-answering systems integrate data using knowledge graphs, which has improved results compared to previous technologies, current knowledge graphs are mostly statically constructed based on historical data and cannot adapt to dynamic changes. This can lead to a significant lag in answer information and inaccurate answers.

[0006] Third, the monotony of answer content and poor user experience: Traditional knowledge graph-based answers are generated based on several sets of rule models. The answers users receive are fixed and the language is limited. Although many people are currently trying to use LLM models to address the diversity of question answers, the general large model has low accuracy in answering questions in specific fields. At the same time, the current answers are single-function dialogues without more reasons and suggestions. Ultimately, the current content answers are monotonous, with low accuracy and poor user experience. Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide a crop knowledge graph question-answering method and system based on a large model and data assimilation, so as to at least solve the problems existing in the question-answering system of the related art, such as weak cross-modal correlation analysis capability, large lag in the information of the answer, and single and low accuracy of the current content answer and poor user experience.

[0008] To achieve the above objectives, the embodiments of the present application provide the following technical solutions.

[0009] According to one embodiment of the present application, a crop knowledge graph question-answering method based on a large model and data assimilation is provided;

[0010] The following steps are involved:

[0011] Obtain a multimodal dataset of crops, which includes text data and image data;

[0012] Based on the graph attention network (GAT) model, feature vectors of different modalities are fused to generate a unified feature vector and construct a multimodal data assimilation network.

[0013] Based on the assimilation network, the initial knowledge graph is generated based on the dynamic graph neural network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on the gated recurrent unit (GRU) model.

[0014] Based on the large language model (LLM), knowledge is extracted and understood from user questions. The output of the LLM is processed to generate a graph-structured query template. The prompt query template constructed by the LLM is obtained, and the prompt query template is integrated with the knowledge graph. The optimal information query is obtained using the reinforcement learning path optimization method. The graph query results obtained from each prompt query template are integrated to generate a graph path query candidate set.

[0015] The question-answering results are generated based on the optimized and adjusted LLM fusion graph query results. The attention mechanism is used to fuse the query question and the graph path query candidate set, calculate the answer confidence, and obtain the answer candidate set. By constructing multi-layer prompts for multi-dimensional output, the final question-answering results are obtained.

[0016] Furthermore, the steps of constructing a multimodal data assimilation network include:

[0017] Vector extractors are used to extract vectors from text data and image data respectively. For text data, the Word2vec vector extractor is used for vector extraction, and for image data, the Resnet network is used for vector extraction.

[0018] The cross-modal alignment module is used to learn the feature weights of the feature vectors of different modalities, calculate the inter-modal correlation weight matrix, and obtain the standardized feature vector of each entity;

[0019] Multimodal feature fusion is performed on the standardized feature vector, and the fused feature vector is input into the contrastive learning loss function. The contrastive learning loss function is used to optimize the assimilation network. The multimodal features of the same entity are used as positive sample pairs, and negative sample pairs of different entities are constructed by randomly selecting and combining them. Through contrastive learning optimization, the multimodal features of the same entity are close in the vector space, while the multimodal features of different entities are far away in the vector space.

[0020] Furthermore, the steps of generating an initial knowledge graph based on the dynamic graph neural network include:

[0021] Based on the assimilated dataset, a crop entity set is constructed; each entity includes multiple nodes, and each node contains multiple attributes, which can be expressed as: ;

[0022] Construct a set of relationships between entities in an entity set, expressed as: ;

[0023] The features of each time node are spatially aggregated, and then the temporal dependency is captured through one-dimensional hole convolution;

[0024] The spatiotemporal features are fused, and the spatiotemporal graph convolutional network ST-GCN is used to construct the initial graph of the multimodal assimilated dataset, which is expressed as: ,in, , represents the attribute vector, Represents a set of attribute vectors; , represents the relationship weight, Represents a set of relationship weights.

[0025] Furthermore, the step of updating the initial knowledge graph based on the dynamic incremental graph update algorithm includes:

[0026] The node status update uses the gated recurrent unit GRU model to combine the current node status and the information of neighboring nodes to update the attributes. The node status update is expressed as:

[0027]

[0028] Among them, N(i) is the neighbor node, is the relationship type weight matrix.

[0029] Furthermore, the steps of extracting knowledge and understanding user questions based on the Large Language Model (LLM) include:

[0030] For image input data, the Qwen2.5-VL model is called to understand the input data;

[0031] For plain text data, Qwen2.5-7B is called for understanding and knowledge extraction;

[0032] The problem comprehension expression is as follows:

[0033]

[0034] Where LLM represents the large language model, MLLM represents the multimodal large language model; Query_result represents the query result;

[0035] According to the input data format, the corresponding large model is called and different prompts are constructed to extract and understand knowledge. Among them, prompt is an instruction to the LLM model, and prompt uses the few shot method to understand and extract the problem.

[0036] Furthermore, in the step of obtaining the optimal information query using the reinforcement learning path optimization method, the reinforcement learning path optimization method includes:

[0037] Status: Current Node ;

[0038] Action: Select next hop relationship ;

[0039] Reward function: ;

[0040] Select the optimal path through Q-learning iteration:

[0041]

[0042] Where Q(s,a) represents the expected reward for taking action a in state s, is the discount factor, is the learning rate, Indicates the next state All possible moves The maximum Q value in the ” represents the operation of updating the Q value in Q-learning.

[0043] Furthermore, after obtaining the optimal information query step, the method further includes:

[0044] Perform redundancy filtering on paths whose similarity is greater than a similarity threshold to obtain a first path result;

[0045] Among them, the formula for calculating the similarity between two nodes is:

[0046]

[0047] Where, 、 Represents node 1 and node 2, 、 Represent the vector values ​​of node 1 and node 2 respectively;

[0048] Calculate the node similarity value if the conditions are met: ,and , then it is determined to be a redundant path, where, is the similarity threshold;

[0049] Prune the redundant path and get the second path result, which is expressed as:

[0050]

[0051] Where, Indicates the reserved path, 、 Represent the weight values ​​of path 1 and path 2 respectively;

[0052] Based on the first path result and the second path result, the graph query results obtained by each query template are used to generate a graph path query candidate set.

[0053] Furthermore, the optimized LLM is obtained by the following steps:

[0054] Collect a large number of query question datasets q of user historical searches with different distributions offline;

[0055] Use the constructed knowledge graph to traverse the collected historical data set q and obtain the graph path set A corresponding to q;

[0056] Build Prompt to process graph path data, generate prediction results, management opinions, and answer results, and modify and adjust the answer result data based on business rules;

[0057] The collected QA question-answer pairs are formatted as model input, expressed as follows:

[0058]

[0059] For QA question-answer pairs, the LLM model is used for supervised learning, which maximizes the likelihood function of the target sequence and minimizes the cross entropy loss. The loss function is:

[0060]

[0061] Among them, N is the number of qa question-answer pairs, L is the length of the label, is the output of the t-1 step before the i-th sample; Indicates that given an input sequence and the output sequence of the previous time step Under the condition of , the model parameters are When , predict the output of the i-th sample at the current time step t probability;

[0062] By optimizing the training samples and parameters, the fine-tuned LLM model with the best effect on the validation set was obtained.

[0063] Furthermore, by constructing multiple layers of prompts for multi-dimensional output, the steps to obtain the final question-answering result include: constructing multiple prompts for the question in parallel, including a question analysis prompt, an original construction prompt, and a suggestion prompt, then inserting a result integration prompt in the outermost layer, and finally merging the model result output of the first layer to obtain the final question-answering result.

[0064] Another embodiment of the present invention provides a crop knowledge graph question-answering system based on a large model and data assimilation. The crop knowledge graph question-answering system includes the following modules:

[0065] A data acquisition module is used to acquire a multimodal dataset of crops, where the multimodal dataset includes text data and image data;

[0066] The assimilation network construction module is used to fuse feature vectors of different modalities based on the graph attention network (GAT) model, generate a unified feature vector, and build a multimodal data assimilation network.

[0067] The knowledge graph construction and update module is used to generate the initial knowledge graph based on the dynamic graph neural network on the basis of the assimilation network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on the gated recurrent unit (GRU) model.

[0068] The candidate set generation module is used to extract and understand user questions based on the large language model (LLM), process the LLM output results, and generate graph-structured query templates. It obtains the Prompt query template constructed by the LLM, integrates the Prompt query template with the knowledge graph, uses the reinforcement learning path optimization method to obtain the optimal information query, and integrates the graph query results obtained from each Prompt query template to generate a graph path query candidate set.

[0069] The question-answering result generation module is used to generate question-answering results based on the optimized and adjusted LLM fusion graph query results. The attention mechanism is used to fuse the query question and the graph path query candidate set, calculate the answer confidence, and obtain the answer candidate set. By constructing a multi-layer prompt for multi-dimensional output, the final question-answering result is obtained.

[0070] Compared with the existing technology, the crop knowledge graph question-answering method and system based on large models and data assimilation have the following beneficial effects:

[0071] First, the present invention utilizes multimodal data fusion technology, fusing data from multiple modalities through the GAT algorithm to enrich the construction of the graph, enabling the question-answering system's content library to accurately cover questions, significantly improving the final question-answering accuracy. This addresses the current issues in the bulk grain sector where question-answering systems cannot handle image questions and have relatively limited question-answering content.

[0072] Second, the present invention uses the GRU algorithm to implement a graph update mechanism on the knowledge graph, and proposes a reinforcement learning-based path optimization method to obtain optimal information query. This not only enables the knowledge graph to update node information in real time, but also dynamically optimizes query information, ensuring that the content input into the large model is the latest, most accurate, and most comprehensive, thereby improving the timeliness, accuracy, and coverage of the final question-answering system.

[0073] Third, the present invention adopts a multi-layer prompt mechanism combined with the collection of QA questions and answers to train the LLM model in a specific field, which makes up for the problem of insufficient generalization ability of fine-tuning large models, and greatly improves the professionalism and accuracy of LLM in answering questions in the field of bulk grain. At the same time, due to the multi-layer prompt strategy to correct the LLM model results, the user experience has been greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention.

[0075] Figure 1 This is an overall flow chart of the crop knowledge graph question-answering method based on a large model and data assimilation of the present invention;

[0076] Figure 2 This is a sub-flowchart of the crop knowledge graph question-answering method based on a large model and data assimilation of the present invention;

[0077] Figure 3 Schematic diagram of the multimodal data fusion and knowledge graph construction of the present invention;

[0078] Figure 4 Schematic diagram of LLM model fine-tuning and content generation in the present invention;

[0079] Figure 5 This is a structural block diagram of the crop knowledge graph question-answering system based on a large model and data assimilation according to the present invention;

[0080] Figure 6 The figure is a structural block diagram of a computer device according to the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0082] The present invention integrates the content richness of multimodal data, combines historical user query questions q to generate highly accurate qa question-answer pairs, fine-tunes the vertical field model, and constructs a dynamic knowledge graph. Finally, by building a multi-layer prompt strategy, the accuracy, versatility and performance of the overall bulk grain question system are improved.

[0083] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0084] Please refer to Figure 1 In one embodiment of the present invention, a crop knowledge graph question answering method based on a large model and data assimilation is provided, comprising the following steps:

[0085] S101. Acquire a multimodal dataset of crops, where the multimodal dataset includes text data and image data.

[0086] In step S101 of the present invention, which is a process of multimodal agricultural data collection and preprocessing, firstly, a multimodal dataset D={d1, d2, ..., d i ,...,d M}, where d i Represents the dataset of type i, which mainly includes the following types of datasets:

[0087] Meteorological related data: mainly temperature, precipitation, wind speed, etc.;

[0088] Soil sensor data: mainly temperature, pH value, nutrient content, etc.;

[0089] Remote sensing image data: mainly crop-related image data;

[0090] Market data: price, supply and demand, etc.

[0091] Agricultural text data: policy documents, scientific research reports, etc.

[0092] Specifically, in an embodiment of the present invention, a crawler is used to crawl data related to weather, market conditions, and agricultural text. Remote sensing images and soil growth data are collected through sensors. The data obtained by the crawler is updated in real time by setting a scheduled task. The input is passed to a database for storage. The image data is saved to an address accessible on the intranet to facilitate subsequent processing. The data collected by the sensors is updated in real time to the database.

[0093] Furthermore, step S101 of the present invention also includes a data preprocessing process, wherein each type of data is standardized, numerical data is normalized to the interval [0-1], text data is vectorized, and image data is subjected to multi-scale feature vector extraction, wherein the standardization and normalization formulas are as follows:

[0094] ;

[0095] ;

[0096] Where x is the original data point, For normalized data, is the standardized data, u is the mean of the data, where is the variance of the original data; min(x) represents the minimum input value, and max(x) represents the maximum input value.

[0097] Please continue to refer to Figure 1 The steps for implementing the question-answering method provided in the embodiment of the present disclosure further include:

[0098] S102. Based on the graph attention network (GAT) model, feature vectors of different modalities are fused to generate a unified representation feature vector and build a multimodal data assimilation network.

[0099] For further information, please refer to Figure 2 The steps of constructing a multimodal data assimilation network in the present invention specifically include:

[0100] S201, using a vector extractor to extract vectors from text data and image data respectively; for text data, using a Word2vec vector extractor to extract vectors, and for image data, using a Resnet network to extract vectors;

[0101] S202, using a cross-modal alignment module to perform feature weight learning on feature vectors of different modalities, calculating an inter-modal correlation weight matrix, and obtaining a standardized feature vector for each entity;

[0102] In step S202, for the multimodal vector data composed of the extracted text vectors and image vectors, feature weight learning is performed and the inter-modal association weight matrix is ​​calculated using the following formula, which is expressed as:

[0103] )

[0104] Among them, Q and K are the feature matrices of different modes respectively, and d is the feature dimension. represents the incidence matrix, T represents the transpose operation;

[0105] In step S202 of the present invention, a standardized feature vector of each entity such as a land parcel, a weather station, etc. is finally obtained;

[0106] S203, perform feature fusion on the standardized feature vector, input the fused feature vector into the contrastive learning loss function, use the contrastive learning loss function to optimize the assimilation network, use the multimodal features of the same entity as the positive sample pair, and construct the negative sample pair of different entities by randomly selecting and combining them; through contrastive learning optimization, the multimodal features of the same entity are close in the vector space, and the multimodal features of different entities are far away in the vector space; specifically, use the GAT model to fuse the normalized and standardized different modal feature vectors, and the fused representation feature vector is expressed as: ,in, represents the sample set, Represents dimension, Represents each sample; the generated feature vector E i The input is fed into the contrastive learning loss function, which is used to optimize the assimilation network. The multimodal features of the same entity are used as positive sample pairs, and negative sample pairs of different entities are constructed by random combination.

[0107] Through contrastive learning optimization, we ensure that the multimodal features of the same entity are close in the vector space, while the features of different entities are far apart. For example, the multimodal features of the same entity can be sensor data and satellite images of a certain plot of land. The loss function of contrastive learning is expressed as:

[0108] in, is a hyperparameter variable used to control the smoothness of the distribution; is the cosine similarity, which is used to calculate the distance between the distribution of positive sample a and the distribution of negative sample b. T represents the transposition operation; M is the maximum number of samples; k represents the sample variable; represents the input vector, represents the positive correlation vector, represents the negative correlation vector;

[0109] In calculating the loss function of contrastive learning When 、 Substitution In, that is ;

[0110] Similarly, .

[0111] Please continue to refer to Figure 1 The crop knowledge graph question-answering method based on a large model and data assimilation provided by the present invention further includes the following steps:

[0112] S103. Based on the assimilation network, an initial knowledge graph is generated based on a dynamic graph neural network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on a gated recurrent unit (GRU) model.

[0113] S104. Extract and understand the user's question based on the large language model (LLM), process the LLM output, and generate a graph-structured query template; obtain the Prompt query template constructed by the LLM, fuse the Prompt query template with the knowledge graph, use the reinforcement learning path optimization method to obtain the optimal information query, integrate the graph query results obtained from each Prompt query template, and generate a graph path query candidate set;

[0114] S105. Generate question-and-answer results based on the optimized and adjusted LLM fusion graph query results; wherein, the attention mechanism is used to fuse the query question and the graph path query candidate set, calculate the answer confidence, and obtain the answer candidate set. By constructing multi-layer prompts for multi-dimensional output, the final question-and-answer result is obtained.

[0115] like Figure 1-Figure 3 As shown, in this embodiment of the present invention, in step S103, the step of generating an initial knowledge graph based on a dynamic graph neural network includes:

[0116] Based on the assimilated dataset, a set of generated entities for food crops is constructed; each entity includes multiple nodes, and each node contains attributes, which are expressed as: ;

[0117] Similarly, define the related relationship set, that is, construct the relationship set between entities in the entity set, which is expressed as:

[0118] ;

[0119] The features of each time node are spatially aggregated, and then the temporal dependency is captured by one-dimensional hole convolution, which is expressed as: ,in, represents the convolution operation, Represents node features, Represents spatiotemporal characteristics;

[0120] The spatiotemporal features are fused and expressed as:

[0121]

[0122] in, represents the fusion feature, Represents normalization operation;

[0123] Use the spatiotemporal graph convolutional network ST-GCN to construct the initial graph of the dataset after multimodal assimilation, which is expressed as: ,in, , represents the attribute vector, Represents a set of attribute vectors; , represents the relationship weight, Represents a set of relationship weights.

[0124] Furthermore, in step S103 of the present invention, when real-time data arrives, the graph update process is initiated. Specifically, the initial knowledge graph is updated based on a dynamic incremental graph update algorithm, including:

[0125] The node status update uses the gated recurrent unit GRU model to combine the current node status and the information of neighboring nodes to update the attributes. The node status update is expressed as:

[0126] ;

[0127] Among them, N(i) is the set of neighbor nodes, is the relationship type weight matrix; represents the state of node i at time t, j represents the node, represents the relationship weight between nodes i and j at time t, Represents the updated state of node i at time t+1.

[0128] Therefore, the present invention utilizes multimodal data fusion technology to address the problems that the current question-answering system in the bulk grain field cannot handle image questions and the question-answering content is relatively single. By fusing data from multiple modalities to enrich the construction of the graph, the question-answering system content library can accurately cover the questions, thereby significantly improving the final question-answering accuracy.

[0129] In addition, the present invention utilizes the dynamic knowledge graph construction capability, which not only enables the knowledge graph to update node information in real time, but also accelerates the data update cycle so that the content input into the large model is the latest content, thereby improving the timeliness of the final question-answering system. At the same time, the accuracy of the questions is also improved as the knowledge changes.

[0130] Furthermore, the steps of extracting knowledge and understanding user questions based on the large language model (LLM) include:

[0131] For multimodal data input by the user, the data type is determined by the input button. For image input data, the Qwen2.5-VL model is called to extract and understand the input data; for plain text data, the Qwen2.5-7B model is called to extract and understand the input data. The specific problem judgment expression is as follows:

[0132]

[0133] In the formula, LLM represents the large language model, MLLM represents the multimodal large language model, and Q It is the model recognition result, that is, the final output obtained after the model processes the user input.

[0134] In the embodiment of the present invention, Qwen2.5-VL is a model that can process multimodal data such as images; Qwen2.5-7B processes text and is a unimodal language model.

[0135] In the embodiment of the present invention, the user's input question is understood by performing model processing of different modalities based on the user's data source identification. Specifically, image data is directly understood through MLLM, and text data is understood through the graph fusion model of the present invention, so that the question-answering system of the present invention can handle multi-modal question input.

[0136] Furthermore, the present invention calls the corresponding large model and constructs different prompts to extract and understand knowledge according to the input data format. Prompt is an instruction to the LLM model for understanding and extracting the problem. Prompt uses a few shot method to understand and extract the problem.

[0137] Finally, the embodiment of the present invention processes the output results of LLM to generate a graph structured query template.

[0138] Furthermore, the present invention obtains the Prompt query template constructed by LLM, integrates the Prompt query template with the knowledge graph, and uses the reinforcement learning path optimization method to obtain the optimal information query. The specific reinforcement learning path optimization method includes:

[0139] Status: Current Node ;

[0140] Action: Select next hop relationship ; Among them, r represents a relationship, and R represents a relationship set;

[0141] Reward function: ; Among them, Reward represents the reward function, 、 Represents a weight system;

[0142] Select the optimal path through Q-learning iteration:

[0143]

[0144] Where Q(s,a) represents the expected reward (Q value) of taking action a in state s, is the discount factor, is the learning rate, Indicates the next state All possible moves The maximum Q value in the " represents the operation of updating the Q value in Q-learning;

[0145] Furthermore, after obtaining the optimal information query step, the method further includes:

[0146] Perform redundancy filtering on paths whose similarity is greater than a similarity threshold to obtain a first path result;

[0147] Among them, the formula for calculating the similarity between two nodes is:

[0148]

[0149] Where, 、 Represents node 1 and node 2, 、 Represent the vector values ​​of node 1 and node 2 respectively;

[0150] Calculate the node similarity value if the conditions are met: ,and , then it is determined to be a redundant path, where, is the similarity threshold;

[0151] Prune the redundant paths and retain the paths with larger weights to obtain the second path result, which is expressed as:

[0152]

[0153] Where, Indicates the reserved path, 、 Represent the weight values ​​of path 1 and path 2 respectively; illustratively: when the weight value of path 1 is greater than the weight value of path 2, path 1 is retained as the second path result; when the weight value of path 1 is not greater than the weight value of path 2, path 2 is retained as the second path result;

[0154] Based on the first path result and the second path result, the graph query results obtained by each query template are used to generate a graph path query candidate set.

[0155] Further, such as Figure 4 As shown, in this embodiment of the present invention, the optimized and adjusted LLM is obtained by the following steps:

[0156] Collect a large number of query question datasets q of user historical searches with different distributions offline;

[0157] Use the constructed knowledge graph to traverse the collected historical data set q and obtain the graph path set A corresponding to q;

[0158] Build Prompt to process graph path data, generate prediction results, management opinions, and answer results, and modify and adjust the answer result data based on business rules;

[0159] The collected QA question-answer pairs are formatted as model input, expressed as follows:

[0160] ; 、 、 The meaning is the input sample x variable (input problem);

[0161] ; 、 、 It means the input sample y variable (output result);

[0162] For QA question-answer pairs, the LLM model is used for supervised learning, which maximizes the likelihood function of the target sequence and minimizes the cross entropy loss. The loss function is: ; Where N is the number of qa question-answer pairs, L is the length of the label, represents the input sequence of the i-th sample, is the output of the t-1 step before the i-th sample; Indicates that given an input sequence and the output sequence of the previous time step Under the condition of , the model parameters are When , predict the output of the i-th sample at the current time step t probability.

[0163] By optimizing the training samples and parameters, the fine-tuned LLM model with the best effect on the validation set was obtained.

[0164] Furthermore, in the embodiment of the present invention, in step S105, the answer confidence is calculated using the formula:

[0165]

[0166] in, represents the probability of answer a given question q, represents the activation function, represents the weight associated with the answer, represents the information related to the answer a in the knowledge graph, and q represents the question;

[0167] The present invention takes as input a query question and a high-confidence answer candidate set a, and constructs a multi-layer prompt to perform a comprehensive multi-dimensional output.

[0168] The steps of constructing multiple layers of prompts for multi-dimensional output to obtain the final question-answering result include: constructing multiple prompts for the question in parallel, including question analysis prompts, original construction prompts, and suggestion prompts, then inserting a result integration prompt in the outermost layer, and finally merging the model result outputs of the first layer to obtain the final question-answering result.

[0169] This invention utilizes the capabilities of the LLM model. It not only fine-tunes the LLM model in specific fields by using QA question-answer pairs, significantly improving the professionalism and accuracy of LLM in answering questions in the bulk grain field, but also adopts a multi-layer prompt mechanism strategy to make the output results of the final model multifunctional, greatly improving the user experience.

[0170] Please refer to Figure 5 In another embodiment of the present application, a crop knowledge graph question-answering system based on a large model and data assimilation is provided;

[0171] The question-answering system includes the following modules:

[0172] A data acquisition module 301 is used to acquire a multimodal dataset of crops, where the multimodal dataset includes text data and image data;

[0173] Assimilation network construction module 302, used to perform feature fusion on feature vectors of different modalities based on the graph attention network (GAT) model, generate a unified representation feature vector, and construct a multimodal data assimilation network;

[0174] The knowledge graph construction and update module 303 is used to generate an initial knowledge graph based on the dynamic graph neural network on the basis of the assimilation network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on the gated recurrent unit (GRU) model.

[0175] The candidate set generation module 304 is used to extract and understand the user's question based on the large language model (LLM), process the LLM output results, and generate a graph structured query template; obtain the prompt query template constructed by the LLM, fuse the prompt query template with the knowledge graph, use the reinforcement learning path optimization method to obtain the optimal information query, and integrate the graph query results obtained by each prompt query template to generate a graph path query candidate set;

[0176] The question-and-answer result generation module 305 is used to generate question-and-answer results based on the optimized and adjusted LLM fusion graph query results; wherein, the attention mechanism is used to fuse the query question and the graph path query candidate set, calculate the answer confidence, obtain the answer candidate set, and obtain the final question-and-answer result by constructing multi-layer prompts for multi-dimensional output.

[0177] like Figure 6 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus.

[0178] The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When executed by the processor, the computer program enables the processor to implement a crop knowledge graph question-answering method based on a large model and data assimilation.

[0179] The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute a crop knowledge graph question-answering method based on a large model and data assimilation.

[0180] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the crop knowledge graph question-answering method based on a large model and data assimilation provided in the above embodiment.

[0182] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0183] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A crop knowledge graph question-answering method based on a large model and data assimilation, characterized by: The following steps are involved: Obtain a multimodal dataset of crops, which includes text data and image data; Based on the graph attention network (GAT) model, feature vectors of different modalities are fused to generate a unified feature vector and construct a multimodal data assimilation network. Based on the assimilation network, the initial knowledge graph is generated based on the dynamic graph neural network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on the gated recurrent unit (GRU) model. Extract and understand user questions based on the Large Language Model (LLM), process the LLM output, and generate a graph-structured query template. Obtain the Prompt query template constructed by LLM, fuse the Prompt query template with the knowledge graph, use the reinforcement learning path optimization method to obtain the optimal information query, integrate the graph query results obtained by each Prompt query template, and generate a graph path query candidate set; The question-answering results are generated based on the optimized and adjusted LLM fusion graph query results. The attention mechanism is used to fuse the query question and the graph path query candidate set, calculate the answer confidence, and obtain the answer candidate set. By constructing multi-layer prompts for multi-dimensional output, the final question-answering results are obtained.

2. The crop knowledge graph question answering method based on large model and data assimilation according to claim 1 is characterized in that: The steps to build a multimodal data assimilation network include: Use vector extractors to extract vectors from text data and image data respectively; For text data, we use the Word2vec vector extractor to extract vectors, and for image data, we use the Resnet network to extract vectors. The cross-modal alignment module is used to learn the feature weights of the feature vectors of different modalities, calculate the inter-modal correlation weight matrix, and obtain the standardized feature vector of each entity; Perform multimodal feature fusion on the standardized feature vector, input the fused feature vector into the contrastive learning loss function, use the contrastive learning loss function to optimize the assimilation network, use the multimodal features of the same entity as the positive sample pair, and construct the negative sample pair of different entities by randomly selecting and combining them; Through contrastive learning optimization, the multimodal features of the same entity are brought close in the vector space, while the multimodal features of different entities are moved away in the vector space.

3. The crop knowledge graph question answering method based on large model and data assimilation according to claim 2 is characterized in that: The steps to generate the initial knowledge graph based on the dynamic graph neural network include: Based on the assimilated dataset, a crop entity set is constructed; each entity includes multiple nodes, and each node contains multiple attributes, which can be expressed as: ; Construct a set of relationships between entities in an entity set, expressed as: ; Spatial aggregation of features at each time node, capturing temporal dependencies through one-dimensional dilated convolution; The spatiotemporal features are fused and the spatiotemporal graph convolutional network ST-GCN is used to construct the initial graph of the dataset after multimodal assimilation.

4. The crop knowledge graph question answering method based on large model and data assimilation according to claim 3 is characterized in that: The steps of updating the initial knowledge graph using a dynamic incremental graph update algorithm based on the gated recurrent unit (GRU) model include: The node status update uses the gated recurrent unit GRU model to combine the current node status and the information of neighboring nodes to update the attributes. The node status update is expressed as: in, is the set of neighbor nodes, is the relationship type weight matrix; represents the state of node i at time t, j represents the node, represents the relationship weight between node i and node j at time t, Represents the updated state of node i at time t+1.

5. The crop knowledge graph question answering method based on large model and data assimilation according to claim 4 is characterized in that: The steps for extracting knowledge and understanding user questions based on the Large Language Model (LLM) include: For image input data, the Qwen2.5-VL model is called to extract and understand the input data; For plain text data, knowledge extraction and understanding are performed by calling Qwen2.5-7B; The problem judgment expression is as follows: ； In the formula, LLM represents the large language model, MLLM represents the multimodal large language model, and Q Identify the results for the model; According to the input data format, the corresponding large model is called and different prompts are constructed to extract and understand knowledge. Among them, prompt is an instruction to the LLM model, and prompt uses the few shot method to understand and extract the problem.

6. The crop knowledge graph question answering method based on large model and data assimilation according to claim 5 is characterized in that: In the step of obtaining the optimal information query by using the reinforcement learning path optimization method, the reinforcement learning path optimization method includes: Status: Current Node ; Action: Select next hop relationship ; Among them, r represents a relationship, and R represents a relationship set; Reward function: ;in, represents the reward function, 、 Represents a weight system; Select the optimal path through Q-learning iteration: Where Q(s,a) represents the expected reward for taking action a in state s, is the discount factor, is the learning rate, Indicates the next state All possible moves The maximum Q value in " ” represents the operation of updating the Q value in Q-learning.

7. The crop knowledge graph question answering method based on large model and data assimilation according to claim 6 is characterized in that: After obtaining the optimal information query step, it also includes: Perform redundancy filtering on paths whose similarity is greater than a similarity threshold to obtain a first path result; Among them, the formula for calculating the similarity between two nodes is: Where, 、 Represents node 1 and node 2, 、 Represent the vector values ​​of node 1 and node 2 respectively; Calculate the node similarity value if the conditions are met: ,and , then it is determined to be a redundant path, where, is the similarity threshold; Prune the redundant path and get the second path result, which is expressed as: Where, Indicates the reserved path, 、 Represent the weight values ​​of path 1 and path 2 respectively; Based on the first path result and the second path result, the graph query results obtained by each query template are used to generate a graph path query candidate set.

8. The crop knowledge graph question answering method based on large model and data assimilation according to claim 7 is characterized in that: The optimized LLM is obtained through the following steps: Collect a large number of query question datasets q of user historical searches with different distributions offline; Use the constructed knowledge graph to traverse the collected historical data set q and obtain the graph path set corresponding to q; Build Prompt to process graph path data, generate prediction results, management opinions, and answer results, and modify and adjust the answer result data based on business rules; Format the model input for the collected QA question-answer pairs; For QA question-answer pairs, the LLM model is used for supervised learning, minimizing the cross entropy loss by maximizing the likelihood function of the target sequence; By optimizing the training samples and parameters, the fine-tuned LLM model with the best effect on the validation set was obtained.

9. The crop knowledge graph question answering method based on large model and data assimilation according to claim 8 is characterized in that: The steps to obtain the final question-answering result by constructing multi-layer prompts for multi-dimensional output include: Construct multiple prompts for the problem in parallel, including the problem resolution prompt, the original construction prompt, and the suggestion prompt; Then add a result integration prompt in the outermost layer; Finally, the model results of the first layer are output and merged to obtain the final question-answering result.

10. A crop knowledge graph question-answering system for implementing the crop knowledge graph question-answering method based on a large model and data assimilation as described in any one of claims 1 to 9, characterized in that: Includes the following modules: A data acquisition module is used to acquire a multimodal dataset of crops, where the multimodal dataset includes text data and image data; The assimilation network construction module is used to fuse feature vectors of different modalities based on the graph attention network (GAT) model, generate a unified feature vector, and build a multimodal data assimilation network. The knowledge graph construction and update module is used to generate the initial knowledge graph based on the dynamic graph neural network on the basis of the assimilation network. When real-time data arrives, the initial knowledge graph is updated using a dynamic incremental graph update algorithm based on the gated recurrent unit (GRU) model. The candidate set generation module is used to extract and understand user questions based on the large language model (LLM), process the LLM output, and generate a graph-structured query template. Obtain the Prompt query template constructed by LLM, fuse the Prompt query template with the knowledge graph, use the reinforcement learning path optimization method to obtain the optimal information query, integrate the graph query results obtained by each Prompt query template, and generate a graph path query candidate set; The question-answering result generation module is used to generate question-answering results based on the optimized and adjusted LLM fusion graph query results. The attention mechanism is used to fuse the query question and the graph path query candidate set, calculate the answer confidence, and obtain the answer candidate set. By constructing a multi-layer prompt for multi-dimensional output, the final question-answering result is obtained.

Images