Question answering method based on graph neural network retriever to enhance large language model generation
By enhancing the large language model by a searcher based on graph neural network, the efficiency problems caused by the superposition of inference time and increase of graph scale in the intelligent question-and-answer system are solved, and efficient and accurate question-and-answer generation is achieved.
Patent Information
- Application Number
- CN202411128115.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-16
AI Technical Summary
The existing intelligent question-and-answer system calls large language models multiple times during the process of retrieving subgraphs, resulting in the superposition of inference time. When the scale of the knowledge graph becomes larger, the time consumed by the graph theory algorithm increases, affecting the efficiency of the question-and-answer process.
The searcher based on graph neural network is used to enhance the question-and-answer method of generating large language models. By building data sets and graph neural network models, the graph neural network searcher is trained, and the knowledge graph is used for structured storage and node association, and a path sub-graph collection is generated. And input it into a large language model for inference.
It improves the inference generation ability of large language models, reduces time costs, improves the accuracy and efficiency of reply content, solves the "illusion" problem of large language models, and improves the quality of reply content.
Smart Images

Figure CN118656475B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing, and in particular to a question-answering method based on a graph neural network retriever to enhance the generation of a large language model. Background Art
[0002] The birth of large language models (such as GPT, ChatGLM, etc.) has shown excellent performance in various NLP tasks (such as intelligent question answering, machine translation, etc.), but at the same time there are also a series of drawbacks and have been criticized, such as lack of flexibility, hallucinations, poor interpretability, etc. Supplementing with additional knowledge when reasoning with large language models can effectively solve these weaknesses and limitations. Retrieval-augmented Generation (RAG) is a widely used method.
[0003] The knowledge graph stores a large number of facts in the form of triples. It is a structured and deterministic way of representing knowledge, with advantages such as specific domain knowledge, accuracy, and interpretability. In the field of intelligent question answering, there is an endless stream of work on using knowledge graph retrieval to enhance the generation of large language models. However, there are two major defects in the existing retrieval methods: first, in the process of retrieving subgraphs, many works tend to let the large language model decide the direction of the next step. Multiple calls to the large language model will inevitably result in the superposition of reasoning time, which increases the time cost; there are also works that use graph theory algorithms as retrieval rules, but when the scale of the knowledge graph becomes larger, the time consumed by the graph theory algorithm will also increase accordingly, without considering the impact of time complexity on the entire question-answering process.
[0004] Therefore, a question-answering method based on a graph neural network retriever to enhance the generation of a large language model is proposed to enhance the reasoning generation capability of the large language model, reduce time cost, and improve performance and efficiency. Summary of the invention
[0005] The purpose of this application is to propose a question-answering method based on a graph neural network retriever to enhance the generation of a large language model in response to the above-mentioned technical problems.
[0006] In a first aspect, the present invention provides a question answering method based on a graph neural network retriever to enhance the generation of a large language model, comprising the following steps:
[0007] Build a data set and a graph neural network model, use the data set to train the graph neural network model, and obtain a graph neural network retriever;
[0008] Obtain the knowledge graph, store it in a structured manner, obtain an entity set and a relationship list, and abstract all entities in the entity set and all relationships in the relationship list to obtain a directed acyclic graph;
[0009] Get the input question and its context, extract keywords related to the input question from the context, map the keywords to an entity set, and obtain a node set;
[0010] According to the node set, the directed acyclic graph is associated with each other to obtain an associated directed acyclic graph, the associated directed acyclic graph is input into the graph neural network retriever to obtain a path subgraph set, and the path subgraph set is restored to the knowledge graph to obtain a knowledge subgraph set;
[0011] The knowledge subgraph set is input into the large language model for reasoning to generate the response content corresponding to the input question.
[0012] Preferably, the process of constructing the data set specifically includes:
[0013] Create an initially empty directed graph with given parameters and ,in represents the number of nodes in the directed graph, Represents the probability of an edge between two nodes in a directed graph. Nodes in the directed graph are randomly selected for connection, and the probability of an edge between two nodes is calculated based on the probability of an edge between two nodes. Add edges to the directed graph to get a directed acyclic graph;
[0014] Generate the corresponding data object according to the directed acyclic graph, as shown in the following formula:
[0015] ;
[0016] Among them, x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph. In the edge index vector, the corresponding position values of the first dimension and the second dimension can be combined into an edge. The edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension.
[0017] Randomly select several nodes in a directed acyclic graph to form a node set. For the nodes in the node set, explore the shortest path between two nodes, perform "two-hop" expansion on each node on the shortest path, and extend the expansion path based on the shortest path. Both the shortest path and the expansion path include several edges. All shortest paths and expansion paths constitute a path subgraph set.
[0018] The feature vectors of the nodes of the directed acyclic graph are assigned values according to the node set, the feature vectors of the nodes in the node set are assigned values of 1, and the feature vectors of the remaining nodes in the directed acyclic graph except the nodes in the node set are assigned values of 0, as shown in the following formula:
[0019] ;
[0020] in, represents the feature vector of the i-th node in the directed acyclic graph, nodes represents the node set, Indicates that the i-th node belongs to the node set;
[0021] Construct a label vector of edge indices for a directed acyclic graph , y is a dimensional vector, assign the labels of the edge indexes of the directed acyclic graph according to the path subgraph set, assign the edge index labels corresponding to the edges in the path subgraph set to 1, and assign the edge index labels of the directed acyclic graph other than the edge index labels corresponding to the edges in the path subgraph set to 0, as shown in the following formula:
[0022] ;
[0023] in, Representation and edge index The corresponding label, Graph Algorithm, represents the path subgraph set, Represents edge index The edge edge represented belongs to the path subgraph set;
[0024] Repeat the above steps to generate several data objects corresponding to the directed acyclic graph;
[0025] According to the data objects corresponding to several directed acyclic graphs and their corresponding A data set is constructed and divided into training set, validation set and test set.
[0026] Preferably, a knowledge graph is obtained, and the knowledge graph is structured and stored to obtain an entity set and a relationship list, and all entities in the entity set and all relationships in the relationship list are abstracted to obtain a directed acyclic graph, specifically including:
[0027] Get the knowledge graph, number all entities in the knowledge graph, and store them in the entity collection;
[0028] Create a relationship list and a relationship dictionary. The relationship list is used to store the edge information of the knowledge graph. In the relationship list, Indicates the number The entity and number are There is an edge between entities, and the edge points to the number The entity pointer number is The relationship dictionary is used to store the specific meaning of the edge. In the relationship dictionary, Represents edge The specific meaning is r;
[0029] The entities in the entity set are abstracted as nodes of a directed acyclic graph, and the edges in the relationship list are abstracted as edges of a directed acyclic graph. The resulting directed acyclic graph has the following corresponding data object: , where x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph. In the edge index vector, the corresponding position values of the first dimension and the second dimension can be combined into an edge. The edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension.
[0030] Preferably, the input question and its context are obtained, keywords related to the input question are extracted from the context, and the keywords are mapped to an entity set to obtain a node set, which specifically includes:
[0031] Get an input question and a set of examples ,in, represents the nth question in the example set, express Corresponding keyword set, question and the corresponding keyword set Constitute a pair of examples, N represents the number of examples in the set of examples, and the entity set and the example set As the context of the input question, the large language model is prompted to extract a set of keywords related to the input question from the entity set through context learning;
[0032] The cosine similarity between the keywords in the keyword set and the entities in the entity set is calculated. The entity with the largest cosine similarity is the entity mapped by the keyword in the entity set. All entities mapped by the keyword in the entity set constitute a node set.
[0033] Preferably, the directed acyclic graph is associated with nodes according to the node set to obtain an associated directed acyclic graph, and the associated directed acyclic graph is input into a graph neural network retriever to obtain a path subgraph set, which specifically includes:
[0034] Obtain the serial number of the entity in the node set, assign a value of 1 to the characteristic vector of the node corresponding to the serial number of the entity in the node set in the data object of the directed acyclic graph, and obtain the data object corresponding to the associated directed acyclic graph;
[0035] Input the data object corresponding to the associated directed acyclic graph into the graph neural network retriever, calculate the probability of the existence of an edge between two nodes in the associated directed acyclic graph, and obtain the probability value of the edge;
[0036] Set a threshold, extract all edges whose probability values are greater than the threshold and their edge indexes, and obtain the edge set and corresponding edge index set used to generate the path subgraph;
[0037] Select two edge indexes in the edge index set. If the value of the first dimension of one of the edge indexes is the same as the value of the second dimension of the other edge index, connect the edges corresponding to the two edge indexes end to end, traverse the edge index set, and obtain p path subgraphs. The p path subgraphs constitute the path subgraph set.
[0038] Preferably, the path subgraph set is restored to the knowledge graph to obtain a knowledge subgraph set, which specifically includes:
[0039] The edges in the path subgraph set and the nodes connected by the edges are mapped to the relationship list and the entity set, and the specific meaning of each edge in the path subgraph is indexed according to the relationship dictionary to obtain the knowledge subgraph set.
[0040] In a second aspect, the present invention provides a question-answering device based on a graph neural network retriever to enhance large language model generation, comprising:
[0041] The retriever training module is configured to construct a data set and a graph neural network model, and train the graph neural network model using the data set to obtain a graph neural network retriever;
[0042] The knowledge graph processing module is configured to obtain the knowledge graph, store the knowledge graph in a structured manner, obtain an entity set and a relationship list, and abstract all entities in the entity set and all relationships in the relationship list to obtain a directed acyclic graph;
[0043] The question content identification module is configured to obtain an input question and its context, extract keywords related to the input question from the context, map the keywords to an entity set, and obtain a node set;
[0044] The knowledge subgraph retrieval module is configured to associate nodes of the directed acyclic graph according to the node set to obtain an associated directed acyclic graph, input the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, and restore the path subgraph set to the knowledge graph to obtain a knowledge subgraph set;
[0045] The reply content generation module is configured to input the knowledge subgraph set into the large language model for reasoning and generate reply content corresponding to the input question.
[0046] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.
[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.
[0048] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.
[0049] Compared with the prior art, the present invention has the following beneficial effects:
[0050] (1) The question-answering method based on graph neural network retriever to enhance the generation of large language model proposed in the present invention retrieves background knowledge related to the input question through the graph neural network retriever, improves the retrieval efficiency, enhances the reasoning generation capability of the large language model, improves the accuracy of the reply content generated by the large language model, and achieves performance and efficiency improvements.
[0051] (2) The question-answering method based on graph neural network retriever to enhance the generation of large language model proposed in the present invention retrieves background knowledge related to the input question as supplementary input of the large language model through the graph neural network retriever, guides the large language model to reason on the retrieved knowledge and generate reply content, solves the "hallucination" problem of the large language model, and improves the quality of the reply content.
[0052] (3) In the question-answering method based on the graph neural network retriever to enhance the generation of a large language model proposed in the present invention, the graph neural network-based retriever has powerful parallel computing capabilities and can quickly retrieve the path subgraph, thereby restoring the knowledge subgraph based on the knowledge graph, thereby greatly improving its efficiency and reducing the time cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1 The overall flow chart of the question-answering method based on graph neural network retriever enhanced large language model generation in an embodiment of the present application;
[0055] Figure 2 A detailed flow chart of a question-answering method based on a graph neural network retriever to enhance large language model generation according to an embodiment of the present application;
[0056] Figure 3 A training loss curve diagram of the graph neural network model of an embodiment of the present application;
[0057] Figure 4 A verification loss curve diagram of the graph neural network model of an embodiment of the present application;
[0058] Figure 5 This is a comparison chart of the accuracy of large language model generation based on GAT-RAG, None, and LKG-RAG in the embodiments of the present application;
[0059] Figure 6 This is a time cost comparison chart for generating a large language model based on GAT-RAG, None, and LKG-RAG in an embodiment of the present application;
[0060] Figure 7 A schematic diagram of a question-answering device based on a graph neural network retriever to enhance large language model generation according to an embodiment of the present application;
[0061] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] Figure 1 A question-answering method based on graph neural network retriever enhanced large language model generation provided by an embodiment of the present application is shown, comprising the following steps:
[0064] S1, build a data set and a graph neural network model, use the data set to train the graph neural network model, and obtain a graph neural network retriever.
[0065] In a specific embodiment, the process of constructing a data set specifically includes:
[0066] Create an initially empty directed graph with given parameters and ,in represents the number of nodes in the directed graph, Represents the probability of an edge between two nodes in a directed graph. Nodes in the directed graph are randomly selected for connection, and the probability of an edge between two nodes is calculated based on the probability of an edge between two nodes. Add edges to the directed graph to get a directed acyclic graph;
[0067] Generate the corresponding data object according to the directed acyclic graph, as shown in the following formula:
[0068] ;
[0069] Among them, x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph. In the edge index vector, the corresponding position values of the first dimension and the second dimension can be combined into an edge. The edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension.
[0070] Randomly select several nodes in a directed acyclic graph to form a node set. For the nodes in the node set, explore the shortest path between two nodes, perform "two-hop" expansion on each node on the shortest path, and extend the expansion path based on the shortest path. Both the shortest path and the expansion path include several edges. All shortest paths and expansion paths constitute a path subgraph set.
[0071] The feature vectors of the nodes of the directed acyclic graph are assigned values according to the node set, the feature vectors of the nodes in the node set are assigned values of 1, and the feature vectors of the remaining nodes in the directed acyclic graph except the nodes in the node set are assigned values of 0, as shown in the following formula:
[0072] ;
[0073] in, represents the feature vector of the i-th node in the directed acyclic graph, nodes represents the node set, Indicates that the i-th node belongs to the node set;
[0074] Construct a label vector of edge indices for a directed acyclic graph , y is a dimensional vector, assign the labels of the edge indexes of the directed acyclic graph according to the path subgraph set, assign the edge index labels corresponding to the edges in the path subgraph set to 1, and assign the edge index labels of the directed acyclic graph other than the edge index labels corresponding to the edges in the path subgraph set to 0, as shown in the following formula:
[0075] ;
[0076] in, Representation and edge index The corresponding label, Graph Algorithm, represents the path subgraph set, Represents edge index The edge edge represented belongs to the path subgraph set;
[0077] Repeat the above steps to generate several data objects corresponding to the directed acyclic graph;
[0078] According to the data objects corresponding to several directed acyclic graphs and their corresponding A data set is constructed and divided into training set, validation set and test set.
[0079] Specifically, step S1 is the graph neural network retriever training step, in which the function is designed , used to generate a random directed acyclic graph (DAG), where the function parameters Indicates the number of nodes in a directed graph, parameter Represents the probability of generating an edge between two nodes of a directed graph, randomly selects nodes in the directed graph to connect, ensures that the generated directed graph is connected, and Add edges to the directed graph, ensuring that the added edges do not form a cycle, and finally obtain a directed acyclic graph and generate its corresponding data object , at this time, the eigenvectors of all nodes in the directed acyclic graph are assigned a value of 0.
[0080] Design Function To construct a graph theory algorithm, the parameter G of the function represents a directed acyclic graph randomly generated by the function, and nodes represents a node set consisting of several nodes randomly selected from the directed acyclic graph. The specific graph theory algorithm is as follows: For the nodes in the node set, explore the shortest path between two nodes, perform a "two-hop" expansion on each node on the shortest path, and extend the expansion path based on the shortest path. All shortest paths and expansion paths constitute a path subgraph set. Both the shortest path and the expansion path include several edges. Therefore, the label vector of the edge index of the directed acyclic graph is constructed , assign the edge index labels corresponding to the edges in the path subgraph set to 1, and assign the edge index labels in the directed acyclic graph other than the edge index labels corresponding to the edges in the path subgraph set to 0. At the same time, assign the feature vectors of the nodes in the node set to 1.
[0081] Specifically, Graph Neural Network (GNN) refers to a general term for algorithms that use neural networks to learn graph structured data, extract and discover features and patterns in graph structured data, and meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation. Graph Attention Network (GAT) is a new type of neural network architecture based on graph structured data. It introduces the attention mechanism into the graph neural network based on the spatial domain to optimize some defects of the graph convolutional neural network. In an embodiment of the present application, the graph neural network model uses the graph attention network as the basic training architecture. Compared with the graph convolutional network (GCN), it gets rid of the limitations of the graph structure on downstream tasks, is more suitable for inductive tasks, and has better generalization. First, define the graph attention network GAT and initialize it, create a module list containing the GATConv layer, and the number of network layers is In the first layer of the network, the input training set sample features are mapped to the hidden layer, and the dimension of the input features is , the hidden layer feature dimension is ; The middle layer maps the hidden layer features to the next hidden layer; The last layer maps the hidden layer features to the output features, and the dimension of the output features is At the same time, in each layer, the number of attention heads in each layer is set to heads, and the Dropout rate is dropout. In the forward propagation function of the graph neural network model, its input is the feature vector "x" containing the node and the edge index " "In the forward propagation of the graph neural network model, we first traverse all convolutional layers except the last layer of the network, apply convolution, ReLU activation function and Dropout, and only apply convolution to the last layer of the network. The output of the forward propagation function is the output feature "x" of the last layer of the network.
[0082] The data object corresponding to the directed acyclic graph is input into the graph neural network model, the node features of the directed acyclic graph are obtained through the forward propagation function, and the edge score is calculated using the dot product of the connected node features. and its eigenvector , V represents the set of all nodes in the directed acyclic graph, and the node features are calculated through the forward propagation function of the graph neural network model , the calculation formula is as follows:
[0083] ;
[0084] in, Represents the forward propagation function of the graph neural network model;
[0085] For an edge in a directed acyclic graph , E represents the set of all edges of the directed acyclic graph, and the edge prediction function is used to calculate the edge The edge score , the calculation formula is as follows:
[0086] ;
[0087] in, represents the jth node in a directed acyclic graph, and Respectively represent the edge The two nodes connected, and Respectively represent nodes and The node features of the directed acyclic graph can be used to calculate the scores of all edges in the directed acyclic graph through the forward propagation function and the edge prediction function.
[0088] In the training phase of the graph neural network model, the model and optimizer are first initialized, and the optimizer is defined as Adam, and the learning rate is ; Define the loss function is the binary cross entropy loss BCEWithLogitsLoss. The loss function calculation process is as follows: for each edge , the edge score obtained by the edge prediction function is ,side The edge index labels are , use the sigmoid activation function to Score Map to the range [0,1] to get the probability value , the calculation formula is as follows:
[0089] ;
[0090] in, represents the exponential function with the real number e as the base;
[0091] Calculated based on binary cross entropy loss and edge The edge index label The loss between , the calculation formula is as follows:
[0092] ;
[0093] in, Represents the logarithmic function with the real number e as the base.
[0094] Then perform cyclic iterative training, define the specified number of iterations as epochs and the input batch training data size as batch_size.
[0095] For each batch of training data, the optimizer gradient is first cleared and the edge score is calculated through the edge prediction function. Then the loss is calculated and accumulated, and then optimized through back propagation. At the same time, in the verification stage, each batch of verification data in the verification set is evaluated to observe the effect of the model on the verification set. In addition, during the training process, in order to prevent the model from overfitting during the training process, an early stopping strategy is added. By continuously monitoring the error of the verification set, when the error of the training set no longer decreases, but the loss of the verification set has not reached the next minimum value, the early stopping strategy will terminate the model training in advance and use the model trained in the epoch with the minimum value of the verification set as the optimal model.
[0096] In the embodiment of the present application, the training parameters of the graph neural network model are shown in Table 1, and the training loss and validation loss during the training process are respectively as follows: Figure 3 and Figure 4 shown.
[0097] Table 1 Training parameters
[0098]
[0099] After the graph neural network model is trained, a graph neural network retriever (GAT-RAG) is obtained. It only needs to input a directed acyclic graph and several points on the graph, and the graph neural network retriever can predict the related paths of these points in the directed acyclic graph, that is, the path subgraph, and set a threshold. When the probability value of the edge in the aforementioned related path is greater than the threshold, the edge is correctly predicted. Table 2 shows the performance of the graph neural network model on the test set under different thresholds. The three indicators of accuracy (Accuracy), recall (Recall) and F1 score (F1) are used to evaluate the model performance. Therefore, in the embodiment of the present application, the threshold is set to 0.8.
[0100] Table 2 Performance of the neural network model on the test set under different thresholds
[0101]
[0102] In the embodiments of the present application, the graph neural network model can learn the graph theory algorithm from the data set, and use the graph neural network retriever to replace the graph theory algorithm to realize the retrieval function. The question-answering method based on the graph neural network retriever to enhance the generation of a large language model proposed in the present application can automatically generate replies to input questions under limited hardware resources. By combining it with the graph neural network retriever, the quality of the reply content can be effectively improved. At the same time, compared with the traditional graph theory algorithm, the design of the graph neural network retriever can speed up the efficiency of retrieval and save time costs.
[0103] S2, obtain the knowledge graph, store the knowledge graph in a structured manner, obtain an entity set and a relationship list, and abstract all entities in the entity set and all relationships in the relationship list to obtain a directed acyclic graph.
[0104] In a specific embodiment, a knowledge graph is obtained, the knowledge graph is structured and stored, an entity set and a relationship list are obtained, and all entities in the entity set and all relationships in the relationship list are abstracted to obtain a directed acyclic graph, which specifically includes:
[0105] Get the knowledge graph, number all entities in the knowledge graph, and store them in the entity collection;
[0106] Create a relationship list and a relationship dictionary. The relationship list is used to store the edge information of the knowledge graph. In the relationship list, Indicates the number The entity and number are There is an edge between entities, and the edge points to the number The entity pointer number is The relationship dictionary is used to store the specific meaning of the edge. In the relationship dictionary, Represents edge The specific meaning is r;
[0107] The entities in the entity set are abstracted as nodes of a directed acyclic graph, and the edges in the relationship list are abstracted as edges of a directed acyclic graph. The resulting directed acyclic graph has the following corresponding data object: , where x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph. In the edge index vector, the corresponding position values of the first dimension and the second dimension can be combined into an edge. The edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension.
[0108] Specifically, step S2 is a knowledge graph processing step. In this step, three functions are designed for query convenience, namely: The function can query the entity by number, The function can query the corresponding number according to the entity. The function queries the relationship between two entities based on their IDs.
[0109] S3, obtains the input question and its context, extracts keywords related to the input question in the context, maps the keywords to an entity set, and obtains a node set.
[0110] In a specific embodiment, an input question and its context are obtained, keywords related to the input question are extracted from the context, and the keywords are mapped to an entity set to obtain a node set, which specifically includes:
[0111] Get an input question and a set of examples ,in, represents the nth question in the example set, express Corresponding keyword set, question and the corresponding keyword set Constitute a pair of examples, N represents the number of examples in the set of examples, and the entity set and the example set As the context of the input question, the large language model is prompted to extract a set of keywords related to the input question from the entity set through context learning;
[0112] The cosine similarity between the keywords in the keyword set and the entities in the entity set is calculated. The entity with the largest cosine similarity is the entity mapped by the keyword in the entity set. All entities mapped by the keyword in the entity set constitute a node set.
[0113] Specifically, the BERT encoder is a pre-trained natural language processing model that uses the Transformer architecture. The main feature of BERT is that it understands contextual information in a bidirectional manner. Compared with traditional unidirectional models, it can better capture the relationship and context between words. Step S3 is the question content identification step. In this step, the BERT encoder is used to encode all keywords in the entity set and keyword set into dense embedding vectors respectively, and the encoded entity set is obtained. and the encoded keyword set , where S represents the encoded entity set The number of entities en in the encoding, T represents the number of keywords k in the encoded keyword set K;
[0114] For the tth keyword in the encoded keyword set K ,calculate With the encoded entity set The sth entity in The cosine similarity is calculated as follows:
[0115] ;
[0116] ;
[0117] ;
[0118] ;
[0119] Where D is the dimension of the dense embedding vector, and They are vectors and The d-th element of represents the norm;
[0120] The keywords Mapped to the encoded entity collection , get the keyword Most similar entities , The expression is as follows:
[0121] ;
[0122] in, Represents the sth entity and When the cosine similarity of is the maximum value, return the sth entity and get ,all Make up a collection of nodes.
[0123] S4, associate the nodes of the directed acyclic graph according to the node set to obtain the associated directed acyclic graph, input the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, restore the path subgraph set to the knowledge graph to obtain a knowledge subgraph set.
[0124] In a specific embodiment, the directed acyclic graph is associated with nodes according to the node set to obtain an associated directed acyclic graph, and the associated directed acyclic graph is input into a graph neural network retriever to obtain a path subgraph set, which specifically includes:
[0125] Obtain the serial number of the entity in the node set, assign a value of 1 to the characteristic vector of the node corresponding to the serial number of the entity in the node set in the data object of the directed acyclic graph, and obtain the data object corresponding to the associated directed acyclic graph;
[0126] Input the data object corresponding to the associated directed acyclic graph into the graph neural network retriever, calculate the probability of the existence of an edge between two nodes in the associated directed acyclic graph, and obtain the probability value of the edge;
[0127] Set a threshold, extract all edges whose probability values are greater than the threshold and their edge indexes, and obtain the edge set and corresponding edge index set used to generate the path subgraph;
[0128] Select two edge indexes in the edge index set. If the value of the first dimension of one of the edge indexes is the same as the value of the second dimension of the other edge index, connect the edges corresponding to the two edge indexes end to end, traverse the edge index set, and obtain p path subgraphs. The p path subgraphs constitute the path subgraph set.
[0129] Specifically, step S4 is a knowledge subgraph retrieval step, in which the function Query the IDs of all entities in a node collection , there are T entities in the node set, among which, It represents the number of the tth node in the node set. ; Assign the feature vector of the node corresponding to the number ids in the data object Data corresponding to the directed acyclic graph to 1, and obtain the data object corresponding to the associated directed acyclic graph; input the data object corresponding to the associated directed acyclic graph into the graph neural network retriever, calculate the probability of the existence of an edge between two nodes in the associated directed acyclic graph, and obtain the probability value of the edge; set a threshold, extract all edges whose probability values are greater than the threshold and their edge indexes, and obtain the edge set and corresponding edge index set used to generate the path subgraph ,in, Represents a set of edge indices The edge index of the u-th edge in , , U represents the number of predicted edges; if two edge indices are selected from the edge index set , , edge index The value of the first dimension and the edge index The values of the second dimension of are different, then reselect two edge indices. If two edge indices are selected from the edge index set , , edge index The value of the first dimension and the edge index If the values of the second dimension of are the same, the edges corresponding to the two edge indices are connected end to end, and the edge index set is traversed to obtain p path subgraphs. The p path subgraphs constitute the path subgraph set. ,in, express The p-th path subgraph in , P represents the number of path subgraphs.
[0130] In a specific embodiment, the path subgraph set is restored to the knowledge graph to obtain the knowledge subgraph set, which specifically includes:
[0131] The edges in the path subgraph set and the nodes connected by the edges are mapped to the relationship list and the entity set, and the specific meaning of each edge in the path subgraph is indexed according to the relationship dictionary to obtain the knowledge subgraph set.
[0132] Specifically, according to the specific meaning of each edge in the relational dictionary index path subgraph, the knowledge subgraph set is obtained. ,in, express The knowledge subgraph corresponding to the p-th path subgraph in .
[0133] S5, input the knowledge subgraph set into the large language model for reasoning, and generate the response content corresponding to the input question.
[0134] Specifically, the LangChain framework is a powerful framework designed to help developers build end-to-end applications using language models. It provides a set of tools, components, and interfaces that simplify the process of creating applications powered by large language models (LLMs) and chat models; LangChain can easily manage interactions with language models, link multiple components together, and integrate additional resources such as APIs and databases. The Prompt-based approach, also called PromptEngineering, uses the ICL capabilities of the model to proxy the fine-tuning training of traditional Bert-like models, using more efficient Prompts to guide LLMs to produce the desired results without changing the weights of the model. PromptTemplates is a reproducible way to generate Prompts, which contains a text string that can accept a set of parameters from the end user and generate a Prompt. Prompt Templates can contain instructions, a small number of examples, and a question to the language model. Prompt Templates technology is usually used to guide language models to generate higher quality text to better complete tasks. Step S5 is the reply content generation step. In this step, the LangChain framework is used to build a PromptTemplate to prompt the large language model based on the knowledge subgraph set retrieved. Perform inference and generate responses corresponding to the input questions.
[0135] In the embodiment of the present application, the large language model in the field of trademark law is experimentally verified as follows: in addition to the trademark legal knowledge graph obtained by step S2, the legal examination questions and exercises in the field of trademarks on the Internet in recent years are further collected to construct an experimental data set, including 185 single-choice questions, 108 multiple-choice questions and 96 judgment questions. In addition to the question-answering method based on the graph neural network retriever to enhance the generation of a large language model proposed in the embodiment of the present application, two other methods are selected: 1. None: This method means that no additional retrieval enhancement generation (RAG) module is added, and the large language model is directly used for reasoning; 2. Legal knowledge graph retrieval enhancement method (abbreviated as LKG-RAG): This method adds an additional retrieval enhancement generation (RAG) module to the large language model. The retrieval enhancement generation method uses traditional graph theory algorithms and large language models to guide decision-making. Specifically, LKG-RAG first extracts keywords from the question and maps them to the entity set of the knowledge graph, then uses the graph theory algorithm to find the shortest path between entities in the entity set to construct a knowledge subgraph, and finally uses the large language model guidance strategy to expand the scope of the knowledge subgraph. In terms of evaluation indicators, the accuracy and time cost are used to evaluate the above three methods. Based on GAT-RAG, None, and LKG-RAG, the large language model is generated to obtain the answers corresponding to each question in the experimental data set. Table 3 shows the accuracy and time cost of the answers generated by the large language model based on GAT-RAG, None, and LKG-RAG. Figure 5 The following figure shows the accuracy comparison of large language model generation based on GAT-RAG, None, and LKG-RAG. Figure 6 The figure shows the time cost comparison chart for generating large language models based on GAT-RAG, None, and LKG-RAG, where Method represents the method used, Multiple-choice represents single-choice questions, Double-choice represents multiple-choice questions, True / False represents judgment questions, Acc represents accuracy, and Time represents time cost in seconds.
[0136] Table 3 Answer accuracy and time cost of large language model generation based on GAT-RAG, None, and LKG-RAG
[0137]
[0138] The above steps S1-S5 do not necessarily represent the order between the steps, but are represented by step symbols, and the order between the steps can be adjusted.
[0139] Further references Figure 7As an implementation of the methods shown in the above figures, the present application provides an embodiment of a question-answering device based on a graph neural network retriever to enhance the generation of a large language model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0140] The embodiment of the present application provides a question-answering device based on a graph neural network retriever to enhance the generation of a large language model, including:
[0141] The retriever training module 1 is configured to construct a data set and a graph neural network model, and train the graph neural network model using the data set to obtain a graph neural network retriever;
[0142] The knowledge graph processing module 2 is configured to obtain the knowledge graph, store the knowledge graph in a structured manner, obtain an entity set and a relationship list, and abstract all entities in the entity set and all relationships in the relationship list to obtain a directed acyclic graph;
[0143] The question content identification module 3 is configured to obtain the input question and its context, extract keywords related to the input question from the context, map the keywords to an entity set, and obtain a node set;
[0144] The knowledge subgraph retrieval module 4 is configured to associate nodes of the directed acyclic graph according to the node set to obtain an associated directed acyclic graph, input the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, and restore the path subgraph set to the knowledge graph to obtain a knowledge subgraph set;
[0145] The reply content generation module 5 is configured to input the knowledge subgraph set into the large language model for reasoning, and generate reply content corresponding to the input question.
[0146] Figure 8 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 8 As shown, the electronic device of this embodiment includes: a processor 801 and a memory 802; wherein the memory 802 is used to store computer-executable instructions; the processor 801 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.
[0147] Optionally, the memory 802 may be independent or integrated with the processor 801 .
[0148] When the memory 802 is independently provided, the electronic device further includes a bus 803 for connecting the memory 802 and the processor 801 .
[0149] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 801 executes the computer execution instructions, the above method is implemented.
[0150] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 801, the above method is implemented.
[0151] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0152] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.
[0153] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0154] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 801 to perform some steps of the methods of various embodiments of the present application.
[0155] It should be understood that the processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 801 may be any conventional processor 801, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 801, or may be executed by a combination of hardware and software modules in the processor 801.
[0156] The memory 802 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.
[0157] The bus 803 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 803 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 803 in the drawings of the present application is not limited to only one bus 803 or one type of bus 803.
[0158] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0159] An exemplary storage medium is coupled to the processor 801, so that the processor 801 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 801. The processor 801 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 801 and the storage medium can also exist as discrete components in an electronic device or a main control device.
[0160] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A question-answering method based on graph neural network retriever to enhance large language model generation, characterized in that: The following steps are involved: Constructing a data set and a graph neural network model, and using the data set to train the graph neural network model to obtain a graph neural network retriever; Acquire a knowledge graph, perform structured storage on the knowledge graph, obtain an entity set and a relationship list, and abstract all entities in the entity set and all relationships in the relationship list to obtain a directed acyclic graph, and the acquisition of the knowledge graph, perform structured storage on the knowledge graph, obtain an entity set and a relationship list, and abstract all relationships in the entity set and the relationship list to obtain a directed acyclic graph, specifically including: Obtain a knowledge graph, number all entities in the knowledge graph, and store them in an entity set; Create a relationship list and a relationship dictionary, wherein the relationship list is used to store the edge information of the knowledge graph. In the relationship list, Indicates the number The entity and number are There is an edge between entities, and the edge points to the number The entity pointer number is The relationship dictionary is used to store the specific meaning of the edge. In the relationship dictionary, Represents edge The specific meaning is r; The entities in the entity set are abstracted as nodes of the directed acyclic graph, and the edges in the relationship list are abstracted as edges of the directed acyclic graph, to obtain a directed acyclic graph, whose corresponding data object is , where x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph, where the corresponding position values of the first dimension and the second dimension in the edge index vector can be combined into an edge, and the edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension; Obtaining an input question and its context, extracting keywords related to the input question from the context, mapping the keywords to the entity set, and obtaining a node set; Associating nodes of the directed acyclic graph according to the node set to obtain an associated directed acyclic graph, inputting the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, restoring the path subgraph set to the knowledge graph to obtain a knowledge subgraph set, associating nodes of the directed acyclic graph according to the node set to obtain an associated directed acyclic graph, and inputting the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, specifically includes: Obtaining the serial number of the entity in the node set, assigning a characteristic vector of a node in the data object of the directed acyclic graph corresponding to the serial number of the entity in the node set to 1, and obtaining the data object corresponding to the associated directed acyclic graph; Inputting the data object corresponding to the associated directed acyclic graph into the graph neural network retriever, calculating the probability of an edge existing between two nodes in the associated directed acyclic graph, and obtaining a probability value of the edge; A threshold is set, and edges and edge indexes corresponding to all edges whose probability values are greater than the threshold are extracted to obtain an edge set and a corresponding edge index set for generating a path subgraph; Select two edge indexes from the edge index set, if the value of the first dimension of one of the edge indexes is the same as the value of the second dimension of the other edge index, then connect the edges corresponding to the two edge indexes end to end, traverse the edge index set, and obtain p path subgraphs, the p path subgraphs constitute a path subgraph set; The restoring the path subgraph set to the knowledge graph to obtain the knowledge subgraph set specifically includes: Mapping the edges in the path subgraph set and the nodes connected by the edges to the relationship list and entity set, and indexing the specific meaning of each edge in the path subgraph according to the relationship dictionary to obtain a knowledge subgraph set; The knowledge subgraph set is input into a large language model for reasoning to generate a response content corresponding to the input question.
2. The question answering method based on graph neural network retriever enhanced large language model generation according to claim 1, characterized in that: The process of constructing the dataset specifically includes: Create an initially empty directed graph with given parameters and ,in represents the number of nodes in the directed graph, represents the probability of generating an edge between two nodes of a directed graph, randomly selecting nodes in the directed graph to connect, and according to Adding edges to the directed graph to obtain a directed acyclic graph; The corresponding data object is generated according to the directed acyclic graph, as shown in the following formula: ; Among them, x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph, where the corresponding position values of the first dimension and the second dimension in the edge index vector can be combined into an edge, and the edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension; A number of nodes are randomly selected in the directed acyclic graph to form a node set. For the nodes in the node set, the shortest path between any two nodes is explored, and each node on the shortest path is expanded by two hops. An expanded path is obtained by extending the shortest path, wherein both the shortest path and the expanded path include a number of edges, and all the shortest paths and the expanded paths constitute a path subgraph set. The feature vectors of the nodes of the directed acyclic graph are assigned values according to the node set, the feature vectors of the nodes in the node set are assigned values of 1, and the feature vectors of the remaining nodes in the directed acyclic graph except the nodes in the node set are assigned values of 0, as shown in the following formula: ; in, Represents the directed acyclic graph The feature vector of each node, Represents a collection of nodes. Indicates Nodes belong to a node set; Construct the label vector of the edge index of the directed acyclic graph , y is a dimensional vector, assigning labels of edge indexes of the directed acyclic graph according to the path subgraph set, assigning edge index labels corresponding to edges in the path subgraph set to 1, and assigning other edge index labels in the directed acyclic graph except edge index labels corresponding to edges in the path subgraph set to 0, as shown in the following formula: ; in, Representation and edge index The corresponding label, Graph Algorithm, represents the path subgraph set, Represents edge index The edge edge represented belongs to the path subgraph set; Repeat the above steps to generate several data objects corresponding to the directed acyclic graph; According to the data objects corresponding to several directed acyclic graphs and their corresponding A data set is constructed and divided into a training set, a validation set and a test set.
3. The question answering method based on graph neural network retriever enhanced large language model generation according to claim 1, characterized in that: The step of obtaining an input question and its context, extracting keywords related to the input question from the context, and mapping the keywords to the entity set to obtain a node set specifically includes: Get an input question and a set of examples ,in, represents the nth question in the example set, express Corresponding keyword set, question and the corresponding keyword set Constitute a pair of examples, N represents the number of examples in the set of examples, and the entity set and the example set As the context of the input question, prompt the large language model to extract a set of keywords related to the input question from the entity set through context learning; The cosine similarity between the keywords in the keyword set and the entities in the entity set is calculated, the entity with the largest cosine similarity is the entity mapped by the keyword in the entity set, and all entities mapped by the keyword in the entity set constitute a node set.
4. A question-answering device based on graph neural network retriever to enhance large language model generation, characterized in that: include: A retriever training module is configured to construct a data set and a graph neural network model, and train the graph neural network model using the data set to obtain a graph neural network retriever; The knowledge graph processing module is configured to obtain a knowledge graph, perform structured storage on the knowledge graph, obtain an entity set and a relationship list, and abstract all entities in the entity set and all relationships in the relationship list to obtain a directed acyclic graph. The obtaining of the knowledge graph, performing structured storage on the knowledge graph, obtaining an entity set and a relationship list, and abstracting all relationships in the entity set and the relationship list to obtain a directed acyclic graph specifically includes: Obtain a knowledge graph, number all entities in the knowledge graph, and store them in an entity set; Create a relationship list and a relationship dictionary, wherein the relationship list is used to store the edge information of the knowledge graph. In the relationship list, Indicates the number The entity and number are There is an edge between entities, and the edge points to the number The entity pointer number is The relationship dictionary is used to store the specific meaning of the edge. In the relationship dictionary, Represents edge The specific meaning is r; The entities in the entity set are abstracted as nodes of the directed acyclic graph, and the edges in the relationship list are abstracted as edges of the directed acyclic graph, to obtain a directed acyclic graph, whose corresponding data object is , where x is a dimensional vector, representing the feature vector of the node of the directed acyclic graph, is a dimensional vector, representing the edge index vector of the directed acyclic graph, Represents the number of edges in a directed acyclic graph, where the corresponding position values of the first dimension and the second dimension in the edge index vector can be combined into an edge, and the edge points from the node represented by the value of the first dimension to the node represented by the value of the second dimension; A question content identification module is configured to obtain an input question and its context, extract keywords related to the input question from the context, map the keywords to the entity set, and obtain a node set; The knowledge subgraph retrieval module is configured to associate nodes of the directed acyclic graph according to the node set to obtain an associated directed acyclic graph, input the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, restore the path subgraph set to the knowledge graph to obtain a knowledge subgraph set, associate nodes of the directed acyclic graph according to the node set to obtain an associated directed acyclic graph, input the associated directed acyclic graph into the graph neural network retriever to obtain a path subgraph set, and specifically includes: Obtaining the serial number of the entity in the node set, assigning a characteristic vector of a node in the data object of the directed acyclic graph corresponding to the serial number of the entity in the node set to 1, and obtaining the data object corresponding to the associated directed acyclic graph; Inputting the data object corresponding to the associated directed acyclic graph into the graph neural network retriever, calculating the probability of an edge existing between two nodes in the associated directed acyclic graph, and obtaining a probability value of the edge; A threshold is set, and edges and edge indexes corresponding to all edges whose probability values are greater than the threshold are extracted to obtain an edge set and a corresponding edge index set for generating a path subgraph; Select two edge indexes from the edge index set, if the value of the first dimension of one of the edge indexes is the same as the value of the second dimension of the other edge index, then connect the edges corresponding to the two edge indexes end to end, traverse the edge index set, and obtain p path subgraphs, the p path subgraphs constitute a path subgraph set; The restoring the path subgraph set to the knowledge graph to obtain the knowledge subgraph set specifically includes: Mapping the edges in the path subgraph set and the nodes connected by the edges to the relationship list and entity set, and indexing the specific meaning of each edge in the path subgraph according to the relationship dictionary to obtain a knowledge subgraph set; The reply content generation module is configured to input the knowledge subgraph set into the large language model for reasoning, and generate reply content corresponding to the input question.
5. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.
7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.
Citation Information
Patent Citations
Knowledge graph multi-hop question and answer method and model based on cognitive reasoning
CN113360604A
Knowledge reasoning method based on heterogeneous knowledge fusion
CN117235216A
Intelligent trademark law question and answer method for enhancing big language model reasoning through knowledge graph
CN117807202A
Knowledge graph question and answer method and device and storage medium
CN118170894A
