A question and answer method and device based on knowledge graph retrieval and large language model

Abstract

The application provides a question and answer method and device based on knowledge graph retrieval and a large language model. The question and answer method comprises: in response to receiving natural language question information, using a knowledge graph retrieval system to obtain a corresponding candidate path set; using a large language model to coarsely filter and finely filter the candidate path set to obtain a coarsely filtered path set and a target path set and construct a structure prompt instruction, to obtain a first answer result output by the knowledge graph retrieval system and a second answer result output by the large language model, respectively, and to calculate the corresponding confidence values, respectively, to screen and correspondingly fuse the first answer result and the second answer result according to the confidence values, respectively, to generate a target answer result corresponding to the natural language question information. Through the above method, the accuracy of answer generation of the large language model in the question and answer reasoning task is improved, and the stability and performance of the large language model based on knowledge graph retrieval are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence question answering technology, and in particular to a question answering method and apparatus based on knowledge graph retrieval and large language model. Background Technology

[0002] Large Language Models (LLMs) perform well in reasoning tasks in the field of Natural Language Processing (NLP), but their generated content suffers from the "illusion" problem, that is, generating information that does not match real knowledge, which is particularly prominent in tasks such as knowledge-based question answering and fact-based reasoning. To overcome this problem, Retrieval-Augmented Generation (RAG) technology has been applied to large language models, effectively improving their accuracy and stability.

[0003] Currently, when using large language models with knowledge graph-supported retrieval enhancement generation systems for natural language question-answering reasoning tasks, the retrieved graph information may contain noise or irrelevant paths, reducing the accuracy of answer generation in question-answering reasoning tasks. In addition, large language models rely excessively on external knowledge, neglecting the reasoning and knowledge capabilities they already possess, and may even be misled in scenarios where the model could have answered correctly, thus reducing the stability and performance of knowledge graph-based large language models. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a question-answering method and apparatus based on knowledge graph retrieval and a large language model. The method obtains a candidate path set corresponding to the input natural language question information through a knowledge graph retrieval system, and then uses a large language model to perform coarse and fine filtering on the candidate path set sequentially. This removes low-relevance paths and reduces noise interference while retaining highly relevant paths, resulting in a coarsely filtered path set and a target path set. A structural prompt instruction corresponding to the natural language question information is constructed, and the knowledge graph retrieval system and the large language model are used to infer and output the corresponding answer result. Furthermore, the answer result is filtered and fused based on its confidence level to generate the target answer result corresponding to the natural language question information. This improves the accuracy of answer generation by the large language model in question-answering reasoning tasks, thereby improving the stability and performance of the knowledge graph retrieval-based large language model.

[0005] This application provides a question-answering method based on knowledge graph retrieval and a large language model, the question-answering method including:

[0006] In response to receiving natural language question information input by the user, a set of candidate paths corresponding to the natural language question information is obtained using a preset knowledge graph retrieval system;

[0007] The candidate path set is sequentially subjected to coarse filtering and fine filtering using a preset large language model, resulting in a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering.

[0008] Based on the coarse filtering path set and the target path set, a structural prompt instruction corresponding to the natural language question information is constructed, and the structural prompt instruction is input into the knowledge graph retrieval system and the large language model respectively, to obtain the first answer result output by the knowledge graph retrieval system and the second answer result output by the large language model respectively;

[0009] Calculate the confidence scores corresponding to the first answer result and the second answer result respectively, and filter and fuse the first answer result and the second answer result according to the confidence scores respectively to generate the target answer result corresponding to the natural language question information.

[0010] Furthermore, the step of obtaining the candidate path set corresponding to the natural language question information using a preset knowledge graph retrieval system includes:

[0011] The keywords in the natural language question information are determined using a preset knowledge graph retrieval system, and the target keyword information corresponding to the keywords is matched in the knowledge graph set by the preset knowledge graph retrieval system;

[0012] Based on the target keyword information, vector semantic retrieval is performed in the knowledge graph to determine the semantic information corresponding to each keyword in the target keyword information;

[0013] Based on the target keyword information and the semantic information, a graph structure traversal is performed in the knowledge graph to generate multiple candidate paths corresponding to the natural language question information, so as to obtain a set of candidate paths corresponding to the natural language question information.

[0014] Furthermore, the step of sequentially performing coarse filtering and fine filtering on the candidate path set using a preset large language model to obtain a coarsely filtered path set output by the large language model through coarse filtering, and a target path set output by the large language model through fine filtering, includes:

[0015] The attention value, representing the degree of attention of each candidate path to the reasoning of the large language model, is calculated using a preset large language model. The attention value is then filtered using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, which is used to coarsely filter the candidate path set.

[0016] The semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set is evaluated using the large language model, and the semantic score value is filtered using a preset semantic score threshold to obtain the target path set output by the large language model, so as to perform fine filtering on the target path set.

[0017] Furthermore, the step of calculating the attention value corresponding to each candidate path in the candidate path set, representing the degree of attention the candidate path pays to the reasoning of the large language model, using a preset large language model, and filtering the attention values ​​using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, for coarse filtering of the candidate path set, includes:

[0018] The candidate path set and the natural language question information are concatenated to construct a coarse filtering inference instruction corresponding to each candidate path in the candidate path set;

[0019] The coarse filtering inference instruction is input into a preset large language model, and the large language model is used to calculate the attention value corresponding to each candidate path, which represents the degree of attention of the candidate path to the inference of the large language model.

[0020] The attention value corresponding to each candidate path is compared with a preset attention threshold to obtain the coarse filtering comparison result for each candidate path.

[0021] Candidate paths whose attention value is less than the preset attention threshold are removed from the candidate path set by the coarse filtering comparison result, and a coarsely filtered path set output by the large language model is obtained to perform coarse filtering on the candidate path set.

[0022] Furthermore, the step of evaluating the semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set using the large language model, and filtering the semantic score values ​​using a preset semantic score threshold to obtain the target path set output by the large language model, for fine filtering of the target path set, includes:

[0023] The coarse filtering path set and the natural language question information are concatenated to construct the fine filtering reasoning instruction corresponding to each coarse filtering path in the coarse filtering path set;

[0024] The fine filtering inference instruction is input into the large language model, and the large language model is used to perform semantic understanding on the fine filtering inference instruction. The semantic score value corresponding to each coarse filtering path in the coarse filtering path set is evaluated to assess the semantic similarity between each coarse filtering path and the corresponding natural language question information.

[0025] The semantic score value corresponding to each coarse filtering path is compared with a preset semantic score threshold to obtain the fine filtering comparison result corresponding to each coarse filtering path.

[0026] The coarse-filtered paths whose semantic score is less than the preset semantic score threshold are removed from the coarse-filtered path set to obtain the target path set output by the large language model, and then the target path set is fine-filtered.

[0027] Furthermore, the step of constructing the structured prompt instruction corresponding to the natural language question information based on the coarse-filtered path set and the target path set includes:

[0028] The target path set is determined as the priority path set, and the difference between the coarse-filtered path set and the target path set is calculated to obtain the additional reference path set;

[0029] Each priority path in the priority path set is placed in the first priority position corresponding to the structure prompt instruction corresponding to the natural language question information, and each additional reference path in the additional reference path set is placed in the second priority position corresponding to the structure prompt instruction, so as to construct the structure prompt instruction.

[0030] Further, the step of calculating the confidence scores corresponding to the first answer result and the second answer result, and filtering and fusing the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, includes:

[0031] The first confidence score corresponding to the first answer result is calculated using the large language model, and the second confidence score corresponding to the second answer result is calculated.

[0032] The first confidence value is filtered using a first confidence threshold to obtain a first target answer set corresponding to the knowledge graph retrieval system, and the second confidence value is filtered using a second confidence threshold to obtain a second target answer set corresponding to the large language model.

[0033] The first target answer set and the second target answer set are fused using a preset fusion mechanism based on external instructions to generate the target answer result corresponding to the natural language question information; wherein, the preset fusion mechanism includes at least one of confidence priority, fusion reasoning score ranking, voting mechanism and merge set ranking.

[0034] Furthermore, the step of calculating the confidence scores corresponding to the first answer result and the second answer result, and filtering and fusing the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, further includes:

[0035] In response to the inconsistency between the first answer result and the second answer result, a verification instruction is generated and input into the knowledge graph retrieval system and the large language model respectively;

[0036] The knowledge graph retrieval system and the large language model perform self-evaluation respectively to determine the target answer result in the first answer result and the second answer result.

[0037] This application also provides a question-answering device based on knowledge graph retrieval and a large language model, the question-answering device comprising:

[0038] The path acquisition module is used to, in response to receiving natural language question information input by the user, use a preset knowledge graph retrieval system to obtain a set of candidate paths corresponding to the natural language question information;

[0039] The path filtering module is used to perform coarse filtering and fine filtering on the candidate path set in sequence using a preset large language model, so as to obtain the coarse-filtered path set output by the large language model through coarse filtering and the target path set output by the large language model through fine filtering.

[0040] The answer reasoning module is used to construct a structural prompt instruction corresponding to the natural language question information based on the coarse filter path set and the target path set, and input the structural prompt instruction into the knowledge graph retrieval system and the large language model respectively to obtain the first answer result output by the knowledge graph retrieval system and the second answer result output by the large language model respectively.

[0041] The answer fusion module is used to calculate the confidence scores corresponding to the first answer result and the second answer result, respectively, and to filter and fuse the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information.

[0042] Furthermore, when the path acquisition module is used to obtain the candidate path set corresponding to the natural language question information using a preset knowledge graph retrieval system, the path acquisition module is used to:

[0043] The keywords in the natural language question information are determined using a preset knowledge graph retrieval system, and the target keyword information corresponding to the keywords is matched in the knowledge graph set by the preset knowledge graph retrieval system;

[0044] Based on the target keyword information, vector semantic retrieval is performed in the knowledge graph to determine the semantic information corresponding to each keyword in the target keyword information;

[0045] Based on the target keyword information and the semantic information, a graph structure traversal is performed in the knowledge graph to generate multiple candidate paths corresponding to the natural language question information, so as to obtain a set of candidate paths corresponding to the natural language question information.

[0046] Furthermore, when the path filtering module is used to sequentially perform coarse filtering and fine filtering on the candidate path set using a preset large language model, corresponding to obtaining a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering, the path filtering module is used to:

[0047] The attention value, representing the degree of attention of each candidate path to the reasoning of the large language model, is calculated using a preset large language model. The attention value is then filtered using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, which is used to coarsely filter the candidate path set.

[0048] The semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set is evaluated using the large language model, and the semantic score value is filtered using a preset semantic score threshold to obtain the target path set output by the large language model, so as to perform fine filtering on the target path set.

[0049] Furthermore, when the path filtering module calculates the attention value corresponding to each candidate path in the candidate path set, representing the degree of attention the candidate path pays to the reasoning of the large language model, using a preset large language model, and filters the attention value using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, for coarse filtering of the candidate path set, the path filtering module is used to:

[0050] The candidate path set and the natural language question information are concatenated to construct a coarse filtering inference instruction corresponding to each candidate path in the candidate path set;

[0051] The coarse filtering inference instruction is input into a preset large language model, and the large language model is used to calculate the attention value corresponding to each candidate path, which represents the degree of attention of the candidate path to the inference of the large language model.

[0052] The attention value corresponding to each candidate path is compared with a preset attention threshold to obtain the coarse filtering comparison result for each candidate path.

[0053] Candidate paths whose attention value is less than the preset attention threshold are removed from the candidate path set by the coarse filtering comparison result, and a coarsely filtered path set output by the large language model is obtained to perform coarse filtering on the candidate path set.

[0054] Furthermore, when the path filtering module uses the large language model to evaluate the semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set, and uses a preset semantic score threshold to filter the semantic score values ​​to obtain the target path set output by the large language model, and then performs fine filtering on the target path set, the path filtering module is used to:

[0055] The coarse filtering path set and the natural language question information are concatenated to construct the fine filtering reasoning instruction corresponding to each coarse filtering path in the coarse filtering path set;

[0056] The fine filtering inference instruction is input into the large language model, and the large language model is used to perform semantic understanding on the fine filtering inference instruction. The semantic score value corresponding to each coarse filtering path in the coarse filtering path set is evaluated to assess the semantic similarity between each coarse filtering path and the corresponding natural language question information.

[0057] The semantic score value corresponding to each coarse filtering path is compared with a preset semantic score threshold to obtain the fine filtering comparison result corresponding to each coarse filtering path.

[0058] The coarse-filtered paths whose semantic score is less than the preset semantic score threshold are removed from the coarse-filtered path set to obtain the target path set output by the large language model, and then the target path set is fine-filtered.

[0059] Furthermore, when the answer reasoning module constructs the structural prompt instruction corresponding to the natural language question information based on the coarse-filtered path set and the target path set, the answer reasoning module is used to:

[0060] The target path set is determined as the priority path set, and the difference between the coarse-filtered path set and the target path set is calculated to obtain the additional reference path set;

[0061] Each priority path in the priority path set is placed in the first priority position corresponding to the structure prompt instruction corresponding to the natural language question information, and each additional reference path in the additional reference path set is placed in the second priority position corresponding to the structure prompt instruction, so as to construct the structure prompt instruction.

[0062] Furthermore, when the answer fusion module calculates the confidence scores corresponding to the first answer result and the second answer result, and filters and fuses the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, the answer fusion module is used to:

[0063] The first confidence score corresponding to the first answer result is calculated using the large language model, and the second confidence score corresponding to the second answer result is calculated.

[0064] The first confidence value is filtered using a first confidence threshold to obtain a first target answer set corresponding to the knowledge graph retrieval system, and the second confidence value is filtered using a second confidence threshold to obtain a second target answer set corresponding to the large language model.

[0065] The first target answer set and the second target answer set are fused using a preset fusion mechanism based on external instructions to generate the target answer result corresponding to the natural language question information; wherein, the preset fusion mechanism includes at least one of confidence priority, fusion reasoning score ranking, voting mechanism and merge set ranking.

[0066] Furthermore, when the answer fusion module calculates the confidence scores corresponding to the first answer result and the second answer result, and filters and fuses the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, the answer fusion module is also used for:

[0067] In response to the inconsistency between the first answer result and the second answer result, a verification instruction is generated and input into the knowledge graph retrieval system and the large language model respectively;

[0068] The knowledge graph retrieval system and the large language model perform self-evaluation respectively to determine the target answer result in the first answer result and the second answer result.

[0069] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the question-answering method based on knowledge graph retrieval and large language model described above are performed.

[0070] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the question-answering method based on knowledge graph retrieval and large language model as described above.

[0071] This application provides a question-answering method and apparatus based on knowledge graph retrieval and a large language model. The question-answering method includes: responding to receiving natural language question information input by a user, obtaining a set of candidate paths corresponding to the natural language question information using a preset knowledge graph retrieval system; sequentially performing coarse filtering and fine filtering on the candidate path set using a preset large language model to obtain a set of coarse-filtered paths output by the large language model through coarse filtering, and a set of target paths output by the large language model through fine filtering; constructing a structured prompt instruction corresponding to the natural language question information based on the set of coarse-filtered paths and the set of target paths, and inputting the structured prompt instruction into the knowledge graph retrieval system and the large language model respectively to obtain a first answer result output by the knowledge graph retrieval system and a second answer result output by the large language model respectively; calculating the confidence scores corresponding to the first answer result and the second answer result respectively, and filtering and fusing the first answer result and the second answer result according to the confidence scores to generate a target answer result corresponding to the natural language question information.

[0072] Compared to existing technologies that rely on noisy or irrelevant paths in the retrieved graph information and excessive dependence on external knowledge by large language models, neglecting the model's inherent reasoning and knowledge capabilities, this approach obtains a candidate path set corresponding to the input natural language question information through a knowledge graph retrieval system. Then, a large language model sequentially performs coarse and fine filtering on this candidate path set to remove low-relevance paths and reduce noise interference while retaining highly relevant paths, resulting in a coarsely filtered path set and a target path set. A structural prompt instruction corresponding to the natural language question information is constructed, and the knowledge graph retrieval system and the large language model are used to reason and output the corresponding answer result. Finally, based on the confidence level of the answer result, further filtering and fusion are performed to generate the target answer result corresponding to the natural language question information. This improves the accuracy of answer generation by the large language model in question-answering reasoning tasks, thereby enhancing the stability and performance of the knowledge graph retrieval-based large language model.

[0073] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A flowchart illustrating a question-answering method based on knowledge graph retrieval and a large language model, provided as an embodiment of this application;

[0076] Figure 2 A schematic diagram of the structure of a question-answering device based on knowledge graph retrieval and a large language model provided in an embodiment of this application;

[0077] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0079] Research has found that current mainstream knowledge graph-based retrieval enhancement generation systems (such as ROG, GNN-RAG, and SubgraphRAG) all use graph-structured information supplementation large language models for generation. Their core process includes: receiving the user's natural language query; retrieving relevant paths or triples in the knowledge graph; concatenating the retrieved content with the original question to form a prompt instruction which is then input into the large language model; and the model outputting the answer.

[0080] In particular, when using a large language model with a knowledge graph-supported retrieval enhancement generation system for natural language question-answering reasoning tasks, the retrieved graph information may contain noise or irrelevant paths, reducing the accuracy of the large language model in generating answers during question-answering reasoning tasks. In addition, the large language model relies excessively on external knowledge, neglecting the reasoning and knowledge capabilities that the model itself already possesses, and may even be misled in scenarios where the model could have answered correctly, thus reducing the stability and performance of the knowledge graph-based large language model.

[0081] Based on this, this application provides a question-answering method based on knowledge graph retrieval and a large language model. The method obtains a candidate path set corresponding to the input natural language question information through a knowledge graph retrieval system, and then uses a large language model to perform coarse and fine filtering on the candidate path set sequentially. This removes low-relevance paths and reduces noise interference while retaining highly relevant paths, resulting in a coarsely filtered path set and a target path set. A structural prompt instruction corresponding to the natural language question information is then constructed. The knowledge graph retrieval system and the large language model are used to infer and output the corresponding answer result. Finally, the confidence level of the answer result is used for filtering and fusion to generate the target answer result corresponding to the natural language question information. This improves the accuracy of answer generation by the large language model in question-answering reasoning tasks, thereby improving the stability and performance of the knowledge graph retrieval-based large language model.

[0082] Please see Figure 1 , Figure 1This is a flowchart illustrating a question-answering method based on knowledge graph retrieval and a large language model, provided as an embodiment of this application. Figure 1 As shown in the embodiments of this application, the question-answering method based on knowledge graph retrieval and large language model includes:

[0083] S101. In response to receiving natural language question information input by the user, obtain the candidate path set corresponding to the natural language question information using a preset knowledge graph retrieval system.

[0084] In this embodiment, the preset knowledge graph retrieval system is a GraphRAG system, which is an extension and enhancement of Retrieval-Augmented Generation (RAG) technology. It introduces knowledge graph or graph structure information on the basis of traditional RAG to improve the accuracy of information retrieval and generation and the ability to understand context.

[0085] Here, the natural language question information input by the user is a natural language question in which the user expects to obtain the corresponding answer through artificial intelligence question answering, such as "Who is the CEO of Company A?" or "Which team won the 2022 FIFA World Cup?"

[0086] The natural language question information input by the user is a string-type data in natural language form, denoted as Q.

[0087] In the embodiments of this application, each candidate path in the candidate path set is a graph path composed of a series of entities and relations. The path form of each candidate path may include triple form (head, relation, tail) and multiple path form (entity 1 → relation 1 → entity 2 → relation 2 → entity 3), etc.

[0088] For example, a set of paths can be represented as "P = {p1, p2, ..., p n}”, p i This can include: "h1→r1→h2→r2→h3"; where h represents an entity, r represents a relation, and p... n This represents each candidate path.

[0089] In one possible implementation of this application, the step S101 of obtaining the candidate path set corresponding to the natural language question information using a preset knowledge graph retrieval system may include:

[0090] S1011. Use a preset knowledge graph retrieval system to determine the keywords in the natural language question information, and match the target keyword information corresponding to the keywords in the knowledge graph set by the preset knowledge graph retrieval system.

[0091] In this step, a knowledge graph retrieval system is used to extract keywords from the natural language question information, and the keywords are then searched in the knowledge graph set by the knowledge graph retrieval system to determine the target keyword information corresponding to the matching keywords.

[0092] S1012. Based on the target keyword information, perform vector semantic retrieval in the knowledge graph to determine the semantic information corresponding to each keyword in the target keyword information.

[0093] In this step, the knowledge graph retrieval system performs vector semantic retrieval in the knowledge graph based on the target keyword information to determine the semantic information corresponding to each keyword in the target keyword information.

[0094] S1013. Based on the target keyword information and the semantic information, perform graph structure traversal in the knowledge graph to generate multiple candidate paths corresponding to the natural language question information, so as to obtain a set of candidate paths corresponding to the natural language question information.

[0095] In the embodiments of this application, graph structure traversal may include at least one of the methods such as shortest path and PageRank.

[0096] For example, the feature information corresponding to each candidate path in the candidate path set includes: text format representation (p i .content, for example, "Apple → Company → Apple Inc."), and encoded text unit (Token) sequences (p i .tokens), path semantic vectors (p i . embedding) and attention score (a i )wait.

[0097] S102. The candidate path set is sequentially subjected to coarse filtering and fine filtering using a preset large language model, resulting in a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering.

[0098] In this embodiment of the application, the candidate path set is subjected to coarse filtering and fine filtering in sequence, which can solve the problem of inference being interfered with by irrelevant or noisy paths in the prior art.

[0099] In one possible implementation of this application, step S102 may include:

[0100] S1021. Calculate the attention value corresponding to each candidate path in the candidate path set using a preset large language model, which represents the degree of attention the candidate path pays to the reasoning of the large language model. Then, filter the attention value using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, so as to coarsely filter the candidate path set.

[0101] In this embodiment, the coarse filtering is based on attention scores. It uses the intermediate layer inside the large language model to perform initial screening on the attention value assigned to each candidate path in the candidate path set, discarding low attention paths.

[0102] Specifically, coarse filtering utilizes the attention weights of the intermediate Transformer layer within the large language model to analyze the attention level of each candidate path to the model's inference, i.e., the attention value. Only candidate paths with attention values ​​higher than a preset attention threshold are retained to quickly eliminate a large number of noise and / or low-relevance paths, reducing subsequent computational overhead.

[0103] In one possible implementation of this application, step S1021 may include:

[0104] S10211. The candidate path set and the natural language question information are concatenated to construct a coarse filtering inference instruction corresponding to each candidate path in the candidate path set.

[0105] In this step, for each candidate path in the candidate path set, the candidate path is concatenated with natural language question information to construct a coarse filtering inference instruction represented as a graph enhancement.

[0106] For example, for each candidate path P i The coarse-filtered inference instruction represented as graph enhancement is constructed as "Prompt: Prompt(pᵢ) = "Please refer to the following path to answer the question: {pᵢ.content}, Question: {Q}"".

[0107] S10212. Input the coarse filtering inference instruction into the preset large language model, and use the large language model to calculate the attention value corresponding to each candidate path, which represents the degree of attention of the candidate path to the inference of the large language model.

[0108] In this step, the coarse filtering inference instructions are input into the preset large language model, and the attention value corresponding to each candidate path is calculated using the Transformer layer in the middle of the large language model.

[0109] The attention value corresponding to each candidate path represents the degree of attention that candidate path pays to the reasoning of the large language model.

[0110] Specifically, the attention weights of the L / 2+2th layer (intermediate layer) in the Transformer layer of the large language model for each candidate path are obtained, and the average attention of the large language model for each candidate path is used as the attention value of each candidate path.

[0111] Here, the formula for calculating the attention value corresponding to each candidate path is shown below.

[0112] .

[0113] in, Indicate each candidate path The corresponding token set; This represents the attention value corresponding to each candidate path; This represents the attention weight calculated for each candidate path's token in the L / 2+2th (intermediate) layer of the Transformer layer in the large language model.

[0114] S10213. The attention value corresponding to each candidate path is compared with the preset attention threshold to obtain the coarse filtering comparison result corresponding to each candidate path.

[0115] In this step, the attention value corresponding to each candidate path is compared with a preset attention threshold to determine whether the attention value corresponding to each candidate path is less than the preset attention threshold, thus obtaining the coarse filtering comparison result for each candidate path.

[0116] S10214. Remove the candidate paths in the candidate path set whose coarse filtering comparison result is less than the preset attention threshold, and obtain the coarsely filtered path set output by the large language model, so as to perform coarse filtering on the candidate path set.

[0117] In this embodiment of the application, the expression for the coarse-filtered path set is as follows.

[0118] .

[0119] in, Represents the set of coarse filtering paths; Represents the set of candidate paths. This represents each candidate path in the candidate path set; This indicates a preset attention threshold; This represents the attention value corresponding to each candidate path.

[0120] Here, the preset attention threshold is generally selected in the range of [0.1, 0.3]. To ensure the quality of the path, the number of paths in the output coarse-filtered path set is usually controlled within the preset number.

[0121] It should be noted that, in order to reduce computational overhead, multiple coarse filtering inference instructions can be inferred simultaneously in batch processing. If the large language model supports multi-head attention, the maximum head or the average value can be used as the attention value.

[0122] S1022. The semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set is evaluated using the large language model, and the semantic score value is filtered using a preset semantic score threshold to obtain the target path set output by the large language model, so as to perform fine filtering on the target path set.

[0123] In this embodiment, fine filtering is based on large language model evaluation. The path set that has passed coarse filtering is input into the large language model again for semantic relevance evaluation, and only highly relevant paths are retained.

[0124] In one possible implementation of this application, step S1022 may include:

[0125] S10221. The coarse filtering path set and the natural language question information are concatenated to construct the fine filtering inference instruction corresponding to each coarse filtering path in the coarse filtering path set.

[0126] In this step, for each coarse filtering path in the coarse filtering path set, the coarse filtering path is concatenated with natural language question information to construct the fine filtering inference instruction corresponding to each coarse filtering path.

[0127] For example, for each coarse filtering path P i Construct a question-and-answer verification-type fine-filtering inference instruction prompt: "Prompt(pᵢ) = "Does the path: {pᵢ.content} help answer the question: {Q}?".

[0128] S10222. Input the fine filtering inference instruction into the large language model, use the large language model to perform semantic understanding on the fine filtering inference instruction, and evaluate the semantic score value corresponding to each coarse filtering path in the coarse filtering path set, so as to evaluate the semantic similarity between each coarse filtering path and the corresponding natural language question information.

[0129] In this embodiment, the output of the large language model is used to evaluate whether the coarse filtering path is useful, that is, whether the coarse filtering path is related to the semantics of the question. The semantic understanding output of the large language model has two forms, depending on which discrimination strategy is adopted.

[0130] The discrimination strategies include classification discrimination (yes or no or probability) and scoring discrimination (scores after Softmax or Sigmoid).

[0131] Specifically, for classification discrimination, the large language model returns a classification result (yes or no or probability), such as "f(p) = 0.89"; for scoring discrimination, the large language model generates a semantic similarity score for the coarse-filtered path and the natural language question, which can be achieved by using the last layer logit or scoring vector of the large language model, or by using a lightweight language model (e.g., BERT) to output a (0,1) relevance score.

[0132] It should be noted that using a lightweight language model that is of the same origin as the large language model for fine filtering helps to ensure semantic consistency. Low-rank fine-tuning models such as LoRA and QLoRA can be used to quickly deploy the discrimination module.

[0133] S10223. The semantic score value corresponding to each coarse filtering path is compared with a preset semantic score threshold to obtain the fine filtering comparison result corresponding to each coarse filtering path.

[0134] In this step, the semantic score value corresponding to each coarse filtering path is compared with the preset semantic score threshold to determine whether the semantic score value corresponding to each coarse filtering path is less than the preset semantic score threshold, thus obtaining the fine filtering comparison result corresponding to each coarse filtering path.

[0135] Here, the preset semantic score threshold is generally selected within the numerical range of [0.6, 0.8].

[0136] S10224. Remove the coarse-filtered paths from the coarse-filtered path set whose semantic score is less than the preset semantic score threshold, and obtain the target path set output by the large language model, so as to perform fine filtering on the target path set.

[0137] In this embodiment of the application, the expression for the target path set is as follows.

[0138] .

[0139] in, Represents the set of target paths; Represents the set of coarse-filtered paths. This represents each coarse filter path in the set of coarse filter paths; This represents the semantic score value corresponding to each coarse-filtered path; This indicates the preset semantic score threshold.

[0140] S103. Based on the coarse filtering path set and the target path set, construct the structural prompt instruction corresponding to the natural language question information, and input the structural prompt instruction into the knowledge graph retrieval system and the large language model respectively, to obtain the first answer result output by the knowledge graph retrieval system and the second answer result output by the large language model respectively.

[0141] It should be noted that this application provides an architecture model that integrates a large language model and a knowledge graph retrieval system. This architecture model is bidirectionally enhanced and deeply integrated, rather than simply connecting the large language model and the graph. It includes: the knowledge graph is used for external knowledge supplementation; the large language model is used for path evaluation, semantic understanding, and native generation; and the large language model and the knowledge graph retrieval system are deeply integrated at the two levels of path selection and answer output.

[0142] In this embodiment, the first answer result may include a path-enhanced answer candidate set, generated by the knowledge graph retrieval system processing the path-enhanced structural prompts; the second answer result may include a native answer candidate set, generated by the large language model processing the natural language question information in the structural prompts without using path enhancement.

[0143] Here, the structure prompt instruction may include a priority path set and an additional reference path set; wherein, the priority path set is the target path set, and the additional reference path set is the path set obtained by subtracting the target path set from the coarsely filtered path set.

[0144] Here, the structure of each answer item in the answer result is: "AnswerItem: text: str; logits:float; source: str", where "text: str" represents the answer text; "logits: float" represents the confidence score; and "source: str" represents the source of the answer (knowledge graph retrieval system or large language model).

[0145] In one possible implementation of this application, in specific implementation, the step S103 of constructing the structured prompt instruction corresponding to the natural language question information based on the coarse-filtered path set and the target path set may include:

[0146] S1031. The target path set is determined as the priority path set, and the difference between the coarse-filtered path set and the target path set is calculated to obtain the additional reference path set.

[0147] In this embodiment, the target path set is determined as the priority path set, and each target path in the target path set is identified as a priority path and presented first in the structure prompt instruction to emphasize the importance of the priority path; the difference between the coarse-filtered path set and the target path set is calculated to obtain an additional reference path set representing the difference paths between the coarse-filtered path set and the target path set, which is presented secondarily in the structure prompt instruction and can be used as background knowledge to assist the reasoning of the large language model.

[0148] S1032. Place each priority path in the priority path set in the first priority position corresponding to the structure prompt instruction corresponding to the natural language question information, and place each additional reference path in the additional reference path set in the second priority position corresponding to the structure prompt instruction to construct the structure prompt instruction.

[0149] For example, inserting natural language prompts to enhance contextual understanding, the structured prompt instruction can be represented as: "Please answer the question based on the following knowledge paths, and keep your answers concise and clear. High priority paths: -Path 1; -Path 2; ... Additional paths: -Path A; -Path B; ...; Question: Q."

[0150] Furthermore, optional structural prompt instruction template variants add path annotation information, such as "This path contains person / location relationships," and support interactive embedding prompts between paths and questions, such as "Combining paths and questions for joint reasoning."

[0151] In this way, by constructing structural prompts corresponding to natural language question information, high-priority paths and additional paths are distinguished and organized into prompt words, guiding the large language model to utilize external knowledge more accurately.

[0152] S104. Calculate the confidence scores corresponding to the first answer result and the second answer result respectively, and filter and fuse the first answer result and the second answer result according to the confidence scores respectively to generate the target answer result corresponding to the natural language question information.

[0153] In one possible implementation of this application, step S104 may include:

[0154] S1041. Calculate the first confidence value corresponding to the first answer result using the large language model, and calculate the second confidence value corresponding to the second answer result.

[0155] In this step, the confidence score value corresponding to each answer in the first answer result and the second answer result is obtained respectively; specifically, the confidence score corresponding to each answer is the logits score after being processed by the softmax function.

[0156] Here, logits are the raw values ​​output by the last layer (usually a fully connected layer) of the large language model. These values ​​are not normalized before being processed by the activation function. The softmax activation function converts the logits scores into a probability distribution, so that the confidence value of each answer output is between (0, 1).

[0157] Here, the confidence score for each answer may include any one of the following: token-level average, terminator token (EOS) confidence score, or overall CLS score.

[0158] S1042. The first confidence value is filtered using a first confidence threshold to obtain the first target answer set corresponding to the knowledge graph retrieval system, and the second confidence value is filtered using a second confidence threshold to obtain the second target answer set corresponding to the large language model.

[0159] In this step, the first confidence score value corresponding to each answer in the first answer result is compared with the first confidence threshold to determine whether the first confidence score value is greater than or equal to the first confidence threshold. Each answer in the first answer result whose first confidence score value is greater than or equal to the first confidence threshold is retained, and each answer in the first answer result whose first confidence score value is less than the first confidence threshold is removed, thus obtaining the first target answer set corresponding to the knowledge graph retrieval system.

[0160] Here, the first target answer set can be represented as “A_filtered_G = {a∈A_G |ℓ_G(a)≥τ_G}”; where A_filtered_G represents the first target answer set, A_G represents the first answer result, a represents each answer in the first answer result, ℓ_G(a) represents the first confidence value corresponding to each answer in the first answer result, and τ_G represents the first confidence threshold.

[0161] Furthermore, the second confidence score value corresponding to each answer in the second answer result is compared with the second confidence threshold to determine whether the second confidence score value is greater than or equal to the second confidence threshold. Each answer in the second answer result whose second confidence score value is greater than or equal to the second confidence threshold is retained, and each answer in the second answer result whose second confidence score value is less than the second confidence threshold is removed, thus obtaining the second target answer set corresponding to the large language model.

[0162] Here, the first target answer set can be represented as “A_filtered_L = {b∈A_L |ℓ_L(b)≥τ_L}”; where A_filtered_L represents the second target answer set, A_L represents the second answer result, b represents each answer in the second answer result, ℓ_L(b) represents the second confidence value corresponding to each answer in the second answer result, and τ_L represents the second confidence threshold.

[0163] S1043. Using a preset fusion mechanism based on external instructions, the first target answer set and the second target answer set are fused to generate the target answer result corresponding to the natural language question information.

[0164] The preset fusion mechanism includes at least one of the following: confidence priority, fusion reasoning score ranking, voting mechanism, and merge set ranking.

[0165] Specifically, confidence priority means that the answer with the highest confidence score is selected first, and the answer with the highest confidence score is taken as the final target answer.

[0166] The fusion reasoning score ranking is performed by calculating the weighted fusion score between the first target answer set and the second target answer set to generate the target answer result. The expression for calculating the weighted fusion score is shown below.

[0167] K=α×ℓ_G(a)+β×ℓ_L(b).

[0168] Where K represents the weighted fusion score; ℓ_G(a) represents the first target answer set; ℓ_L(b) represents the second target answer set; α represents the first fusion weight; and β represents the second fusion weight.

[0169] The voting mechanism involves selecting answers from the multiple generated answers in the first and second target answer sets according to their frequency.

[0170] Merging and sorting the sets involves merging the first target answer set with the second target answer set, and selecting a predetermined number of answers from the order of confidence scores as the target answer results.

[0171] For example, the decision function corresponding to the preset fusion mechanism can be expressed as "def combine(A_filtered_G,A_filtered_L):all_answers=A_filtered_G+A_filtered_L;sorted_ans=sorted(all_answers,key=lambda x:x.logits,reverse=True);return sorted_ans[0].text".

[0172] In another possible implementation of this application, step S104 further includes:

[0173] S1044. In response to the inconsistency between the first answer result and the second answer result, a verification instruction is generated, and the verification instruction is input into the knowledge graph retrieval system and the large language model respectively.

[0174] For example, verification instructions could include "Which of the two answers is more credible? Please explain your reasoning. Please judge which is more correct based on your knowledge?"

[0175] S1045. The knowledge graph retrieval system and the large language model perform self-evaluation respectively to determine the target answer result in the first answer result and the second answer result.

[0176] In this way, when answers are inconsistent, the large language model is guided to self-assess its credibility or use validation hints to determine the correct answer.

[0177] The question-answering method based on knowledge graph retrieval and a large language model provided in this application obtains a set of candidate paths corresponding to the input natural language question information through a knowledge graph retrieval system. The large language model then performs coarse and fine filtering on the candidate path set to remove low-interest paths and reduce noise interference while retaining highly relevant paths, resulting in a coarsely filtered path set and a target path set. A structural prompt instruction corresponding to the natural language question information is constructed, and the knowledge graph retrieval system and the large language model are used to infer and output the corresponding answer result. Furthermore, the answer result is filtered and fused based on its confidence level to generate the target answer result corresponding to the natural language question information. This improves the accuracy of answer generation by the large language model in question-answering reasoning tasks, thereby enhancing the stability and performance of the knowledge graph retrieval-based large language model.

[0178] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a question-answering device based on knowledge graph retrieval and a large language model, provided as an embodiment of this application. Figure 2As shown, the question-and-answer device 200 includes:

[0179] The path acquisition module 210 is used to, in response to receiving natural language question information input by the user, acquire a set of candidate paths corresponding to the natural language question information using a preset knowledge graph retrieval system;

[0180] The path filtering module 220 is used to perform coarse filtering and fine filtering on the candidate path set in sequence using a preset large language model, so as to obtain a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering.

[0181] The answer reasoning module 230 is used to construct a structural prompt instruction corresponding to the natural language question information based on the coarse filter path set and the target path set, and input the structural prompt instruction into the knowledge graph retrieval system and the large language model respectively, so as to obtain the first answer result output by the knowledge graph retrieval system and the second answer result output by the large language model respectively;

[0182] The answer fusion module 240 is used to calculate the confidence scores corresponding to the first answer result and the second answer result, respectively, and to filter and fuse the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information.

[0183] Furthermore, when the path acquisition module 210 is used to obtain the candidate path set corresponding to the natural language question information using a preset knowledge graph retrieval system, the path acquisition module 210 is used to:

[0184] The keywords in the natural language question information are determined using a preset knowledge graph retrieval system, and the target keyword information corresponding to the keywords is matched in the knowledge graph set by the preset knowledge graph retrieval system;

[0185] Based on the target keyword information, vector semantic retrieval is performed in the knowledge graph to determine the semantic information corresponding to each keyword in the target keyword information;

[0186] Based on the target keyword information and the semantic information, a graph structure traversal is performed in the knowledge graph to generate multiple candidate paths corresponding to the natural language question information, so as to obtain a set of candidate paths corresponding to the natural language question information.

[0187] Furthermore, when the path filtering module 220 sequentially performs coarse filtering and fine filtering on the candidate path set using a preset large language model, corresponding to obtaining a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering, the path filtering module 220 is used to:

[0188] The attention value, representing the degree of attention of each candidate path to the reasoning of the large language model, is calculated using a preset large language model. The attention value is then filtered using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, which is used to coarsely filter the candidate path set.

[0189] The semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set is evaluated using the large language model, and the semantic score value is filtered using a preset semantic score threshold to obtain the target path set output by the large language model, so as to perform fine filtering on the target path set.

[0190] Furthermore, when the path filtering module 220 calculates the attention value corresponding to each candidate path in the candidate path set, representing the degree of attention of the candidate path to the reasoning of the large language model, using a preset large language model, and filters the attention value using a preset attention threshold to obtain a coarsely filtered path set output by the large language model, for coarse filtering of the candidate path set, the path filtering module 220 is used to:

[0191] The candidate path set and the natural language question information are concatenated to construct a coarse filtering inference instruction corresponding to each candidate path in the candidate path set;

[0192] The coarse filtering inference instruction is input into a preset large language model, and the large language model is used to calculate the attention value corresponding to each candidate path, which represents the degree of attention of the candidate path to the inference of the large language model.

[0193] The attention value corresponding to each candidate path is compared with a preset attention threshold to obtain the coarse filtering comparison result for each candidate path.

[0194] Candidate paths whose attention value is less than the preset attention threshold are removed from the candidate path set by the coarse filtering comparison result, and a coarsely filtered path set output by the large language model is obtained to perform coarse filtering on the candidate path set.

[0195] Furthermore, when the path filtering module 220 evaluates the semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set using the large language model, and filters the semantic score values ​​using a preset semantic score threshold to obtain the target path set output by the large language model, and then performs fine filtering on the target path set, the path filtering module 220 is used to:

[0196] The coarse filtering path set and the natural language question information are concatenated to construct the fine filtering reasoning instruction corresponding to each coarse filtering path in the coarse filtering path set;

[0197] The fine filtering inference instruction is input into the large language model, and the large language model is used to perform semantic understanding on the fine filtering inference instruction. The semantic score value corresponding to each coarse filtering path in the coarse filtering path set is evaluated to assess the semantic similarity between each coarse filtering path and the corresponding natural language question information.

[0198] The semantic score value corresponding to each coarse filtering path is compared with a preset semantic score threshold to obtain the fine filtering comparison result corresponding to each coarse filtering path.

[0199] The coarse-filtered paths whose semantic score is less than the preset semantic score threshold are removed from the coarse-filtered path set to obtain the target path set output by the large language model, and then the target path set is fine-filtered.

[0200] Furthermore, when the answer reasoning module 230 constructs the structural prompt instruction corresponding to the natural language question information based on the coarse-filtered path set and the target path set, the answer reasoning module 230 is used to:

[0201] The target path set is determined as the priority path set, and the difference between the coarse-filtered path set and the target path set is calculated to obtain the additional reference path set;

[0202] Each priority path in the priority path set is placed in the first priority position corresponding to the structure prompt instruction corresponding to the natural language question information, and each additional reference path in the additional reference path set is placed in the second priority position corresponding to the structure prompt instruction, so as to construct the structure prompt instruction.

[0203] Furthermore, when the answer fusion module 240 calculates the confidence scores corresponding to the first answer result and the second answer result, and filters and fuses the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, the answer fusion module 240 is used to:

[0204] The first confidence score corresponding to the first answer result is calculated using the large language model, and the second confidence score corresponding to the second answer result is calculated.

[0205] The first confidence value is filtered using a first confidence threshold to obtain a first target answer set corresponding to the knowledge graph retrieval system, and the second confidence value is filtered using a second confidence threshold to obtain a second target answer set corresponding to the large language model.

[0206] The first target answer set and the second target answer set are fused using a preset fusion mechanism based on external instructions to generate the target answer result corresponding to the natural language question information; wherein, the preset fusion mechanism includes at least one of confidence priority, fusion reasoning score ranking, voting mechanism and merge set ranking.

[0207] Furthermore, when the answer fusion module 240 calculates the confidence scores corresponding to the first answer result and the second answer result, and filters and fuses the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, the answer fusion module 240 is also used to:

[0208] In response to the inconsistency between the first answer result and the second answer result, a verification instruction is generated and input into the knowledge graph retrieval system and the large language model respectively;

[0209] The knowledge graph retrieval system and the large language model perform self-evaluation respectively to determine the target answer result in the first answer result and the second answer result.

[0210] The question-answering device based on knowledge graph retrieval and a large language model provided in this application obtains a set of candidate paths corresponding to the input natural language question information through a knowledge graph retrieval system. It then uses a large language model to perform coarse and fine filtering on the candidate path set, removing low-interest paths and reducing noise interference while retaining highly relevant paths, resulting in a coarsely filtered path set and a target path set. A structural prompt instruction corresponding to the natural language question information is constructed, and the knowledge graph retrieval system and the large language model are used to infer and output the corresponding answer result. Furthermore, the confidence level of the answer result is used for filtering and corresponding fusion to generate the target answer result corresponding to the natural language question information. This improves the accuracy of answer generation by the large language model in question-answering reasoning tasks, thereby enhancing the stability and performance of the knowledge graph retrieval-based large language model.

[0211] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0212] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the question-answering method based on knowledge graph retrieval and large language model in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0213] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the question-answering method based on knowledge graph retrieval and large language model in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0214] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0216] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0218] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A question-answering method based on knowledge graph retrieval and a large language model, characterized in that, The question-and-answer method includes: In response to receiving natural language question information input by the user, a set of candidate paths corresponding to the natural language question information is obtained using a preset knowledge graph retrieval system; The candidate path set is sequentially subjected to coarse filtering and fine filtering using a preset large language model, resulting in a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering. The step of using a preset large language model to sequentially perform coarse filtering and fine filtering on the candidate path set, corresponding to obtaining a coarsely filtered path set output by the large language model through coarse filtering, and a target path set output by the large language model through fine filtering, includes: A pre-defined large language model is used to calculate the attention value corresponding to each candidate path in the candidate path set, representing the degree of attention the candidate path pays to the reasoning of the large language model. The attention value is then filtered using a pre-defined attention threshold to obtain a coarsely filtered path set output by the large language model, which is used to coarsely filter the candidate path set. Specifically, the attention weights calculated for each candidate path's token by the intermediate layer in the Transformer layer of the large language model are obtained, and the average attention calculated for each candidate path's token by the large language model is used as the attention value of each candidate path. The semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set is evaluated using the large language model, and the semantic score value is filtered using a preset semantic score threshold to obtain the target path set output by the large language model, so as to perform fine filtering on the target path set; Based on the coarse filtering path set and the target path set, a structural prompt instruction corresponding to the natural language question information is constructed, and the structural prompt instruction is input into the knowledge graph retrieval system and the large language model respectively, to obtain the first answer result output by the knowledge graph retrieval system and the second answer result output by the large language model respectively; Calculate the confidence scores corresponding to the first answer result and the second answer result respectively, and filter and fuse the first answer result and the second answer result according to the confidence scores respectively to generate the target answer result corresponding to the natural language question information.

2. The method according to claim 1, characterized in that, The step of obtaining the candidate path set corresponding to the natural language question information using a preset knowledge graph retrieval system includes: The keywords in the natural language question information are determined using a preset knowledge graph retrieval system, and the target keyword information corresponding to the keywords is matched in the knowledge graph set by the preset knowledge graph retrieval system; Based on the target keyword information, vector semantic retrieval is performed in the knowledge graph to determine the semantic information corresponding to each keyword in the target keyword information; Based on the target keyword information and the semantic information, a graph structure traversal is performed in the knowledge graph to generate multiple candidate paths corresponding to the natural language question information, so as to obtain a set of candidate paths corresponding to the natural language question information.

3. The method according to claim 1, characterized in that, The process involves calculating an attention value, representing the degree of attention a candidate path receives from the inference of the large language model, for each candidate path in the candidate path set using a preset large language model, and then filtering the attention values ​​using a preset attention threshold to obtain a coarsely filtered path set output by the large language model. This coarse filtering of the candidate path set includes: The candidate path set and the natural language question information are concatenated to construct a coarse filtering inference instruction corresponding to each candidate path in the candidate path set; The coarse filtering inference instruction is input into a preset large language model, and the large language model is used to calculate the attention value corresponding to each candidate path, which represents the degree of attention of the candidate path to the inference of the large language model. The attention value corresponding to each candidate path is compared with a preset attention threshold to obtain the coarse filtering comparison result for each candidate path. Candidate paths whose attention value is less than the preset attention threshold are removed from the candidate path set by the coarse filtering comparison result, and a coarsely filtered path set output by the large language model is obtained to perform coarse filtering on the candidate path set.

4. The method according to claim 1, characterized in that, The step of evaluating the semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set using the large language model, and filtering the semantic score values ​​using a preset semantic score threshold to obtain the target path set output by the large language model, and then finely filtering the target path set, includes: The coarse filtering path set and the natural language question information are concatenated to construct the fine filtering reasoning instruction corresponding to each coarse filtering path in the coarse filtering path set; The fine filtering inference instruction is input into the large language model, and the large language model is used to perform semantic understanding on the fine filtering inference instruction. The semantic score value corresponding to each coarse filtering path in the coarse filtering path set is evaluated to assess the semantic similarity between each coarse filtering path and the corresponding natural language question information. The semantic score value corresponding to each coarse filtering path is compared with a preset semantic score threshold to obtain the fine filtering comparison result corresponding to each coarse filtering path. The coarse-filtered paths whose semantic score is less than the preset semantic score threshold are removed from the coarse-filtered path set to obtain the target path set output by the large language model, and then the target path set is fine-filtered.

5. The method according to claim 1, characterized in that, The step of constructing the structured prompting instructions corresponding to the natural language question information based on the coarse-filtered path set and the target path set includes: The target path set is determined as the priority path set, and the difference between the coarse-filtered path set and the target path set is calculated to obtain the additional reference path set; Each priority path in the priority path set is placed in the first priority position corresponding to the structure prompt instruction corresponding to the natural language question information, and each additional reference path in the additional reference path set is placed in the second priority position corresponding to the structure prompt instruction, so as to construct the structure prompt instruction.

6. The method according to claim 1, characterized in that, The calculation of the confidence scores corresponding to the first and second answer results, and the filtering and fusion of the first and second answer results according to the confidence scores to generate the target answer result corresponding to the natural language question information, includes: The first confidence score corresponding to the first answer result is calculated using the large language model, and the second confidence score corresponding to the second answer result is calculated. The first confidence value is filtered using a first confidence threshold to obtain a first target answer set corresponding to the knowledge graph retrieval system, and the second confidence value is filtered using a second confidence threshold to obtain a second target answer set corresponding to the large language model. The first target answer set and the second target answer set are fused using a preset fusion mechanism based on external instructions to generate the target answer result corresponding to the natural language question information; wherein, the preset fusion mechanism includes at least one of confidence priority, fusion reasoning score ranking, voting mechanism and merge set ranking.

7. The method according to claim 6, characterized in that, The step of calculating the confidence scores corresponding to the first answer result and the second answer result, and filtering and fusing the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information, further includes: In response to the inconsistency between the first answer result and the second answer result, a verification instruction is generated and input into the knowledge graph retrieval system and the large language model respectively; The knowledge graph retrieval system and the large language model perform self-evaluation respectively to determine the target answer result in the first answer result and the second answer result.

8. A question-answering device based on knowledge graph retrieval and a large language model, characterized in that, The question-and-answer device includes: The path acquisition module is used to, in response to receiving natural language question information input by the user, use a preset knowledge graph retrieval system to obtain a set of candidate paths corresponding to the natural language question information; The path filtering module is used to perform coarse filtering and fine filtering on the candidate path set in sequence using a preset large language model, so as to obtain the coarse-filtered path set output by the large language model through coarse filtering and the target path set output by the large language model through fine filtering. The answer reasoning module is used to construct a structural prompt instruction corresponding to the natural language question information based on the coarse filter path set and the target path set, and input the structural prompt instruction into the knowledge graph retrieval system and the large language model respectively to obtain the first answer result output by the knowledge graph retrieval system and the second answer result output by the large language model respectively. The answer fusion module is used to calculate the confidence scores corresponding to the first answer result and the second answer result, respectively, and to filter and fuse the first answer result and the second answer result according to the confidence scores to generate the target answer result corresponding to the natural language question information; When the path filtering module is used to sequentially perform coarse filtering and fine filtering on the candidate path set using a preset large language model, corresponding to obtaining a coarsely filtered path set output by the large language model through coarse filtering and a target path set output by the large language model through fine filtering, the path filtering module is used to: A pre-defined large language model is used to calculate the attention value corresponding to each candidate path in the candidate path set, representing the degree of attention the candidate path pays to the reasoning of the large language model. The attention value is then filtered using a pre-defined attention threshold to obtain a coarsely filtered path set output by the large language model, which is used to coarsely filter the candidate path set. Specifically, the attention weights calculated for each candidate path's token by the intermediate layer in the Transformer layer of the large language model are obtained, and the average attention calculated for each candidate path's token by the large language model is used as the attention value of each candidate path. The semantic score value corresponding to each coarsely filtered path in the coarsely filtered path set is evaluated using the large language model, and the semantic score value is filtered using a preset semantic score threshold to obtain the target path set output by the large language model, so as to perform fine filtering on the target path set.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the question-answering method based on knowledge graph retrieval and large language model as described in any one of claims 1 to 6.

Images