Knowledge retrieval generation-based open domain question and answer method and system

Abstract

The invention discloses an open domain question and answer method and system based on knowledge retrieval generation, and the system comprises an evidence document generation module, a candidate entity generation module, a candidate subgraph verification module and a candidate subgraph sorting module. The method comprises the following steps: retrieving evidence fragments from a core entity, guiding LLM to generate a more complete evidence document by using the retrieved evidence fragments, generating candidate entities by adopting evidence documents associated with questions, promoting LLM to accurately extract a candidate entity set, verifying candidate sub-graphs, screening a candidate sub-graph set from the core entity to the existence of all candidate entity sub-graphs, and verifying the candidate sub-graphs. The candidate sub-graph ranking uses a large language model to filter and rank all candidate sub-graphs, and selects an entity associated with the highest ranked sub-graph as a final answer. According to the method, the answer is not generated only according to the question text, the question and the candidate answer document are subjected to entity extraction, and the question answer is generated in the knowledge graph through the large model language, so that the question answer is ensured to be more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an open domain question answering method and system based on knowledge retrieval generation. Background Art

[0002] In recent years, with the rapid development of deep learning technology, the field of natural language processing has undergone earth-shaking changes, among which the rapid development of large language models is particularly eye-catching. It has achieved unprecedented achievements in multiple language tasks and greatly promoted the application prospects of artificial intelligence technology in the field of language understanding and generation. Open domain question answering is a natural language processing task that aims to automatically extract answers from large unstructured text data (such as Wikipedia, etc.) to answer any questions raised by users. Unlike traditional question answering systems, open domain question answering systems are not limited to specific fields, but can handle questions on various topics in the open domain question answering research field. Previous studies have widely used document retrieval methods such as neural network ranking models to screen and organize documents related to queries from large-scale corpora. Recently, a dense channel retrieval method has also been proposed, which realizes efficient retrieval of dense vectors of text segments through a dual encoder architecture, greatly improving the accuracy of retrieval, and then accurately extracts answers through the reader module, effectively making up for the shortcomings of traditional retrieval methods.

[0003] Nevertheless, the current task still faces many major challenges, especially when dealing with situations that require access to large datasets or domain-specific knowledge bases. Most of the current open-domain question answering methods for external retrieval enhancement adopt a retrieval-reading pipeline architecture, first retrieving relevant context documents from external sources (such as Wikipedia), and then generating predicted answers based on the content of these documents and the questions asked. This approach faces two major challenges.

[0004] The first is the problem of integrating heterogeneous information. Retrieved documents are usually chunked and fixed, which may result in noisy and irrelevant information. Fixed-size document chunks may not fully capture the context necessary to find accurate answers. Therefore, the presence of irrelevant information in retrieved documents may lead to noise, and there are problems with accuracy and correctness in integrating with structured information, which negatively affects the quality and relevance of the generated answers.

[0005] The second is the accurate alignment of the granularity of structural information and the granularity of answers. Due to the need to efficiently process large corpora, the limitations of retriever model parameters and embedding size restrict the model from fully leveraging the world knowledge and reasoning capabilities of large language models. As a result, the retriever model may have difficulty capturing the rich semantic and contextual information necessary to generate accurate answers.

[0006] Therefore, the present invention provides an open-domain question answering method and system based on knowledge retrieval generation, which combines document retrieval technology with a large language model to provide rich and context-relevant candidate answer information, and uses question-related entities and relationships in the knowledge graph as additional structural information to accurately guide answers to factual questions. Summary of the invention

[0007] Based on the above problems, the present invention provides an open domain question answering method and system based on knowledge retrieval generation. The user inputs a question, uses intensive retrieval to retrieve a group of relevant documents from a corpus, guides the LLM to generate a more complete evidence document, and uses the evidence document associated with the question to prompt the LLM to accurately extract the candidate entity set, screen out candidate subgraphs from the core entity to all candidate entities in the knowledge graph, use a large language model to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final output.

[0008] The solution adopted by the present invention to solve the technical problem is: 1. An open domain question answering method based on knowledge retrieval generation, comprising:

[0009] S1. Based on a given question, use intensive retrieval to retrieve a set of relevant evidence documents from the corpus, and use the retrieved evidence document fragments to guide LLM to generate more complete candidate evidence documents;

[0010] S2, using candidate evidence documents associated with the question to enable LLM to accurately extract candidate entity sets;

[0011] S3, screening candidate reasoning subgraphs in the knowledge graph;

[0012] S4, a large language model is used to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final answer output.

[0013] The steps of using dense retrieval in S1 to retrieve a set of relevant documents from the corpus include:

[0014] S11. Given a question and a corpus;

[0015] S12, Format Search Query, insert the question into a specifically formatted search query to ensure that Serpapi searches only for content in the given corpus;

[0016] S13, get the evidence document, pass the formatted query string to Serpapi, and Serpapi returns the top 10 documents most relevant to the query;

[0017] S14. Obtain new candidate evidence documents. The search results returned by Serpapi are the top 10 document sets most relevant to the question. Based on these 10 document sets, a large language model is used to generate a set of new candidate evidence documents through input prompts. The new candidate evidence documents include potential entities.

[0018] Furthermore, a large language model is used to generate a set of new candidate evidence documents by inputting prompt words, including:

[0019] Use a large language model to generate evidence documents corresponding to multiple entities with the same name, as shown in the following formula:

[0020] Doc q =LLM(prompt ged , q)

[0021] For a given question q and the retrieved evidence documents Generate a new set of candidate evidence documents Doc using a large language model from the input prompt q .

[0022] Further, the S2 includes:

[0023] S21: Concatenate question and evidence documents to jointly prompt a large language model to generate candidate entity sets

[0024] ε cand , as shown below:

[0025] ε cand =LLM(prompt gec ,Doc q , q)

[0026] Where q is the given question, Doc q It is a candidate evidence document, prompt gec It is a prompt template;

[0027] S22: Adopt the following entity linking strategy: Identify the topic entity e of question q m ,The topic entity is the entity that is directly related to the question;

[0028] S23: Get the subject entity e m The set of all entities within two hops distance in the knowledge base ε neighbor , as shown below:

[0029] ε neighbor ={e|dist(e, e m )≤2}

[0030] Among them, dist(e, e m) represents the distance between entity e and the subject entity. A distance less than or equal to 2 indicates that these entities are connected to the subject entity in the knowledge base through at most two relationships.

[0031] Furthermore, the entity linking strategy described in S22 specifically includes:

[0032] In the entity linking process, a soft matching strategy is adopted, and the Transforms-based model RoBERTa is used to encode and transform the entity set ε related to the two-hop neighbor All entity names in the , get the feature representation of each entity name, as shown in the following formula:

[0033] V cand ={v|v=RoBERTa(e), e∈ε cand}

[0034] Construct an entity index using Faiss to perform fast similarity searches in the latent space.

[0035] Search for the most similar entity. In the latent space, a search is performed to find the entity that is most similar to the provided entity representation; as shown in the following formula:

[0036] ε cand ={e|sim(V cand , V neighbor )>θ c}

[0037] The similarity calculation formula is:

[0038]

[0039] Among them, V cand is the feature vector set of candidate entities, v = RoBERTa(e) means that each entity name e is encoded into a vector v, V neighbor Represents the set of feature vectors of neighbor entities, θ c is the similarity threshold, similarity threshold θ c Set to 0.7.

[0040] Further, the S3 screening of candidate reasoning subgraphs includes the following steps:

[0041] S31: Input knowledge graph, subject entity and candidate entity set: The knowledge graph is a widely used and proven knowledge graph containing a large number of entities and relations for a variety of natural language processing and information retrieval tasks;

[0042] S32: Initialize an empty set: initialize an empty candidate reasoning subgraph set;

[0043] S33: traverse the candidate entity set, iterate each candidate entity in the candidate entity set, and determine whether the candidate entity exists in the knowledge graph;

[0044] S34: If there is a path, obtain the triples in the path, the triples describe the path from the subject entity to the candidate entity, and add the triples to the candidate reasoning subgraph set.

[0045] Furthermore, the process by which S4 uses a large language model to output the final answer includes:

[0046] S41: Construct hints: Take the question and candidate reasoning subgraphs as input, and construct a hint template to rank the candidate reasoning subgraphs using LLM. The hint template also considers the effective information provided by the auxiliary subgraphs.

[0047] S42: LLM sorting: LLM will obtain a sorted list of candidate subgraphs based on the input prompt template, taking into account the relevance of each subgraph and the information provided by the auxiliary subgraphs;

[0048] S43: Select the entity associated with the highest ranked subgraph as the final result.

[0049] A system applicable to the above-mentioned open domain question answering method based on knowledge retrieval generation, comprising an evidence document generation module, a candidate entity generation module, a candidate subgraph verification module and a candidate subgraph sorting module;

[0050] Among them, the evidence document generation module is used to retrieve a set of relevant documents from the corpus using intensive retrieval, and use the retrieved evidence fragments to guide the LLM to generate a more complete evidence document;

[0051] Candidate entity generation module: used to accurately extract candidate entity sets;

[0052] Candidate subgraph verification module: used to screen candidate reasoning subgraphs in the knowledge graph;

[0053] Candidate subgraph ranking module: It is used by the large language model to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final answer output.

[0054] Beneficial effects of the present invention: 1. The present invention proposes an open domain question answering method based on knowledge retrieval generation, which combines document retrieval technology with a large language model to provide rich and context-related candidate answer information. Furthermore, the entities and relationships related to the question in the knowledge graph are used as additional structural information to accurately guide the answering of factual questions.

[0055] 2. Compared with the traditional open domain question answering method, the present invention is more reasoning and authentic. Candidate entity generation uses the obtained evidence documents and uses LLM to extract and supplement candidate answer entities to construct a series of candidate entity sets. Candidate subgraph verification and sorting first verify the existence of the subgraph from the central entity to all candidate entities. This process guarantees the validity of the candidate entities to a certain extent and reveals a more complete relationship chain. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic diagram of an open domain question answering method based on knowledge retrieval generation according to an embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the process of the evidence document generation module of an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of a candidate entity generation module flow in an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of the candidate subgraph verification and sorting module flow in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below.

[0061] Example: See Figure 1 The present invention provides an embodiment of an open domain answer method based on knowledge retrieval generation, which comprises the following steps:

[0062] S1: Based on a given question, the evidence document generation module uses intensive retrieval to retrieve a set of relevant documents from the corpus, and uses the retrieved evidence fragments to guide the LLM to generate a more complete evidence document.

[0063] See also Figure 2 ,Evidence document generation flow chart.

[0064] S11: Given a question q and a Wikipedia corpus Doc wiki ={d1,d2…d M}, which is the set of all possible Wikipedia documents.

[0065] S12: In this embodiment, the user may give a question q: "Who plays bob kelso in scrubs?", format the search query, and insert the question q into a search query in a specific format to ensure that Serpapi only searches the content of Wikipedia, and the formatted query is "en.wikipedia.org:q".

[0066] S13: Get evidence documents. Pass the formatted query string to Serpapi, asking it to return the top 10 Wikipedia documents most relevant to the query.

[0067] S14: Get new candidate evidence documents. The search results returned by Serpapi are the top 10 Wikipedia documents most relevant to the question. Based on this generation result, a new set of candidate evidence documents Doc is generated through input prompts based on the large language model. q , which includes potential entities.

[0068] Constructing prompt statement: prompt ged : "Generate a more accurate and comprehensive set of evidence documents based on the provided evidence documents for answering the question.question evidence documents."

[0069] A large language model is used to generate evidence documents corresponding to multiple entities with the same name. The evidence documents are classified and supplemented according to the large model. As shown in the following formula:

[0070] Doc q =LLM(prompt ged , q)

[0071] S2: Candidate entity generation uses evidence documents associated with the question to enable LLM to accurately extract the candidate entity set.

[0072] See also Figure 3 ,Candidate entity generation flow chart.

[0073] S21: Question q and candidate document Doc q Concatenate co-prompts to a large language model to generate candidate entity sets ε cand , as shown below:

[0074] ε cand =LLM(prompt gec ,Doc q , q)

[0075] The prompt template is as follows:

[0076] prompt gec: "Please answer the current question from different dimensions, perspectives, or granularities based on the supporting evidencedocument. The answer should be an entity from Freebase, not a vague statement."

[0077] S21: Continue with the evidence document obtained in S1 and build a prompt template gec : "Please answer the current question from different dimensions, perspectives, orgramularities based on the supporting evidence document. The answer should bean entity from Freebase, not a vague statement.". Document question q and candidate documents q Concatenate co-prompts to a large language model to generate candidate entity sets ε cand , as shown below:

[0078] ε cand =LLM(prompt gec ,Doc q , q)

[0079] S22: Adopt the following entity linking strategy. First, identify the subject entity e of question q. m ('bob kelso'), the topic entity is the entity directly related to the question.

[0080] S23: Get the subject entity e m The set of all entities within two hops distance in the knowledge base ε neighbor , as shown below:

[0081] ε neighbor ={e|dist(e, e m )≤2}

[0082] Among them, dist(e, e m ) represents the distance between entity e and the subject entity. A distance less than or equal to 2 indicates that these entities are connected to the subject entity in the knowledge base through at most two relationships.

[0083] In the entity linking process, a soft matching strategy is adopted, and the Transforms-based model RoBERTa is used to encode and transform the entity set ε related to the two-hop neighbor All entity names in the , get the feature representation of each entity name, as shown in the following formula:

[0084] V cand ={v|v=RoBERTa(e), e∈ε cand}

[0085] Construct an entity index using Faiss to perform fast similarity searches in the latent space.

[0086] Search for the most similar entity,In the latent space, a search is performed to find the entity that is most similar to the provided entity representation. The similarity threshold θ c Set it to 0.7, as shown in the following formula:

[0087] ε cand ={e|sim(V cand , V neighbor )>θ c}

[0088] The similarity calculation formula is:

[0089]

[0090] S2 finally obtains the candidate entity set for question q, and in the case of question q, it obtains ['Ken Jenkins', 'Zach Braff', 'American actor'], etc.

[0091] S3: Screening candidate reasoning subgraphs in the knowledge graph.

[0092] See also Figure 4 ,Verification process in candidate subgraph verification and ranking.

[0093] S31: Input knowledge graph G, subject entity e m , entity candidate set ε cand ,The knowledge graph G is a widely used and verified knowledge graph that contains a large number of entities and relations, and can be used for a variety of natural language processing and information retrieval tasks. ,In this embodiment, the knowledge graph G uses Wikidata.

[0094] S32: Initialize an empty set, initialize an empty candidate reasoning subgraph set

[0095] S33: Traverse the candidate entity set and perform cand Each candidate entity in Iterate and judge candidate entities Whether it exists in the knowledge graph G.

[0096] S34; if there is a path, obtain the triples (relationships) in the path, which describe the relationship between the subject entity e m To candidate entity Path, add these triples to the set of candidate reasoning subgraphs middle.

[0097] S4: A large language model is used to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final answer output.

[0098] S41: Build prompts, taking question q and candidate reasoning subgraphs as input, and build prompts csvr To use LLM to rank candidate reasoning subgraphs, the hint also takes into account the effective information provided by the auxiliary subgraphs. The hint template is as follows:

[0099] prompt csvr :"Please sort the reasoning subgraphs according to theirrelevance to the"Who plays bob kelso in scrubs? "."

[0100] S42: LLM sorting, LLM will consider the relevance of each subgraph and the information provided by the auxiliary subgraph according to the input prompt, and obtain a sorted candidate subgraph list.

[0101] S43: The entity associated with the highest ranked subgraph ('Ken Jenkins') is selected as the final result.

[0102] The present invention provides an embodiment of an open domain question answering system based on knowledge retrieval generation applicable to the above embodiment, which comprises:

[0103] The evidence document generation module is used to retrieve a set of relevant documents from the corpus using intensive retrieval and use the retrieved evidence fragments to guide the LLM to generate a more complete evidence document.

[0104] The candidate entity generation module is used to accurately extract the candidate entity set.

[0105] The candidate subgraph verification module is used to screen out candidate reasoning subgraphs in the knowledge graph G.

[0106] The candidate subgraph ranking module is used by the large language model to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final answer output.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An open domain question answering method based on knowledge retrieval generation, characterized in that: include: S1. Based on a given question, use intensive retrieval to retrieve a set of relevant evidence documents from the corpus, and use the retrieved evidence document fragments to guide LLM to generate more complete candidate evidence documents; S2, using candidate evidence documents associated with the question to enable LLM to accurately extract candidate entity sets; S3, screening candidate reasoning subgraphs in the knowledge graph; S4, a large language model is used to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final answer output.

2. The open domain question answering method according to claim 1, characterized in that: The steps in S1 to retrieve a set of relevant documents from the corpus using dense retrieval include: S11. Given a question and a corpus; S12, Format Search Query, insert the question into a specifically formatted search query to ensure that Serpapi searches only for content in the given corpus; S13, get the evidence document, pass the formatted query string to Serpapi, and Serpapi returns the top 10 documents most relevant to the query; S14. Obtain new candidate evidence documents. The search results returned by Serpapi are the top 10 document sets most relevant to the question. Based on these 10 document sets, a large language model is used to generate a set of new candidate evidence documents through input prompts. The new candidate evidence documents include potential entities.

3. The open domain question answering method according to claim 2, characterized in that: Using a large language model to generate a set of new candidate evidence documents by inputting a prompt word includes: using a large language model to generate evidence documents corresponding to multiple entities with the same name, as shown in the following formula: Doc q =LLM(prompt ged ,q) For a given question q and the retrieved evidence documents Generate a new set of candidate evidence documents Doc using a large language model from the input prompt q .

4. The open domain question answering method according to claim 1, characterized in that: The S2 includes: S21: Concatenate question and evidence documents to jointly prompt a large language model to generate a candidate entity set ε cand , as shown below: ε cand =LLM(prompt gec ,Doc q ,q) Where q is the given question, Doc q It is a candidate evidence document, prompt gec It is a prompt template; S22: Adopt the following entity linking strategy: Identify the subject entity e of question q m ,The topic entity is the entity that is directly related to the question; S23: Get the subject entity e m The set of all entities within two hops distance in the knowledge base ε neighbor , as shown below: ε neighbor {and|dist(and,and m )≤2} Among them, dist(e,e m ) represents the distance between entity e and the subject entity. A distance less than or equal to 2 indicates that these entities are connected to the subject entity in the knowledge base through at most two relationships.

5. The open domain question answering method according to claim 4, characterized in that: The entity linking strategy described in S22 specifically includes: In the entity linking process, a soft matching strategy is adopted, and the Transforms-based model RoBERTa is used to encode and transform the entity set ε related to the two-hop neighbor All entity names in the , get the feature representation of each entity name, as shown in the following formula: V cand ={x|v=RoBERTa(e),e∈ε cand } Construct an entity index using Faiss to perform fast similarity searches in the latent space. Search for the most similar entity. In the latent space, a search is performed to find the entity that is most similar to the provided entity representation; as shown in the following formula: ε cand ={and|yes(V cand ,V neighbor )>θ c } The similarity calculation formula is: Among them, V cand is the feature vector set of candidate entities, v = RoBERTa(e) means that each entity name e is encoded into a vector v, V neighbor Represents the set of feature vectors of neighbor entities, θ c is the similarity threshold.

6. The open domain question answering method according to claim 5, characterized in that: Similarity threshold θ c Set to 0.

7.

7. The open domain question answering method according to claim 1, characterized in that: The S3 screening of candidate inference subgraphs includes the following steps: S31: Input knowledge graph, subject entity and candidate entity set: The knowledge graph is a widely used and proven knowledge graph containing a large number of entities and relations for a variety of natural language processing and information retrieval tasks; S32: Initialize an empty set: initialize an empty candidate reasoning subgraph set; S33: traverse the candidate entity set, iterate each candidate entity in the candidate entity set, and determine whether the candidate entity exists in the knowledge graph; S34: If there is a path, obtain the triples in the path, the triples describe the path from the subject entity to the candidate entity, and add the triples to the candidate reasoning subgraph set.

8. The open domain question answering method according to claim 1, characterized in that: The process by which S4 uses a large language model to output the final answer includes: S41: Construct hints: Take the question and candidate reasoning subgraphs as input, and construct a hint template to rank the candidate reasoning subgraphs using LLM. The hint template also considers the effective information provided by the auxiliary subgraphs. S42: LLM sorting: LLM will consider the relevance of each subgraph and the information provided by the auxiliary subgraph according to the input prompt template, and obtain a sorted list of candidate subgraphs; S43: Select the entity associated with the highest ranked subgraph as the final result.

9. A system applicable to the above open domain question answering method based on knowledge retrieval generation, characterized in that: It includes an evidence document generation module, a candidate entity generation module, a candidate subgraph verification module and a candidate subgraph sorting module; Among them, the evidence document generation module is used to retrieve a set of relevant documents from the corpus using intensive retrieval, and use the retrieved evidence fragments to guide LLM to generate a more complete evidence document; the candidate entity generation module is used to accurately extract the candidate entity set; Candidate subgraph verification module: used to screen candidate reasoning subgraphs in the knowledge graph; Candidate subgraph ranking module: It is used by the large language model to filter and rank all existing candidate subgraphs, and select the entity associated with the highest-ranked subgraph as the final answer output.

Images