A query expansion optimization method and query method in an open-domain question answering task

Abstract

The application discloses a query expansion optimization method and a query method in an open domain question answering task. The query expansion optimization method adopts a three-step progressive prompting strategy of analysis, generation and refinement, utilizes the multi-task capability of LLMs to promote the generation of open domain query expansion; 1) the analysis stage utilizes the semantic analysis capability of LLMs to generate analysis; 2) the generation stage utilizes the knowledge retrieval and integration capability of LLMs to generate diverse answer-oriented query expansions as candidates for retrieving reference texts, to filter out incorrect and irrelevant generated information, and to generate new candidates closer to potential answers; 3) in order to control the quality of query expansion, the refinement stage utilizes the context understanding and reasoning capability of LLMs to self-examine all candidates to generate optimized query expansions. The application can effectively generate high-quality query expansion in cross-domain scenarios, ensuring the accuracy and relevance of information retrieval.

Description

Technical Field

[0001] This invention relates to a query expansion and optimization method and a query method for open-domain question answering (OpenQA) tasks based on large language models (LLMs), specifically designed for open-domain question answering systems, and belongs to the field of natural language processing technology. Background Technology

[0002] Open-domain question answering (OpenQA), a core task in natural language processing, aims to provide accurate answers to factual questions across domains. The challenge lies in retrieving question-related information from large-scale text corpora, especially when specific context is lacking. Therefore, effective retrieval methods are crucial for OpenQA systems, primarily falling into two categories: lexical-based sparse retrieval and embedding-based dense retrieval. While dense retrieval models perform well with ample domain-specific training data, they are computationally demanding, and their reliance on fixed-length embeddings can lead to the omission of key information and relevant details. Conversely, sparse retrieval combined with query expansion techniques can address these semantic challenges and achieve competitive performance.

[0003] In recent years, the rise of generative augmented retrieval has provided OpenQA with a more accurate approach to information retrieval by using sequence-to-sequence models like BART to generate query-related context, including elements such as the answer, the sentence containing the answer, and the title of the channel containing the answer. However, these methods typically require large amounts of supervised training data and may generate poor-quality query expansions (QEs) when faced with zero-shot out-of-domain scenarios.

[0004] Meanwhile, the emergence of large-scale language models (LLMs) such as GPT-3 and Flan-T5 has demonstrated their potential as effective query expansion tools in information retrieval (IR) tasks, operating without the need for training data or external corpora. While these models excel at deriving query expansions directly from LLMs, they often lack mechanisms to ensure the quality of the generated query expansions. Summary of the Invention

[0005] To more effectively address the application of large-model-based query expansion methods in open-domain question answering, this invention provides a query expansion optimization method and query method based on a large-scale language model in open-domain question answering tasks.

[0006] The query expansion optimization method of this invention adopts a three-step progressive suggestion strategy of analysis, generation, and refinement, leveraging the multi-tasking capabilities of LLMs to facilitate the generation of open-domain query expansions. (I) The analysis stage utilizes the semantic analysis capabilities of LLMs to generate analysis; (II) The generation stage utilizes the knowledge retrieval and integration capabilities of LLMs to generate diverse answer-oriented query expansions as candidates. These expansions are then used to retrieve reference texts to filter out erroneous and irrelevant generated information, generating new candidates that are closer to the potential answers. (III) To perform quality control on the query expansions, the refinement stage utilizes the contextual understanding and reasoning capabilities of LLMs to self-examine all candidates, refining them to generate optimized query expansions.

[0007] The technical solution adopted in this invention is summarized as follows:

[0008] A query expansion optimization method for open-domain question answering tasks includes the following steps:

[0009] 1) A prompt template aimed at extracting keywords guides a large language model (LLMs) to extract key phrases from the original query q, resulting in a set of key phrases K; based on the set of key phrases K, the prompt template aimed at analyzing the query of the large language model (LLMs) is filled, and the filled prompt template is input into the large language model (LLMs) to perform question analysis on the original query q, obtaining the analysis text A;

[0010] 2) A prompt template aimed at generating potential answers as query expansions guides the large language model (LLMs) to generate multiple query expansions Q based on the analyzed text A, with the goal of generating potential answers as query expansions. e ={qe1,qe2,…,qe n}, qe n For the nth query expansion oriented towards query answer; then for each of the aforementioned query expansions qe i To perform a recall, multiple documents corresponding to the corpus are retrieved as the query extension qe. i Reference document collection D qei The context reference set CR is obtained by summarizing the reference document sets of each query expansion. s Then the context reference set CR s The large language model LLMs are input along with the original query q, and guided by a prompt template aimed at generating potential answers by fusing a set of reference documents as query expansions, the large language model LLMs generate multiple new query expansions Q that are query answer-oriented. e ={qe1,qe2,…,qe n};qe nExpand the nth candidate query that is query answer oriented;

[0011] 3) Using a prompt template aimed at eliminating irrelevant or erroneous information in the candidate query expansion set, leveraging the contextual understanding and reasoning capabilities of the large language models (LLMs), the candidate query expansion Q is optimized. e ={qe1,qe2,…,qe n The query was reviewed to obtain an optimized query extension, qe. o Then, it is appended to the original query q to obtain the query expansion optimized from the original query q.

[0012] Furthermore, the BM25 sparse searcher is used to perform retrieval recall from the corpus based on each query expansion.

[0013] Furthermore, the large-scale language model (LLMs) generates multiple query expansions Q based on the analyzed text A, each guided by the query answer. e ={qe1,qe2,…,qe n The sampling strategy used at that time is based on the context reference set CR. s Generate multiple new query expansions Q with the original query q, each guided by the query answer. e ={qe1,qe2,…,qe n The sampling strategy used is the same.

[0014] Furthermore, the large-scale language models (LLMs) are dialogue models with more than 7 parameters.

[0015] A query method is characterized by generating an optimized query extension of the original query q according to the above method and inputting it into the large language model LLMs, performing a query in the corpus, and obtaining the query result of the original query q.

[0016] This invention provides a query expansion generation method based on LLMs specifically designed for OpenQA tasks. It employs a three-step progressive suggestion strategy of analysis, generation, and refinement, leveraging the multi-tasking capabilities of LLMs to facilitate the generation of open-domain query expansions. It mainly includes the following parts:

[0017] 1) Analysis steps: Use a prompt aimed at extracting keywords to mine the semantic analysis capabilities of LLMs to parse the original query and extract key phrase information; further, based on the keyword information, use a prompt aimed at analyzing the query to conduct in-depth problem analysis, generate analysis text related to the original query, and improve the accuracy and relevance of subsequent query expansions.

[0018] 2) Generation steps: Based on the analysis information generated in the first step, LLMs are guided to randomly sample and generate a series of query expansions oriented towards query answers through a prompt aimed at generating potential answers as query expansions; then, the BM25 sparse retrieval tool is used to recall all query expansions, and the top k documents corresponding to the relevant corpus are obtained as a reference document set. Then, the prompt aimed at generating potential answers as query expansions by merging the reference document set is used to guide LLMs to correct and optimize the query expansions, thereby generating new query expansions and saving them to the query expansion candidate set;

[0019] 3) Refinement Steps: The query expansion candidate set is given a final review. The prompt, which aims to eliminate irrelevant or erroneous information in the query expansion candidate set, is used to explore the contextual understanding and reasoning capabilities of LLMs to perform self-examination of the query expansion candidate set, eliminating irrelevant content or erroneous information to obtain the final optimized query expansion, which is then attached to the original query for retrieval.

[0020] Furthermore, the large language model involved in Technology 1) is a dialogue model with more than 7B (7 billion) parameters, which is suitable for the currently popular large market models, ensuring the performance of its multi-tasking capabilities and the effectiveness of the prompting framework.

[0021] Furthermore, in technique 2), a random sampling strategy is employed in both query expansion generation steps to ensure that LLMs generate a range of potential expansions, each reflecting a different aspect and interpretation of the original query. The generated query expansions provide a comprehensive candidate set for subsequent refining steps.

[0022] Furthermore, the recall algorithm involved in Technique 2) employs the BM25 sparse searcher, which improves the efficiency of retrieval and recall without relying on a fixed-length embedding vector.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] 1) Strong cross-domain applicability: This invention adopts a query expansion (QE) optimization method based on large language models (LLMs), specifically designed for open-domain question answering (OpenQA) tasks. Through a three-step progressive strategy of analysis, generation, and refinement, it can effectively generate high-quality query expansion in cross-domain scenarios, especially demonstrating superior performance in zero-shot scenarios, thus overcoming the shortcomings of existing technologies in terms of applicability and flexibility in new domains.

[0025] 2) Improve information retrieval quality: By comprehensively utilizing the multi-tasking capabilities of LLMs, this invention can not only more accurately capture the information needs of the query when generating query expansions, but also effectively filter and optimize query expansions through the refining stage, ensuring the accuracy and relevance of information retrieval.

[0026] 3) Practicality and Universality: This invention emphasizes the practicality and universality of using existing large-scale language models for query expansion generation, without relying on large amounts of training data or external corpora. This approach not only lowers the barrier to implementing OpenQA systems but also improves the system's flexibility and accuracy in handling various queries encountered in real-world applications. Attached Figure Description

[0027] Figure 1 This is a flowchart of a query expansion generation method based on a large language model applied to open-domain question answering.

[0028] Figure 2 This is a schematic diagram of the model prompts involved in each of the main steps of the method. Detailed Implementation

[0029] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below.

[0030] This invention comprises three key parts:

[0031] 1) Analysis Phase: In the initial phase, this invention fully leverages the superior semantic understanding capabilities of Large Language Models (LLMs) to extract key phrases K from the original query q using a prompt aimed at extracting keywords. This process not only relies on the LLMs' sensitive ability to capture linguistic details but also demonstrates its advanced cognitive functions in understanding complex queries. By accurately identifying and extracting key elements from the query, a refined and representative set of key phrases K is obtained.

[0032] Following this, based on these key phrases K, the LLMs are guided by prompts aimed at analyzing the query to perform question analysis on q, yielding analysis text A. This analysis process, through the advanced semantic processing capabilities of LLMs, provides a comprehensive and in-depth interpretation of the topic, contextual clues, and potential information needs within the query. Such analysis provides a solid theoretical and semantic foundation for subsequent query expansion generation, ensuring that the generated expansions are not only closely related to the original query but also accurately cover the user's information needs.

[0033] 2) Generation Phase: Based on the initial analysis A, a prompt aimed at generating potential answers as query expansion guides LLMs to generate multiple candidate query expansions Q based on their own knowledge retrieval and integration capabilities. e ={qe1,qe2,…,qe n The core of this step lies in generating diverse expansion options that cover the potential information needs of the query by meticulously mining and understanding the deeper meaning of the original query q. To further ensure the accuracy and relevance of these expansions, we introduce a crucial correction step, using relevant documents in the corpus to correct and optimize candidate expansions.

[0034] Specifically, for each candidate extension, qe for each candidate extension i Collect Topk documents D recalled by the BM25 sparse search engine qei These documents, when compiled, form a rich set of contextual references (CR). s Then, refer to the context CR. s The original query q is re-input into LLMs, and a second round of sampling is performed using a prompt aimed at generating potential answers by fusing a set of reference documents as query expansion. This process combines the inherent knowledge of the LLMs with information retrieved from the corpus, resulting in a richer candidate query expansion Q. e ={qe1,qe2,…,qe n}

[0035] In both stages of generating the query expansion candidate set, a random sampling strategy was employed to ensure that the LLM could produce a series of potential expansion schemes, each reflecting multiple interpretations and meanings of the original query from different perspectives. This diversity not only enriched the dimensions of the candidate expansions but also provided a comprehensive and in-depth pool of alternatives for subsequent refinement steps, ensuring that the final query expansion accurately matches the user's actual information needs.

[0036] 3) Refinement Stage: In this stage, the method leverages the contextual understanding and reasoning capabilities of LLMs, using a prompt aimed at eliminating irrelevant or erroneous information from the candidate query expansion set for all candidate query expansions Q. e ={qe1,qe2,…,qe n A final review is then conducted. This process involves distinguishing between valuable and redundant information. In other words, we refine and distill these candidate query extensions. e ={qe1,qe2,…,qe n This yields a final optimized query extension, qe. o Then, it is appended to the original query q, thereby improving the quality of sparse search results.

[0037] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the principles and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A query expansion optimization method for open-domain question answering tasks, comprising the following steps: 1) A prompt template aimed at extracting keywords guides a large language model (LLMs) to extract key phrases from the original query q, resulting in a set of key phrases K; based on the set of key phrases K, the prompt template aimed at analyzing the query of the large language model (LLMs) is filled, and the filled prompt template is input into the large language model (LLMs) to perform question analysis on the original query q, obtaining the analysis text A; 2) A prompt template aimed at generating potential answers as query expansions guides the large language model (LLMs) to generate multiple query expansions Q based on the analyzed text A, with the goal of generating potential answers as query expansions. e ={qe1,qe2,…,qe n }, qe n For the nth query expansion oriented towards query answer; then for each of the aforementioned query expansions qe i To perform a recall, multiple documents corresponding to the corpus are retrieved as the query extension qe. i Reference document collection D qei The context reference set CR is obtained by summarizing the reference document sets of each query expansion. s Then the context reference set CR s The large language model LLMs are input along with the original query q, and guided by a prompt template aimed at generating potential answers by fusing a set of reference documents as query expansions, the large language model LLMs generate multiple new query expansions Q that are query answer-oriented. e ={qe1,qe2,…,qe n };qe n Expand the nth candidate query that is query answer oriented; 3) Using a prompt template aimed at eliminating irrelevant or erroneous information in the candidate query expansion set, leveraging the contextual understanding and reasoning capabilities of the large language models (LLMs), the candidate query expansion Q is optimized. e ={qe1,qe2,…,qe n The query was reviewed to obtain an optimized query extension, qe. o Then, it is appended to the original query q to obtain the query expansion optimized from the original query q.

2. The method according to claim 1, characterized in that, The BM25 sparse searcher is used to perform retrieval recall from the corpus based on each query expansion.

3. The method according to claim 1, characterized in that, The large language model LLMs generates multiple query expansions Q based on the analyzed text A, each guided by the query answer. e ={qe1,qe2,…,qe n The sampling strategy used at that time is based on the context reference set CR. s Generate multiple new query expansions Q with the original query q, each guided by the query answer. e ={qe1,qe2,…,qe n The sampling strategy used is the same.

4. The method according to claim 1, characterized in that, The large-scale language models (LLMs) mentioned are dialogue models with more than 7 parameters.

5. A query method, characterized in that, The method described in claim 1 generates an optimized query expansion of the original query q and inputs it into the large language model LLMs. The query is then performed in the corpus to obtain the query result of the original query q.

Images