Comprehensive processing framework for open domain question and answer tasks

Abstract

The invention relates to the technical field of natural language processing and artificial intelligence, in particular to a comprehensive processing framework for open domain question and answer tasks, which comprises an extension generation module, an extension evaluation module, a document reordering module, a reordering evaluation module, an answer generation module and an answer verification module, on the basis of supporting multi-field and multi-task questions and answers, the framework adopts a model fine tuning and dynamic cue word optimization strategy, improves the adaptability and performance of a system, proposes an extension answer generation module, ensures that reasonable answers can be generated when documents are insufficient, ensures the continuity of a question and answer process, and meanwhile, adds a verification and feedback mechanism, and improves the system performance. According to the system, the deep fusion architecture of retrieval, generation and verification is adopted, the multi-task and multi-field efficient question and answer ability is achieved, and reliable technical support is provided for scenes such as intelligent customer service, medical consultation and legal advisers.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and artificial intelligence technology, and in particular to a comprehensive processing framework for open domain question answering tasks. Background Art

[0002] With the rapid growth of Internet information, users' demand for intelligent question-answering systems continues to increase. Traditional question-answering systems mostly rely on predefined structured knowledge bases or FAQs, which are difficult to deal with complex and diverse open-domain questions, resulting in insufficient flexibility, limited coverage, and failure to meet user needs. Therefore, developing open-domain question-answering systems that can process unstructured information and dynamically generate answers has become a research focus. An open-domain question-answering system is an intelligent system designed to extract answers from a wide range of information sources and provide natural language responses.

[0003] Open-domain question-answering systems need to extract information from large-scale unstructured data, such as web pages, documents, news, etc., but the dispersion and heterogeneity of such data increase the complexity of retrieval and generation. Modern question-answering systems mostly adopt an architecture that combines retrieval and generation: the retrieval module screens candidate documents, the generation module generates answers, and the verification module ensures the accuracy of the answers. However, existing systems are prone to interruption when there are insufficient documents or retrieval fails, affecting user experience. The emergence of large language models (LLMs) such as GPT and BERT has brought significant improvements to question-answering systems. These models have powerful language understanding and generation capabilities after pre-training, but they also have problems such as high resource consumption and lack of domain expertise.

[0004] Traditional question-answering systems mainly rely on preset structured knowledge bases or FAQs, which are difficult to handle complex and diverse open-domain questions, resulting in deficiencies in system flexibility and information coverage, and making it difficult to fully meet the diverse needs of users. The present invention designs a framework for open-domain question-answering, which expands the generation and verification of open-domain questions, achieves higher flexibility and accuracy in the process of evidence collection and answer generation, and provides reliable technical support for scenarios such as intelligent customer service, medical consultation, and legal counsel. Summary of the invention

[0005] In view of the above-mentioned defects and problems, the present invention provides a comprehensive processing framework for open domain question-answering tasks. On the basis of supporting multi-domain and multi-task question-answering, it adopts model fine-tuning and dynamic prompt word optimization strategies to improve the adaptability and performance of the system. An extended answer generation module is proposed to ensure that reasonable answers can be generated even when documents are insufficient, thereby ensuring the continuity of the question-answering process. At the same time, a verification and feedback mechanism is added to perform a logical review of the answers before output to reduce the occurrence of errors. This system adopts a deep fusion architecture of retrieval, generation, and verification, and realizes efficient question-answering capabilities in multiple tasks and fields, providing reliable technical support for scenarios such as intelligent customer service, medical consultation, and legal counsel.

[0006] The solution adopted by the present invention to solve its technical problem is: a comprehensive processing framework for open domain question answering tasks, the framework includes an extension generation module, an extension evaluation module, a document reordering module, a reordering evaluation module, an answer generation module, and an answer verification module; The following steps are also included: Step 1: Generate extensions: Starting from the question input by the user, use the large language model LLM to generate additional background information. The pre-trained large language model receives the question and its prompt information and outputs the extension. Step 2, extension evaluation: After the extensions are generated, each extension is scored using the LLM evaluator, with a score range of 0 to 1, to select the best extension for subsequent processing. In this way, low-quality extensions are filtered out to ensure that the data processed in the subsequent steps has high quality and relevance; Step 3, document re-ranking: Based on the initially retrieved document set, the evaluated extensions are combined with the original question, and the candidate documents are re-retrieved and ranked. The candidate document set is processed by the LLM ranker, and the optimal document set is output for answer generation. Step 4: Re-ranking evaluation: After the document re-ranking is completed, the sorted document set is further evaluated to ensure the high relevance and quality of the document content. The document is scored using the LLM evaluator, and high-quality documents are further screened out for answer generation based on the scoring results. Step 5, answer generation: After completing the document re-ranking and evaluation, use the LLM reader to generate the final answer based on the question and the filtered document set; Step 6, answer verification: After generating the preliminary answer, a verification mechanism is introduced to check the logical consistency and accuracy of the answer through the LLM verifier. If the verification fails, the system will regenerate the answer.

[0007] Furthermore, in step 1, the extension generation module performs the following operations: S11. Input question: Identify the core questions that the system needs to answer to guide the extended generation module to generate relevant background information, and standardize the questions to adapt them to the input format of large language models; S12. Prompt word generation: Generate a specific prompt word based on the input question e Used to guide large language models to generate background information that meets target requirements; S13, model call: question q and prompt word prompt e As input, call the large language model G e In this step, the large language model generates extended information based on the input content. The model not only processes the initial question, but also generates a variety of background information based on the instructions given by the prompt word: e = G e (q; prompt e ); S14, extended generation: Generate background information related to the question to support the subsequent document retrieval and answer generation steps. The output of this step is the extended content e in the form of a natural language text paragraph. score ev = E e (q; e; prompt ev ).

[0008] Furthermore, in the extended generation module, the genetic algorithm will generate new prompt combinations in each generation. These combinations will be scored according to different settings or even knowledge adjustments in specific fields. When the prompt generated by the genetic algorithm exceeds the current template, it will be set as the new benchmark solution.

[0009] Furthermore, in step 3, the document reordering module reorders the documents according to the following steps: S31. First, we retrieve a set of the top n documents related to a given question by using established information retrieval techniques. These retrieval methods provide a preliminary score for each candidate document to describe the relevance between the document and the question. Then, we use the ranking capability of the large model to manually design the corresponding prompt. d Tip: Use it to reorder documents. S32, then re-rank the documents. This stage involves comparing documents to determine which documents are more relevant and of higher quality to the query, guiding LLM to act as a document re-ranker R d The role of further screening out the best document from the preliminary candidate document pool as the final evidence D, the overall document re-ranking process can be simplified as: D = Rd ( D coarse , q , e ; prompt d ).

[0010] Furthermore, in step 4, the re-ranking evaluation module scores the document using the following formula: S41, taking the re-ranked document set D as the input of the evaluation module, after preliminary re-ranking, the system will select a candidate document set D, which contains documents related to the question and the extended information; S42, system calls large language model E d Each document is analyzed to evaluate whether it accurately answers the input question q and its consistency with the extended information e. The evaluation results are in the form of a score ranging from 0 to 1, representing the relevance and quality of the document; S43. Use prompt word d Guide the language model to evaluate documents to ensure that the evaluation process meets system requirements, prompt word d Used to provide instructions to the language model so that it understands how to evaluate the relevance of documents to questions; S44. The evaluation module calculates the document score using the following formula: Furthermore, in step 5 and step 6, the answer generation module and the answer verification module use the following process and formula to generate and verify the answer: S51. Use LLM to generate answers based on questions and evidence: Input question q and the reordered document set D into the generator, and then use the prompt word prompt at Guide the answer generation process, and finally the final answer a is generated by LLM t : a t = G a (q, D; prompt at ); S52, verify the generated answer: the initially generated answer a t Enter the validator together with the question q and the evidence D, and then use the prompt word prompt av Guide the verification process, and finally verify and output. If the answer passes the verification, the final answer a is output v : a v = G v (q, a t , D; prompt av ); S53, the framework directly generates answers through generated extensions when retrieval fails.

[0011] Further, the steps to regenerate the answer are as follows: S531, when the user inputs question q, the system first calls the retrieval module to try to retrieve the candidate document set D from the knowledge base or document set. coarse After completing the preliminary search, the system checks the search results to determine whether the following conditions are met: the search results are empty; the number of documents is insufficient; the document quality does not meet the standards.

[0012] S532: After determining that the search fails, the system calls the extension generation module to generate extended information e related to question q. The system inputs the extended information and the question into the answer generation module and generates an answer through the model G. a Generate a reasonable answer. At this time, the formula for the answer generation process is as follows: a t = G a ( q , D ; prompt at ); S533, when generating answers, the system dynamically adjusts the prompt word according to the extended information and question type at ,The flexibility of the prompt word design enables the system to generate answers of different styles.

[0013] Furthermore, the framework also supports different types of data sets, and dynamically adjusts the strategies of extension generation and document reordering according to the changes in the data sets to reduce noise interference and improve system performance. The dynamic strategy adjustment of the system can be described by the following formula: strategy t+1 = f ( dataset t+1 , prompt t+1 ).

[0014] Furthermore, both the answer generator and the verifier use fine-tuned large language models and are optimized based on specific datasets for different tasks. LLM is fine-tuned through domain datasets to perform well in different application scenarios. Each dataset is carefully selected to ensure that the model can understand the terminology and common problems in the field. The fine-tuning formula is: θ′ = fine_tune ( θ ; D task ); Among them, θ is the initial parameter of the model; D task is the task dataset used for fine-tuning; θ′ is the model parameters after fine-tuning.

[0015] Further, the workflow of the answer generation module: Generate high-quality answers based on the user's input questions and extended information The system receives the input question q and extended information e, and generates high-quality answers through the model generation module Ga and adaptive prompt at To guide the generation process, the system adjusts the generation format according to the task requirements. The generation formula is: a t = generate( q , e , prompt at ; θ′); The workflow of the answer verification module: The verification module uses the verification model G v To check the generated answer, the system will input question q and the generated answer a t Enter the verification model together with the extended information e, and follow the verification prompt word prompt av Guide the verification process. If the verification fails, the system will regenerate the answer based on the feedback. The verification formula is: a v = verify ( q , at , e , prompt av ; θ′ ).

[0016] Beneficial effects of the present invention: Combining generation and verification: The comprehensive framework proposed in this paper combines generation and verification, evaluates and optimizes in multiple links, and thus significantly improves the performance of the ODQA system in terms of generation expansion, document selection, and answer generation; High-quality extension generation and evaluation: By generating high-quality query extensions and rigorously evaluating them, the present invention can effectively improve the accuracy of document re-ranking and answer generation; Document re-ranking optimization: By introducing the relevance evaluation of extended content in the document re-ranking process, the system can better screen out the documents most relevant to the question and reduce the impact of noise documents on answer generation; Dynamic verification mechanism: A verification mechanism is introduced after the answer is generated to ensure the accuracy and consistency of the output answer, reduce the possibility of error generation, and improve the overall robustness of the system; Multi-level quality control: Through multiple links of quality assessment and screening mechanisms, the present invention can ensure the high quality of processed data at every step, ultimately significantly improving the accuracy and performance of the system; Introduce genetic algorithm to compare and analyze the generated results, select the better ones, and continue the genetic algorithm evolution; In summary, in response to the shortcomings of the existing systems, the present invention, on the basis of supporting multi-field and multi-task question and answer, adopts model fine-tuning and dynamic prompt word optimization strategies, improves the adaptability and performance of the system, and proposes an extended answer generation module to ensure that reasonable answers can be generated even when documents are insufficient, thereby ensuring the continuity of the question and answer process. At the same time, a verification and feedback mechanism is added to perform logical review of the answers before output to reduce the occurrence of errors. The system adopts a deep fusion architecture of retrieval, generation and verification, and realizes efficient question and answer capabilities in multiple tasks and fields, providing reliable technical support for scenarios such as intelligent customer service, medical consultation, and legal consulting. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A comprehensive processing framework for open domain question answering; Figure 2 Flowchart for the generation, ranking and verification of an open domain question answering system; Figure 3 Flowchart of the expansion, retrieval and verification framework for the open domain question answering system; Figure 4 Comparison of the performance of open-domain question answering systems on TriviaQA, WebQ, and NQ datasets; Figure 5 Ablation experiment results of the EGV model framework on TriviaQA, WebQ and NQ datasets. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0019] See also Figure 1-5 , the present invention provides a technical solution for a comprehensive processing framework for open domain question answering tasks: Embodiment 1: This embodiment provides a comprehensive generation and verification framework for open domain question answering (ODQA), which specifically includes the following steps: Step 1: Generate extension First, the system starts with the input question and uses the large language model LLM to generate additional background information, called query expansion. The purpose of generating expansion is to provide richer context to improve the accuracy of subsequent document selection and answer generation. Through pre-trained large language models, such as the GPT series, the question and its prompt information are received and the expansion is output. The generated extension may include relevant background knowledge, contextual explanations or other auxiliary information. These extensions help enrich the semantics of the question, thereby providing stronger support for answer generation and improving the quality of subsequent retrieval and answer generation.

[0020] Step 2: Extended Assessment After the extensions are generated, they need to be evaluated for quality to ensure their logical consistency and relevance to the problem. To this end, the LLM evaluator is used to score each extension, with a score range of 0 to 1, to select the best extension for subsequent processing. In this way, low-quality extensions can be filtered out, ensuring that only high-quality, problem-related extensions are retained in subsequent steps, avoiding noise interference and irrelevant information from affecting the final results.

[0021] The Generate Extension step leverages the knowledge stored in the large language model to generate additional background information to improve the coverage of context and the depth of understanding of the question. Generate Extension and Extend Evaluation perform the following steps: S11. Input question: Clarify the core questions that the system needs to answer to guide the extended generation module to generate relevant background information. The input question q can be a short natural language question, such as "What is quantum computing?" or "Who is Einstein's mentor?". These questions are user-entered queries, which are usually short and lack sufficient contextual information. Standardize the questions, such as removing redundant symbols or stop words, to adapt them to the input format of large language models; S12. Prompt word generation: Generate a specific prompt word based on the input question e Used to guide large language models to generate background information that meets target requirements. e is an instruction or context used to guide the output of a language model; S13, model call: question q and prompt word prompt e As input, call the large language model G e Perform extended generation. In this step, a large language model (such as GPT or other pre-trained language models) generates extended information based on the input content. The model not only processes the initial question, but also generates a variety of background information based on the instructions given by the prompt word: e = G e (q; prompt e ); S14, extension generation: Generate background information related to the question to support the subsequent document retrieval and answer generation steps. The output of this step is the extension content e, which can be in the form of a natural language text paragraph. These extensions not only provide the basic background of the question, but also predict relevant information, which helps the system find more accurate evidence during the retrieval process; score ev = E e (q; e; prompt ev ).

[0022] For the initial generation, the genetic algorithm will generate new prompt combinations in each generation. These combinations will be adjusted according to different settings or even knowledge in a specific field, and the newly generated prompts will be scored. When the prompt generated by the genetic algorithm exceeds the current template, it will be set as the new baseline solution. The system can continuously evolve in different task types. In this way, it breaks the limitations of traditional templates and allows the system to flexibly handle various tasks from historical problems to logical reasoning problems.

[0023] Step 3: Reorder documents The system combines the evaluated extensions with the original question and re-retrieves and re-ranks the candidate documents. The re-ranking mechanism allows the system to prioritize documents that are most likely to contain the answer. The candidate document set is processed by the LLM ranker and the optimal document set is output for answer generation, ensuring that more relevant documents are prioritized for answer generation. The ranking of documents not only considers their direct relevance to the question, but also evaluates the indirect relevance of the documents based on the extension content to ensure that the final selected documents can provide more reliable evidence for answer generation.

[0024] The document reordering module reorders documents according to the following steps: S31. In order to identify the most appropriate document, we first retrieve a set of the top n documents that may be relevant to a given question by using established information retrieval techniques such as DPR. These retrieval methods provide a preliminary score for each candidate document to describe the relevance between the document and the question. However, these methods may not always capture the subtle semantic relationship between the query and the document, resulting in false positive examples or irrelevant documents in the preliminary set. Therefore, the document re-ranking process still utilizes the ranking ability of the large model, and manually designs the corresponding prompts. d Tip, use it to complete the sorting; S32, Reranking: The documents are then reranked to ensure that documents that are more likely to contain the answer are prioritized. This stage involves comparing documents to determine which documents are more relevant and high quality to the query. Instruct LLM to act as a document reranker R d The role of further screening out the best document from the preliminary candidate document pool as the final evidence D. The overall document re-ranking process can be simplified as: D = R d ( D coarse , q , e ; prompt d ).

[0025] Step 4: Re-rank evaluation After the document re-ranking is completed, the sorted document set is further evaluated for quality to ensure the high relevance and quality of the document content. The LLM evaluator is used to score the documents, and high-quality documents are further screened out for answer generation based on the scoring results. Through multiple screening and evaluation, it is ensured that the documents required for answer generation have high credibility and information value, thereby enhancing the quality of answer generation.

[0026] The rerank evaluation module scores documents using the following formula: S41, taking the re-ranked document set D as the input of the evaluation module, after the preliminary re-ranking, the system will select a candidate document set D, which contains documents related to the question and the extended information. However, due to the possible sorting errors or some noise information, the quality of these documents needs to be further evaluated; S42, system calls large language model E d Each document is analyzed to assess whether it accurately answers the input question q and is consistent with the extended information e. The evaluation results are in the form of a score ranging from 0 to 1, representing the relevance and quality of the document; S43. Use prompt word d Guide the language model to evaluate documents to ensure that the evaluation process meets the system requirements. d Used to provide instructions to the language model so that it understands how to evaluate the relevance of a document to the question. For example, the prompt word can instruct the model to score based on whether the document directly answers the question, or based on whether the document is consistent with the extended information; S44. The evaluation module calculates the document score using the following formula: score d = E d ( D ; q ; e ; prompt d ).

[0027] Step 5: Answer Generation After the documents are re-ranked and evaluated, the LLM reader is used to generate the final answer based on the question and the filtered document collection. The answers generated by the model are not only based on the direct information in the document, but also take into account the extended content and question context, thereby generating more accurate and relevant answers.

[0028] Step 6: Answer verification After generating the preliminary answer, a verification mechanism is introduced to verify the answer through the LLM verifier to ensure its consistency and accuracy. If the verification fails, the system will regenerate the answer. This verification process can detect and correct potential errors in the generation process, thereby improving the reliability of the final output answer.

[0029] The answer generation module and answer verification module use the following process and formula to generate and verify the answer: S51. Use LLM to generate answers based on questions and evidence. Input questions and evidence: Input question q and the reordered document set D into the generator. Prompt word design: Use prompt word prompt at Guide the answer generation process. Answer generation: The final answer is generated by LLM. t : a t = G a (q, D; prompt at ); S52. Verify the generated answer to ensure its accuracy and consistency. If the verification fails, regenerate the answer. Input the question and the generated answer: t Enter the verifier together with question q and evidence D. Prompt design: Use prompt word prompt av Guide the verification process; verification and output: if the answer passes the verification, the final answer a is output v : Otherwise, regenerate the answer; a v = G v (q, a t , D; prompt av ); S53. The framework can directly generate answers through the generated extensions when retrieval fails to reduce the impact of document missing on answer generation and ensure the continuity of the question-answering process. The steps to regenerate answers are as follows: S531, when the user inputs question q, the system will first call the retrieval module to try to retrieve the candidate document set D from the knowledge base or document set coarse After completing the preliminary search, the system will check the search results to determine whether the following conditions are met: the search results are empty; the number of documents is insufficient; the document quality does not meet the standards.

[0030] S532: After determining that the search fails, the system will call the extension generation module to generate extended information e related to question q. The system will input the extended information and the question into the answer generation module and generate the extended information e through the model G. a Generate a reasonable answer. At this time, the formula for the answer generation process is as follows: a t = G a ( q , D ; prompt at ); S533, Dynamic prompt word optimization and generation quality control: When the system generates answers, it will dynamically adjust the prompt word according to the extended information and question type. at ,The prompt words not only determine the format and depth of the generated answer, but also ensure that the content output by the model is highly consistent with the extended information.,The flexibility of the prompt word design enables the system to generate answers in different,styles, such as brief answers, paragraph explanations, or structured lists. Embodiment 2: Based on Embodiment 1, the framework also supports different types of data sets, and dynamically adjusts the strategies of extension generation and document reordering according to changes in data sets to reduce noise interference and improve system performance. Specifically, it includes: Dynamic adaptation and update mechanism: Ensure that the open-domain question answering system can adapt to various types of datasets, and automatically adjust the strategies of extension generation and document reordering according to changes in the dataset, thereby improving the flexibility and robustness of the system.

[0031] Dataset support and identification: The system automatically detects the type of input dataset and triggers corresponding strategies based on the type; Structured data: prefers direct retrieval and matching. Unstructured data: relies on extended generation modules to provide rich background information to optimize retrieval.

[0032] Dynamic monitoring of data set changes: When a data set change is detected, the system automatically triggers policy adjustments, reconfigures the extension generation and reordering modules, and monitors data set changes in real time to adapt to the impact of data updates.

[0033] Adjustment of extension generation strategy: When data is updated, the system will adjust the extension generation strategy according to the newly added data to ensure that the generated extension information is consistent with the latest status of the data set. e It will automatically update according to data changes, guiding the model to generate more targeted extended content.

[0034] Adjustment of document re-ranking strategy: When the data set changes, the system will adjust the parameters of the document re-ranking module. For example, when the number of documents increases, the system can dynamically increase the step size of the sliding window to improve the sorting efficiency.

[0035] Sliding window mechanism is used for optimization: the parameters of window size and step length can be automatically adjusted according to the scale of the dataset.

[0036] The dynamic strategy adjustment of the system can be described by the following formula: strategy t+1 = f ( dataset t+1 , prompt t+1 ). Example 3: Both the answer generator and the verifier use a fine-tuned large language model (LLM) and are optimized based on specific datasets for different tasks.

[0037] Fine-tuning Large Language Models (LLMs): LLMs are fine-tuned using domain datasets to excel in different application scenarios. Each dataset is carefully selected to ensure that the model can understand the terminology and common problems in the domain. For example, in medical question answering, the model is fine-tuned using medical literature and medical records, while in legal question answering, it is optimized using laws, regulations, and case library data: θ′ = fine_tune ( θ ; D task ); θ is the initial parameter of the model; D task is the task dataset used for fine-tuning; θ′ is the model parameters after fine-tuning.

[0038] The workflow of the answer generation module: Generate high-quality answers based on the questions and extended information input by the user. The system receives the input question q and extended information e, and generates a high-quality answer through the model generation module G. a and adaptive prompt at Guide the generation process. The system adjusts the generation format according to the task requirements, such as paragraph explanation or short answer: a t = generate( q , e , prompt at ; θ′).

[0039] The workflow of the answer verification module: The verification module uses the verification model G v Check the generated answer. The system will input question q and the generated answer a t Enter the verification model together with the extended information e, and follow the verification prompt word prompt av Guide the verification process. If verification fails, the system will regenerate the answer based on the feedback: a v = verify ( q , at , e , prompt av ; θ′ ).

[0040] Dynamic prompts and adaptive optimization: The system generates different prompts based on the complexity and context of the question. The prompts guide the model to generate answers that meet the requirements and ensure the consistency of the generated content.

[0041] The system's multi-task support and robustness: When processing different tasks, the system can dynamically optimize according to the task data set. Whether it is in the field of education, medical care or law, the system can adapt to the task requirements by fine-tuning the model and prompt words, and reduce the output of incorrect answers through the verification module. Example 4: This example provides an experiment conducted to evaluate the performance of the proposed comprehensive processing framework in the open domain question answering (ODQA) task. Multiple benchmark datasets and comparison models are used in the experiment to comprehensively evaluate the advantages of the proposed method.

[0042] Three widely used ODQA benchmark datasets were selected in the experiment to evaluate the performance of the model: WebQ (WebQuestions), NQ (Natural Questions) and TriviaQA are important datasets for open domain question answering, covering questions and answers from Google Suggest API, Google search queries and quiz websites respectively. In order to verify the performance of the model, the study used two types of benchmark models for comparison: the first type is the traditional method that does not rely on large language models (LLM), such as BM25+BERT combined with sparse retrieval and BERT for text representation, REALM retrieves information from the knowledge base and integrates it into training, DPR uses dense encoders for vector retrieval, RAG combines retrieval and generation tasks, and FiD uses retrieval-reading architecture for question answering; the second type uses LLM as a generator, such as GenRead generates diversified documents based on clustering, and EAR improves the quality of evidence through query reranking.

[0043] The experimental process is as follows Figure 3 As shown, it includes generation and expansion - expansion evaluation - document re-ranking - re-ranking evaluation - genetic evolution - answer generation - answer verification, among which, genetic evolution: compare with the answer generated by the genetic algorithm, and select the best one to proceed to the next step.

[0044] The exact match score is used to evaluate the performance of the model. Figure 4 As shown in Figure 1, the EM score measures the correctness of the answer by comparing the normalized form of the answer to the list of acceptable answers. Through these evaluations, the effectiveness of the proposed framework in the open domain question answering task is verified. The EM score compares the degree of match between the predicted answer and each true answer, and the recall rate is also an indicator for evaluating the quality of evidence.

[0045] EGV achieves significant improvements on three datasets: TriviaQA improves by 13.3 points, WebQ improves by 6.9 points, and NQ improves by 9.5 points. This demonstrates the effectiveness of LLM in open domain question answering (ODQA) and highlights the impact of model scale on the results. In addition, EGV's average performance on the three datasets is 9.9 points higher than FiD-xl, despite using fewer documents, proving that LLM can outperform previously designed models in different role-playing.

[0046] Then, we conducted ablation experiments, removing the extended generator, reranker, and evaluator, respectively, and explored the impact of these four aspects on the performance of the open domain question answering system (ODQA). Figure 5 As shown in the figure, the role of the extended generator in the LLM (Large Language Model) has the greatest impact among the four different roles, which shows that question extension can be used as auxiliary documents as well as evidence documents; the reranker also contributes to ODQA because the reranked documents are more relevant to the question; the feasibility of the evaluator is also demonstrated because it can estimate the quality of evidence and select the most appropriate document verifier. The impact on ODQA is also important, which shows that for existing questions, candidate documents and preliminary answer verifiers can play a good role.

[0047] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A comprehensive processing framework for open domain question answering tasks, characterized by: The framework includes an extension generation module, an extension evaluation module, a document reordering module, a reordering evaluation module, an answer generation module, and an answer verification module; The following steps are also included: Step 1: Generate extensions: Starting from the question input by the user, use the large language model LLM to generate additional background information. The pre-trained large language model receives the question and its prompt information and outputs the extension. Step 2, extension evaluation: After the extensions are generated, each extension is scored using the LLM evaluator, with a score range of 0 to 1, to select the best extension for subsequent processing. In this way, low-quality extensions are filtered out to ensure that the data processed in the subsequent steps has high quality and relevance; Step 3, document re-ranking: Based on the initially retrieved document set, the evaluated extensions are combined with the original question, and the candidate documents are re-retrieved and ranked. The candidate document set is processed by the LLM ranker, and the optimal document set is output for answer generation. Step 4: Re-ranking evaluation: After the document re-ranking is completed, the sorted document set is further evaluated to ensure the high relevance and quality of the document content. The document is scored using the LLM evaluator, and high-quality documents are further screened out for answer generation based on the scoring results. Step 5, answer generation: After completing the document re-ranking and evaluation, use the LLM reader to generate the final answer based on the question and the filtered document set; Step 6, answer verification: After generating the preliminary answer, a verification mechanism is introduced to check the logical consistency and accuracy of the answer through the LLM verifier. If the verification fails, the system will regenerate the answer.

2. A comprehensive processing framework for open domain question answering tasks according to claim 1, characterized in that: In step 1, the extension generation module performs the following operations: S11. Input question: Identify the core questions that the system needs to answer to guide the extended generation module to generate relevant background information, and standardize the questions to adapt them to the input format of large language models; S12. Prompt word generation: Generate a specific prompt word based on the input question e Used to guide large language models to generate background information that meets target requirements; S13, model call: question q and prompt word prompt e As input, call the large language model G e In this step, the large language model generates extended information based on the input content. The model not only processes the initial question, but also generates a variety of background information based on the instructions given by the prompt word: e = G e (q; prompt e ); S14, extended generation: Generate background information related to the question to support the subsequent document retrieval and answer generation steps. The output of this step is the extended content e in the form of a natural language text paragraph. score ev = E e (q; e; prompt ev )。 3. A comprehensive processing framework for open domain question answering tasks according to claim 1 or 2, characterized in that: In the extended generation module, the genetic algorithm will generate new prompt combinations in each generation. These combinations will be scored according to different settings or even knowledge adjustments in specific fields. When the prompt generated by the genetic algorithm exceeds the current template, it will be set as the new benchmark solution.

4. A comprehensive processing framework for open domain question answering tasks according to claim 1, characterized in that: In step 3, the document reordering module reorders the documents according to the following steps: S31. First, we retrieve a set of the top n documents related to a given question by using established information retrieval techniques. These retrieval methods provide a preliminary score for each candidate document to describe the relevance between the document and the question. Then, we use the ranking capability of the large model to manually design the corresponding prompt. d Tip: Use it to reorder documents. S32, then re-rank the documents. This stage involves comparing documents to determine which documents are more relevant and of higher quality to the query, guiding LLM to act as a document re-ranker R d The role of further screening out the best document from the preliminary candidate document pool as the final evidence D, the overall document re-ranking process can be simplified as: D = R d ( D coarse , q , e ; prompt d )。 5. A comprehensive processing framework for open domain question answering tasks according to claim 1, characterized in that: In step 4, the re-ranking evaluation module scores the document using the following formula: S41, taking the re-ranked document set D as the input of the evaluation module, after preliminary re-ranking, the system will select a candidate document set D, which contains documents related to the question and the extended information; S42, system calls large language model E d Each document is analyzed to evaluate whether it accurately answers the input question q and its consistency with the extended information e. The evaluation results are in the form of a score ranging from 0 to 1, representing the relevance and quality of the document; S43. Use prompt word d Guide the language model to evaluate documents to ensure that the evaluation process meets system requirements, prompt word d Used to provide instructions to the language model so that it understands how to evaluate the relevance of documents to questions; S44. The evaluation module calculates the document score using the following formula: score d = E d ( D ; q ; e ; prompt d )。 6. A comprehensive processing framework for open domain question answering tasks according to claim 1, characterized in that: In step 5 and step 6, the answer generation module and the answer verification module use the following process and formula to generate and verify the answer: S51. Use LLM to generate answers based on questions and evidence: Input question q and the reordered document set D into the generator, and then use the prompt word prompt at Guide the answer generation process, and finally the final answer a is generated by LLM t : a t = G a (q, D; prompt at ); S52, verify the generated answer: the initially generated answer a t Enter the validator together with the question q and the evidence D, and then use the prompt word prompt av Guide the verification process, and finally verify and output. If the answer passes the verification, the final answer a is output v : a v = G v (q, a t , D; prompt av ); S53, the framework directly generates answers through generated extensions when retrieval fails.

7. A comprehensive processing framework for open domain question answering tasks according to claim 6, characterized in that: The steps to regenerate the answer are as follows: S531, when the user inputs question q, the system first calls the retrieval module and tries to retrieve the candidate document set D from the knowledge base or document set. coarse ,After completing the preliminary search, the system checks the search results to determine whether the following conditions are met: the search results are empty; the number of documents is insufficient; the ,quality of the documents does not meet the standards; S532: After determining that the search fails, the system calls the extension generation module to generate extended information e related to question q. The system inputs the extended information and the question into the answer generation module and generates an answer through the model G. a Generate a reasonable answer. At this time, the formula for the answer generation process is as follows: a t = G a ( q , D ; prompt at ); S533, when generating answers, the system dynamically adjusts the prompt word according to the extended information and question type at ,The flexibility of the prompt word design enables the system to generate answers of different styles.

8. A comprehensive processing framework for open domain question answering tasks according to claims 1-7, characterized in that: The framework also supports different types of data sets, and dynamically adjusts the strategies of extension generation and document reordering according to changes in the data sets to reduce noise interference and improve system performance. The dynamic strategy adjustment of the system can be described by the following formula: strategy t+1 = f ( dataset t+1 , prompt t+1 )。 9. A comprehensive processing framework for open domain question answering tasks according to claim 1, characterized in that: Both the answer generator and the verifier use fine-tuned large language models and are optimized based on specific datasets for different tasks. LLM is fine-tuned through domain datasets to perform well in different application scenarios. Each dataset is carefully selected to ensure that the model can understand the terminology and common problems in the field. The fine-tuning formula is: θ′ = fine_tune ( θ ; D task ); Among them, θ is the initial parameter of the model; D task is the task dataset used for fine-tuning; θ′ is the model parameters after fine-tuning.

10. A comprehensive processing framework for open domain question answering tasks according to claim 9, characterized in that: Fine-tuning workflow of the answer generation module: Generate high-quality answers based on user input questions and extended information The system receives the input question q and extended information e, and generates answers through the model generation module G a and adaptive prompt at To guide the generation process, the system adjusts the generation format according to the task requirements. The generation formula is: a t = generate( q , e , prompt at ; θ′); The workflow of the answer verification module: The verification module uses the verification model G v To check the generated answer, the system will input question q and the generated answer a t Enter the verification model together with the extended information e, and follow the verification prompt word prompt av Guide the verification process. If the verification fails, the system will regenerate the answer based on the feedback. The verification formula is: a v = verify ( q , at , e , prompt av ; θ′ )。

Images