Large model-based training data generation method and device, and model training method and device

Abstract

The invention provides a training data generation method and device based on a large model and a model training method and device, relates to the technical field of artificial intelligence such as natural language processing, large models, intelligent search and knowledge maps, and can be applied to scenes such as knowledge questions and answers. The method comprises the steps of obtaining triple data based on historical operation data, and screening out triple data with wrong answers from the triple data as triple data to be corrected; rewriting problems in the to-be-corrected triple data on the basis of iterative reflection of the large model, and generating a corrected retrieval fragment on the basis of the rewritten problems and the large model; based on the rewritten question and the corrected retrieval fragment, a large model is adopted to generate a correction answer corresponding to the rewritten question; and on the basis of the rewritten question, the corrected retrieval fragment and the correction answer, updating the to-be-corrected triple data to obtain retrieval enhancement generated RAG training data. The training data quality and the generation efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as natural language processing, big models, intelligent search, knowledge graphs, and specifically to a big model-based training data generation method, knowledge question and answer model training method, device, and electronic equipment, which can be applied to scenarios such as knowledge question and answer. Background Art

[0002] The RAG (Retrieval-Augmented Generation) system is a model that combines retrieval and generation capabilities to answer questions by retrieving relevant information from a large amount of text and generating answers. However, there is currently a lack of effective means for generating RAG training data. Summary of the invention

[0003] The present invention provides a large-model-based training data generation method, a knowledge question-answering model training method, a device, an electronic device, and a storage medium.

[0004] According to a first aspect of the present disclosure, a method for generating training data based on a large model is provided, comprising:

[0005] Based on the historical operation data, N triple data are obtained, where N is a positive integer; wherein each triple data includes a question, an answer corresponding to the question, and a search fragment;

[0006] Filter out triple data with incorrect answers from the N triple data as triple data to be corrected;

[0007] Rewrite the question in the triple data to be corrected based on iterative reflection of the large model, and generate a corrected search fragment based on the rewritten question and the large model;

[0008] Based on the rewritten question and the revised search fragment, using the large model to generate a corrected answer corresponding to the rewritten question;

[0009] Based on the rewritten question, the revised search fragment and the corrected answer, the triple data to be revised is updated to obtain the retrieval enhancement generation RAG training data.

[0010] According to a second aspect of the present disclosure, a knowledge question answering model training method is provided, comprising:

[0011] Acquire training data, where the training data is generated based on the large model-based training data generation method described in the first aspect above;

[0012] The knowledge question answering model is trained based on the training data.

[0013] According to a third aspect of the present disclosure, a training data generating device based on a large model is provided, comprising:

[0014] An acquisition module, configured to acquire N triple data based on historical operation data, where N is a positive integer; wherein each triple data includes a question, an answer corresponding to the question, and a search fragment;

[0015] A screening module, used for screening out triple data with wrong answers from the N triple data as triple data to be corrected;

[0016] A rewriting module, used for rewriting the questions in the triple data to be corrected based on the iterative reflection of the large model, and generating a corrected search fragment based on the rewritten questions and the large model;

[0017] A generating module, configured to generate a corrected answer corresponding to the rewritten question using the large model based on the rewritten question and the corrected search fragment;

[0018] An updating module is used to update the triple data to be corrected based on the rewritten question, the corrected search fragment and the corrected answer to obtain retrieval enhancement and generate RAG training data.

[0019] According to a fourth aspect of the present disclosure, a knowledge question answering model training device is provided, comprising:

[0020] An acquisition module, used to acquire training data, wherein the training data is generated based on the large model-based training data generation method described in the first aspect;

[0021] A training module is used to train the knowledge question answering model based on the training data.

[0022] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first and second aspects.

[0026] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the methods described in the first and second aspects above.

[0027] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in the first and second aspects when executed by a processor.

[0028] According to the technology disclosed in the present invention, by allowing the large model to reflect on the rewriting problem and automatically correct the retrieval fragment error problem, the problem that the retrieval error cannot be automatically corrected in the RAG data generation tool in the prior art can be solved, the quality of RAG training data can be improved, and the efficiency of training data generation can be improved.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0031] Figure 1 is a flow chart of a method for generating training data based on a large model provided in an embodiment of the present disclosure;

[0032] Figure 2 is a flow chart of a method for generating training data based on a large model provided in an embodiment of the present disclosure;

[0033] Figure 3 is a flow chart of a method for generating training data based on a large model provided in an embodiment of the present disclosure;

[0034] Figure 4 is an example diagram of a process for generating new search results based on a reflection method provided by an embodiment of the present disclosure;

[0035] Figure 5 is a flow chart of the knowledge question answering model training method provided by an embodiment of the present disclosure;

[0036] Figure 6 is a block diagram of a training data generation device based on a large model provided in an embodiment of the present disclosure;

[0037] Figure 7 is a block diagram of a knowledge question answering model training device provided by an embodiment of the present disclosure;

[0038] Figure 8 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0040] The terms used in one or more embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present disclosure. The singular forms of "a", "said" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0042] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0043] It is worth noting that in the embodiments of the present disclosure, certain software, components, models, etc. that already exist in the industry may be mentioned. They should be considered as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0044] The embodiments of the present disclosure relate to artificial intelligence technology fields such as natural language processing, large models, intelligent search, and knowledge graphs.

[0045] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0046] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. It is a discipline that uses computer technology to analyze, understand and process natural language, that is, it uses computers as a powerful tool for language research, conducts quantitative research on language information with the support of computers, and provides language descriptions that can be used by humans and computers.

[0047] A large model refers to a "large parameter" model trained using large-scale data and powerful computing power. These models are usually highly versatile and generalizable, and can be applied to natural language processing, image recognition, speech recognition and other fields. Exemplarily, the large model in this article can be a large language model (LLM for short), which refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. The large model can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and is an important way to artificial intelligence. Alternatively, the large model can also be other large models, for example, a multimodal large model or a basic large model.

[0048] Intelligent Retrieval is based on the relevance of documents and search terms, comprehensively examines indicators such as the importance of documents, and sorts the search results to provide higher search efficiency. The result sorting of intelligent search takes both relevance and importance into consideration. Relevance uses a weighted mixed index of each field, and the relevance analysis is more accurate. Importance refers to the evaluation of document quality through analysis of the authority of the document source and the citation relationship. This makes the result sorting more accurate and can put the documents most relevant to the user's wishes at the front, improving search efficiency.

[0049] Knowledge graph is a semantic network used to describe the relationship between entities. It is a representation method of semi-structured data, used to describe entities, attributes and the relationship between entities. Knowledge graphs are often used to build artificial intelligence applications such as intelligent search engines, recommendation systems, and question-answering systems. The core idea of ​​knowledge graphs is to convert information in the real world into graphs, where nodes represent entities and edges represent the relationship between entities. Knowledge graphs are not just a graphical knowledge base, they also contain semantic descriptions of entities and relationships that can be understood and processed by computers. The construction of knowledge graphs requires the extraction, fusion and reasoning of information, which is usually achieved using technologies such as natural language processing, image recognition, and machine learning.

[0050] The RAG (Retrieval-Augmented Generation) system is a model that combines retrieval and generation capabilities to answer questions by retrieving relevant information from a large amount of text and generating answers. However, there is currently a lack of effective means for generating RAG training data.

[0051] Based on this, the embodiments of the present disclosure provide a training data generation method, device and electronic device based on a large model. The present disclosure can improve the quality and generation efficiency of RAG training data by allowing the large model to reflect on rewriting problems and automatically correct the problem of retrieval fragment errors. The training data generation method based on a large model provided by the embodiments of the present disclosure can be applied to enterprise office scenarios, including but not limited to knowledge questions and answers within the enterprise. For example, the training data generation method provided by the embodiments of the present disclosure can be applied to the training of knowledge question and answer models. Through this method, a large amount of high-quality RAG training data can be automatically generated based on historical operation data, thereby improving the effect of the knowledge question and answer model in various business scenarios.

[0052] The following describes the method, device and electronic device for generating training data based on a large model according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0053] It should be noted that the execution subject of the training data generation method based on the large model of the embodiment of the present disclosure may be a training data generation device, which may be implemented by software and / or hardware, and may be configured in an electronic device. Exemplarily, the electronic device may include but is not limited to a terminal, a server, etc.

[0054] Figure 1 is a flow chart of a method for generating training data based on a large model provided in an embodiment of the present disclosure. Figure 1 As shown, the large model-based training data generation method may include but is not limited to the following steps.

[0055] In step 101, N triple data are obtained based on historical operation data, where N is a positive integer; each triple data includes a question, an answer corresponding to the question, and a search fragment.

[0056] In some embodiments, the collected historical operation data can be parsed and processed, and questions (wherein the question may refer to an input query), answers, and retrieval fragments can be parsed from the historical operation data to construct triple data. Exemplarily, the historical operation data may be historical operation data of a RAG system, i.e., historical operation data of using the RAG system, and the historical operation data includes operation data recorded each time the RAG system is used, including an input user query (question), a retrieval fragment obtained by retrieval based on the user query, and an answer generated based on the user query and the retrieval fragment.

[0057] In some embodiments, the RAG system can implement knowledge question and answer based on the big model. Among them, the search fragments obtained by searching based on the user query can be obtained using the big model, that is, the big model searches based on the user query to obtain the search fragments. Exemplarily, the answers generated based on the user query and the search fragments can be obtained using the big model, that is, the big model generates answers based on the user query and the search fragments.

[0058] In some embodiments, the above-mentioned search fragment may refer to the search results obtained based on the user query, for example, may include but is not limited to the retrieved documents and / or document fragments.

[0059] In step 102, triple data with incorrect answers are selected from the N triple data as triple data to be corrected.

[0060] In some embodiments, the large model can be used to filter out triple data with incorrect answers from N triple data as triple data to be corrected.

[0061] In one possible implementation, a first prompt information can be constructed using a first prompt template based on a question in N triple data and a retrieval fragment corresponding to the question; a reference answer corresponding to the question is obtained based on the first prompt information and a large model; and based on the answer corresponding to the question and the reference answer corresponding to the question, triple data with incorrect answers are screened out from the N triple data as triple data to be corrected.

[0062] Exemplarily, for each triple data in the N triple data, the question in the triple data and the search fragment corresponding to the question can be filled into the first prompt template to construct the first prompt information. The first prompt information is input into the large model, and the reference answer to the question is generated by the large model. Based on the answer corresponding to the question in the triple data and the reference answer corresponding to the question, the triple data with incorrect answers are screened out from the N triple data as the triple data to be corrected.

[0063] For example, the first prompt information may be expressed as follows:

[0064] "Current user question (Query): Question A

[0065] The search fragment corresponding to the current question A: Search fragment a

[0066] Please generate answers based on user questions and search snippets".

[0067] After constructing the first prompt information, the first prompt information can be input into the large model, and the large model can generate a corresponding answer based on question A and search fragment a, and the answer generated by the large model is used as the reference answer Aa of question A. The answer corresponding to question A is obtained from the corresponding triple data, and the answer corresponding to question A in the triple data is compared with the reference answer Aa of question A obtained by the large model. For example, similarity calculation can be performed. If the answer corresponding to question A in the triple data is consistent with the reference answer Aa of question A, or the similarity between the answer corresponding to question A in the triple data and the reference answer Aa of question A is greater than or equal to a threshold, it can be considered that the answer of the large model to question A is correct, that is, the triple data is triple data with a correct answer. If the answer corresponding to question A in the triple data is inconsistent with the reference answer Aa of question A, or the similarity between the answer corresponding to question A in the triple data and the reference answer Aa of question A is less than a threshold, it can be considered that the answer of the large model to question A is wrong, that is, the triple data is triple data with an incorrect answer.

[0068] In step 103, the question in the triple data to be corrected is rewritten based on iterative reflection of the large model, and a corrected search fragment is generated based on the rewritten question and the large model.

[0069] In some embodiments, the accuracy and / or relevance of the search fragments in the triple data to be corrected can be evaluated to obtain the evaluation results of the search fragments (such as including accuracy and / or relevance). Based on the evaluation results of the search fragments, the large model is used to continuously iterate and reflect to rewrite the question to generate new search fragments. The new search fragments are continuously evaluated and the question is rewritten until a satisfactory search result is obtained, for example, until the evaluation results of the generated search fragments meet the preset conditions. After the rewriting of the question is completed, the large model can be used to search based on the rewritten question to obtain the search fragment of the question, and the search fragment is used as the corrected search fragment.

[0070] In some embodiments, the large model has the ability to reflect, and the reflective ability of the large model means that the model can evaluate and improve its own output, thereby improving the quality and accuracy of the output. The importance of the reflective ability of the large model is reflected in the following aspects: (1) Error correction, the model can identify and correct its own errors, which is crucial to improving the quality of the output; (2) Learning efficiency, through self-reflection, the model can learn from errors faster; (3) Adaptability, self-reflection enables the model to adapt to new or unseen tasks and optimize performance through self-adjustment; (4) Robustness, enhancing the robustness of the model so that it can maintain stable performance in the face of uncertainty and noise. In an embodiment of the present disclosure, the iterative reflection of the large model can mean that each time the large model obtains a retrieval fragment based on the question and generates an answer based on the question and the retrieval fragment, the large model performs a reflection, and continues to rewrite the question based on the reflection result. Through continuous reflection and rewriting of the question, the large model can output satisfactory retrieval results.

[0071] It is worth noting that in the embodiments of the present disclosure, the problems in the triple data to be corrected are rewritten based on iterative reflection of the large model. The purpose is to automatically correct the retrieval results through iterative reflection of the large model and reduce the situation where training data cannot be generated due to retrieval errors.

[0072] In step 104, based on the rewritten question and the revised search fragment, the large model is used to generate a corrected answer corresponding to the rewritten question.

[0073] In some embodiments, a prompt information can be constructed using a prompt template based on the rewritten question and the corrected search fragment, and the prompt information can be used to instruct the big model to generate an answer based on the rewritten question and the corrected search fragment. The prompt information is input into the big model, and the big model can generate a corrected answer corresponding to the rewritten question based on the prompt information.

[0074] For example, the prompt information may be as follows:

[0075] "Rewritten question: Question A1

[0076] Corrected search segment (i.e., search segment obtained based on question A1): Search segment a1

[0077] Please generate an answer based on the rephrased question and the corrected query snippet."

[0078] After constructing the prompt information, the prompt information can be input into the big model. The big model can generate a corresponding answer based on question A1 and retrieval fragment a1, and the answer generated by the big model can be used as the updated answer corresponding to question A1.

[0079] In step 105, based on the rewritten question, the revised search fragment and the corrected answer, the triple data to be revised is updated to obtain RAG training data.

[0080] In some embodiments, the question in the triple data to be corrected can be replaced with the rewritten question, the retrieval fragment in the triple data to be corrected can be replaced with the corrected retrieval fragment, and the answer in the triple data to be corrected can be replaced with the corrected answer, so as to obtain RAG training data.

[0081] In the above embodiments, the present disclosure automatically generates a large amount of high-quality RAG training data based on real online historical operation data, which can give full play to the advantages of the field scenario, and can continuously fit the real user data distribution after repeated optimization, and finally achieve better question-answering effects in real business scenarios. The present disclosure introduces a retrieval correction mechanism based on reflection, which can automatically correct the retrieval error part and reduce the situation where training data cannot be generated due to retrieval errors, thereby improving the quality of training data and the efficiency of generating training data, and solves the problem that retrieval errors cannot be automatically corrected in RAG data generation tools in the prior art.

[0082] Figure 2 is a flow chart of a method for generating training data based on a large model provided in an embodiment of the present disclosure. Figure 2 As shown, in Figure 1 On the basis of the illustrated embodiment, the optional implementation method of rewriting the problems in the triple data to be corrected based on the iterative reflection of the large model includes but is not limited to the following steps.

[0083] In step 201, the search segments in the triple data to be corrected are evaluated to obtain the evaluation results of the search segments.

[0084] In some embodiments, based on the triple data to be corrected, the accuracy and / or relevance of the search fragment in the triple data to be corrected can be evaluated to obtain an evaluation result of the search fragment, which includes the accuracy and / or relevance.

[0085] Exemplarily, based on the triple data to be corrected, the large model is used to evaluate the accuracy and / or relevance of the search fragment in the triple data to be corrected. For example, the large model can be used to evaluate the accuracy and relevance of the search fragment in the triple data to be corrected to obtain an evaluation result of the search fragment. In some embodiments, the relevance can be used to characterize the relevance between the search fragment and the question.

[0086] In some embodiments, the accuracy can be used to characterize whether the search segment has no answer and / or whether the search segment is complete. Exemplarily, the accuracy can be characterized as: the search segment is obtained based on the question, and whether the answer generated based on the search segment is correct.

[0087] In step 202, second prompt information is constructed using a second prompt template based on the triple data to be corrected and the evaluation result of the search fragment.

[0088] In some embodiments, the search fragment, the user question (Query) and the evaluation result of the search fragment in the triple data to be corrected can be filled into the second prompt template to construct the second prompt information, wherein the second prompt information can be used to indicate the large model rewriting problem. For example, the second prompt information can be expressed as follows:

[0089] "Current search fragment: XXXX

[0090] Evaluation results of the current search fragment: accuracy is XX, relevance is XX

[0091] Please rewrite the query based on the user question (Query) and the evaluation results of the current search fragment so that the recalled answer can be correct. ".

[0092] In step 203, the question in the triple data to be corrected is rewritten based on the second prompt information and the large model to obtain a candidate question, and a retrieval fragment is generated based on the candidate question.

[0093] In some embodiments, the second prompt information can be input into the big model, and the big model rewrites the question in the triple data to be corrected based on the second prompt information, uses the rewritten question as a candidate question, and retrieves related fragments based on the candidate question, and uses the retrieved fragments as the retrieval fragments corresponding to the candidate question.

[0094] In step 204, a first answer is generated using a large model based on the candidate question and the generated search snippet.

[0095] In some embodiments, the candidate question and the generated search fragment can be filled into a preset prompt template to construct prompt information, which is input into the macro model, and the prompt information can be used to instruct the macro model to generate an answer. The macro model generates a corresponding answer based on the prompt information, in combination with the candidate question and the generated search fragment, i.e., the first answer.

[0096] Optionally, in some embodiments, based on the candidate questions and the generated search fragments, multiple large models can be used to generate multiple candidate answers; and a first answer is obtained from the multiple candidate answers. Exemplarily, a prompt template can be used to construct prompt information, for example, the above-mentioned candidate questions and the generated search fragments are filled into a preset prompt template to construct the prompt information. The prompt information is input into multiple large models, and each large model generates a corresponding answer based on the prompt information, combined with the candidate questions and the generated search fragments, that is, the candidate answers generated by each large model are obtained, and then the first answer can be obtained from these multiple candidate answers. For example, the best answer can be selected from the multiple candidate answers as the first answer.

[0097] In some embodiments, a third prompt information can be constructed using a third prompt template, and the third prompt information is used to instruct the large model to score multiple candidate answers on at least one scoring indicator dimension; wherein the at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the search fragment, completeness of the answer, and readability of the answer format. Based on the third prompt information, multiple large models are used to score the above multiple candidate answers on at least one scoring indicator dimension to obtain scoring information of multiple candidate answers; based on the scoring information of the multiple candidate answers, the first answer is obtained from the multiple candidate answers.

[0098] Exemplarily, multiple large models can be used to score multiple candidate answers on at least one scoring indicator dimension. In one possible implementation, each of the multiple large models can score multiple candidate answers on at least one scoring indicator dimension. For example, assuming that the multiple large models include large model A and large model B, and the candidate answers include candidate answer 1 and candidate answer 2, large model A can score candidate answer 1 on dimensions such as answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, readability of the answer format, etc., to obtain the score of candidate answer 1 on each dimension, and average the scores on these dimensions, and the average value obtained is used as the score of candidate answer 1 by large model A; large model A can score candidate answer 2 on dimensions such as answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, readability of the answer format, etc., to obtain the score of candidate answer 2 on each dimension, and average the scores on these dimensions, and the average value obtained is used as the score of candidate answer 2 by large model A. The large model B uses the method of the large model A to score the candidate answers 1 and 2, respectively, and obtains the scores of the large model B for the candidate answers 1 and the scores of the large model B for the candidate answers 2. Based on the scores of the large model A and the large model B for the candidate answers 1, the final score information of the candidate answers 1 is calculated, based on the scores of the large model A and the large model B for the candidate answers 2, the final score information of the candidate answers 2 is calculated, and based on the final score information of the candidate answers 1 and the final score information of the candidate answers 2, the candidate answers with the highest scores are selected from the candidate answers 1 and the candidate answers 2 as the first answer.

[0099] In another possible implementation, each of the multiple large models can score a candidate answer. Exemplarily, the large model can score the candidate answer generated by it. For example, assuming that the multiple large models include large model A and large model B, the candidate answers include candidate answer 1 and candidate answer 2, candidate answer 1 is generated by large model A, and candidate answer 2 is generated by large model B. Large model A can score candidate answer 1 in dimensions such as answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, and readability of the answer format, and obtain the score of candidate answer 1 in each dimension, and average the scores in these dimensions, and the average value obtained is used as the score of candidate answer 1 by large model A. Large model B scores candidate answer 2 in the manner of large model A above, and obtains the score of candidate answer 2 by large model B. Based on the score of candidate answer 1 and the score of candidate answer 2, the candidate answer with the highest score is selected from candidate answer 1 and candidate answer 2 as the first answer. Optionally, the score of the first answer needs to be greater than or equal to a threshold (such as 0.8).

[0100] That is to say, multiple large models can be used to review multiple candidate answers, for example, by letting multiple large models act as reviewers and voting for each large model, for example, letting each large model vote for all other large model pairs, and determining the candidate answer generated by the large model with the most votes as the first answer. It can be seen that by combining multiple large models to generate answers and using multiple large models for review, the best answer can be selected together, which can further ensure the quality of training data.

[0101] In step 205, based on the first answer and the candidate question generated by the large model, the generated search fragment is evaluated to obtain an evaluation result of the generated search fragment.

[0102] In some embodiments, the accuracy, relevance and association of the generated search fragments can be evaluated based on the first answer and candidate questions generated by the large model. Exemplarily, the accuracy can be characterized as: based on the rewritten question, a new search fragment is obtained, and whether the answer generated based on the new search fragment is correct. The relevance can be characterized as: the relevance between the rewritten question and the original question. The association can be characterized as: whether the new answer is related to the rewritten question.

[0103] In step 206, when the evaluation result of the generated search fragment does not meet the preset condition, a reflection iteration is performed to continue evaluating and rewriting the candidate question until the evaluation result of the generated search fragment meets the preset condition.

[0104] Exemplarily, the evaluation result of the searched fragment meeting the preset condition may include: the accuracy of the searched fragment is greater than or equal to the accuracy threshold; the relevance of the searched fragment is greater than or equal to the relevance threshold. Correspondingly, the evaluation result of the searched fragment not meeting the preset condition may include: the accuracy of the searched fragment is less than the accuracy threshold; the relevance of the searched fragment is less than the relevance threshold.

[0105] In the above embodiment, through iterative reflection of the large model, the retrieval errors can be automatically corrected, reducing the situation where training data cannot be generated due to retrieval errors.

[0106] Figure 3 is a flow chart of a method for generating training data based on a large model provided in an embodiment of the present disclosure. Figure 3 As shown, the large model-based training data generation method may include but is not limited to the following steps.

[0107] In step 301, based on the historical operation data, N triple data are obtained, where N is a positive integer; wherein each triple data includes a question, an answer corresponding to the question, and a search fragment.

[0108] Optionally, step 301 may be implemented in any of the implementations in the embodiments of the present disclosure, which is not limited in the embodiments of the present disclosure and will not be described in detail.

[0109] In step 302, triple data with incorrect answers are selected from the N triple data as triple data to be corrected.

[0110] Optionally, step 302 may be implemented in any of the implementations in the embodiments of the present disclosure, which is not limited in the embodiments of the present disclosure and will not be described in detail.

[0111] In step 303, the question in the triple data to be corrected is rewritten based on iterative reflection of the large model, and a corrected search fragment is generated based on the rewritten question and the large model.

[0112] Optionally, step 303 may be implemented in any of the implementations in the embodiments of the present disclosure, which is not limited in the embodiments of the present disclosure and will not be described in detail.

[0113] In step 304, multiple candidate answers are generated using multiple large models based on the rewritten question and the revised search fragment.

[0114] Exemplarily, after the question is rewritten, that is, the part that automatically corrects the search error, multiple large models can be used to generate answers based on the rewritten question and the corrected search fragment. Prompt information can be constructed using a prompt template, for example, the rewritten question and the corrected search fragment are filled into a preset prompt template to construct prompt information. The prompt information is input into multiple large models, and each large model generates a corresponding answer based on the prompt information, combined with the rewritten question and the corrected search fragment, that is, a candidate answer generated by each large model is obtained.

[0115] In step 305, a corrected answer is obtained from a plurality of candidate answers.

[0116] In some embodiments, multiple large models may be used to score multiple candidate answers on at least one scoring indicator dimension to obtain scoring information of multiple candidate answers; and corrected answers may be obtained from multiple candidate answers based on the scoring information of multiple candidate answers. For the optional implementation of obtaining corrected answers from multiple candidate answers, please refer to the optional implementation of obtaining the first answer from multiple candidate answers in the example after step 204 above, which will not be described in detail here.

[0117] That is to say, multiple large models can be used to review multiple candidate answers. For example, multiple large models can be used as reviewers to vote on each large model. For example, each large model can vote for all other large model pairs, and the candidate answer generated by the large model with the most votes will be determined as the corrected answer.

[0118] In step 306, based on the rewritten question, the revised search fragment and the corrected answer, the triple data to be revised is updated to obtain RAG training data.

[0119] Optionally, step 306 may be implemented in any of the implementations in the embodiments of the present disclosure, which is not limited in the embodiments of the present disclosure and will not be described in detail.

[0120] In the above embodiments, the present disclosure automatically generates a large amount of high-quality RAG training data based on the real historical operation data online, which can give full play to the advantages of the field scenario, and can continuously fit the real user data distribution after repeated optimization, and finally achieve better question-answering effects in real business scenarios. The present disclosure introduces a retrieval correction mechanism based on reflection, which can automatically correct the retrieval errors and reduce the situation where training data cannot be generated due to retrieval errors, thereby improving the quality of training data and generation efficiency, and solving the problem that retrieval errors cannot be automatically corrected in RAG data generation tools in the prior art. The present disclosure can also further ensure the quality of training data by combining multiple large models to generate answers, and using multiple large models for review, and jointly selecting the best answer.

[0121] The training data generation method based on the big model provided in the embodiment of the present disclosure can mainly include the following three parts: the online historical operation data acquisition stage, the search data automatic generation stage and the answer automatic generation stage. Figure 4 As shown, online historical operation data can be collected, and triple data of <user query, big model answer, retrieval fragment> can be parsed from the historical operation data, and triple data of incorrect answers can be filtered out through the big model.

[0122] Retrieval data automatic generation stage: Figure 4 As shown, after the triple data with incorrect answers are screened out by the large model, the accuracy, relevance and association of the retrieval fragments in these triple data can be evaluated. If the evaluation result of the retrieval fragment does not meet the preset conditions, such as the retrieval fragment has no answer or is incomplete, or the association between the retrieval fragment and the user's question is poor, then reflection iteration is performed to generate a new retrieval fragment. The new retrieval fragment generation method can be as follows:

[0123] The large model is used to continuously reflect and iterate to rewrite the query, so as to generate new search fragments. The specific generation method is as follows: the rewritten query is generated using the prompt template, and the relevant fragments are retrieved through the Retriever, and the large model generates new answers. Subsequently, the accuracy, relevance and association of the retrieved fragments are evaluated. If the search fragment has no answer or is incomplete, or the correlation with the new query is not high, reflection iteration is performed, and the query is continued to be evaluated and rewritten until a satisfactory search result is obtained. Among them, the relevance refers to: the correlation between the rewritten question and the original question; the accuracy refers to: based on the rewritten question, a new search fragment is obtained, and whether the answer generated based on the new search fragment is correct; the association refers to: whether the new answer is related to the rewritten query.

[0124] In the automatic answer generation stage: multiple large models can be used to generate answers, and multiple large models can be used for review to jointly select the best answer.

[0125] In summary, the model, method and idea adopted by the present disclosure are independent of the product and are applicable to any knowledge question-answering scenario. For example, the present disclosure can be applied to the knowledge question-answering model construction scenario, and can automatically correct the search results through reflection, and introduce multi-model answers and reviews to gradually improve the question-answering effect, so that the answers are more in line with user habits.

[0126] Figure 5 Flowchart of the knowledge question answering model training method provided by the embodiment of the present disclosure. Figure 5 As shown, the knowledge question answering model training method may include but is not limited to the following steps.

[0127] In step 501, training data is obtained.

[0128] In the embodiments of the present disclosure, the training data can be generated based on the large model-based training data generation method described in any of the aforementioned embodiments, which will not be repeated here.

[0129] In step 502, the knowledge question answering model is trained based on the training data.

[0130] In the above embodiment, a large amount of high-quality RAG training data can be automatically generated based on historical operation data. The knowledge question answering model is trained based on the RAG training data to improve the effect of the knowledge question answering model in various business scenarios.

[0131] Figure 6 is a block diagram of a training data generation device based on a large model provided by an embodiment of the present disclosure. Figure 6As shown, the training data generating device based on the large model may include: an acquisition module 601, a screening module 602, a rewriting module 603, a generation module 604 and an update module 605.

[0132] The acquisition module 601 is used to acquire N triple data based on the historical operation data, where N is a positive integer. Each triple data includes a question, an answer corresponding to the question, and a search fragment.

[0133] The screening module 602 is used to screen out the triple data with incorrect answers from the N triple data as the triple data to be corrected. In some embodiments, the screening module 602 is used to: construct first prompt information using a first prompt template based on the question in the N triple data and the search fragment corresponding to the question; obtain a reference answer corresponding to the question based on the first prompt information and the large model; and screen out the triple data with incorrect answers from the N triple data as the triple data to be corrected based on the answer corresponding to the question and the reference answer corresponding to the question.

[0134] The rewriting module 603 is used to rewrite the questions in the triple data to be corrected based on the iterative reflection of the big model, and generate a corrected search fragment based on the rewritten questions and the big model.

[0135] In some embodiments, the rewriting module 603 is used to: evaluate the retrieval fragment in the triple data to be corrected to obtain an evaluation result of the retrieval fragment; based on the evaluation results of the triple data to be corrected and the retrieval fragment, use the second prompt template to construct a second prompt information; the second prompt information is used to indicate the large model to rewrite the problem; based on the second prompt information and the large model, rewrite the question in the triple data to be corrected to obtain a candidate question, and generate a retrieval fragment based on the candidate question; based on the candidate question and the generated retrieval fragment, use the large model to generate a first answer; based on the first answer generated by the large model and the candidate question, evaluate the generated retrieval fragment to obtain an evaluation result of the generated retrieval fragment; when the evaluation result of the generated retrieval fragment does not meet the preset conditions, reflect and iterate, continue to evaluate and rewrite the candidate question until the evaluation result of the generated retrieval fragment meets the preset conditions.

[0136] In some embodiments, the rewriting module 603 is used to: based on the triple data to be corrected, evaluate the accuracy and / or relevance of the retrieval fragment in the triple data to be corrected, and obtain an evaluation result of the retrieval fragment, the evaluation result including accuracy and / or relevance; wherein, the accuracy is used to characterize whether the retrieval fragment has no answer and / or whether the retrieval fragment is complete, and the relevance is used to characterize the relevance between the retrieval fragment and the question.

[0137] In some embodiments, the rewriting module 603 is used to: generate multiple candidate answers based on the candidate questions and the generated search fragments using multiple large models; and obtain a first answer from the multiple candidate answers.

[0138] In some embodiments, the rewriting module 603 is used to: construct a third prompt information using a third prompt template, the third prompt information is used to instruct the large model to score multiple candidate answers on at least one scoring indicator dimension; at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, and readability of the answer format; based on the third prompt information, use multiple large models to score multiple candidate answers on at least one scoring indicator dimension to obtain scoring information of multiple candidate answers; obtain a first answer from multiple candidate answers based on the scoring information of multiple candidate answers.

[0139] The generation module 604 is used to generate a corrected answer corresponding to the rewritten question using the large model based on the rewritten question and the corrected search fragment.

[0140] In some embodiments, the generation module 604 is used to: generate multiple candidate answers based on the rewritten question and the corrected search fragment using multiple large models; and obtain a corrected answer from the multiple candidate answers.

[0141] In some embodiments, the generation module 604 is used to: use multiple large models to score multiple candidate answers on at least one scoring indicator dimension to obtain scoring information of multiple candidate answers; at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, and readability of the answer format; based on the scoring information of multiple candidate answers, obtain corrected answers from multiple candidate answers.

[0142] The updating module 605 is used to update the triple data to be corrected based on the rewritten question, the corrected search fragment and the corrected answer to obtain the search enhancement and generate RAG training data.

[0143] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0144] Figure 7 is a block diagram of a knowledge question answering model training device provided by an embodiment of the present disclosure. Figure 7 As shown, the knowledge question and answer model training device may include: an acquisition module 701 and a training module 702.

[0145] The acquisition module 701 is used to acquire training data. In the embodiments of the present disclosure, the training data can be generated based on the large model-based training data generation method described in any of the above embodiments.

[0146] The training module 702 is used to train the knowledge question answering model based on the training data.

[0147] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0148] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0149] like Figure 8 , is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0150] like Figure 8 As shown, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 801 is taken as an example.

[0151] The memory 802 is a non-transitory computer-readable storage medium provided by the present disclosure. The memory stores instructions executable by at least one processor to enable the at least one processor to execute the large model-based training data generation method or knowledge question answering model training method provided by the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions, which are used to enable a computer to execute the large model-based training data generation method or knowledge question answering model training method provided by the present disclosure.

[0152] The memory 802 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the large model-based training data generation method or the knowledge question answering model training method in the embodiments of the present disclosure (for example, the attached Figure 6 The processor 801 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 802, that is, the training data generation method based on the large model or the knowledge question and answer model training method in the above method embodiment is realized.

[0153] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] The electronic device may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0155] The input device 803 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 804 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0156] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0162] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating training data based on a large model, comprising: Based on the historical operation data, N triple data are obtained, where N is a positive integer; wherein each triple data includes a question, an answer corresponding to the question, and a search fragment; Filter out triple data with incorrect answers from the N triple data as triple data to be corrected; Rewrite the question in the triple data to be corrected based on iterative reflection of the large model, and generate a corrected search fragment based on the rewritten question and the large model; Based on the rewritten question and the revised search fragment, using the large model to generate a corrected answer corresponding to the rewritten question; Based on the rewritten question, the revised search fragment and the corrected answer, the triple data to be revised is updated to obtain the retrieval enhancement generation RAG training data.

2. The method of claim 1, wherein: The step of selecting the triple data with wrong answers from the N triple data as the triple data to be corrected includes: Based on the questions in the N triple data and the search segments corresponding to the questions, construct first prompt information using a first prompt template; Based on the first prompt information and the large model, obtaining a reference answer corresponding to the question; Based on the answer corresponding to the question and the reference answer corresponding to the question, the triple data with incorrect answers are screened out from the N triple data as the triple data to be corrected.

3. The method of claim 1, wherein: The iterative reflection based on the large model rewrites the problems in the triple data to be corrected, including: Evaluating the search fragment in the triple data to be corrected to obtain an evaluation result of the search fragment; Based on the triple data to be corrected and the evaluation result of the search fragment, construct second prompt information using a second prompt template; the second prompt information is used to indicate the large model rewriting problem; Rewrite the question in the to-be-corrected triple data based on the second prompt information and the large model to obtain a candidate question, and generate a search fragment based on the candidate question; Based on the candidate question and the generated search fragment, using the large model to generate a first answer; Based on the first answer generated by the large model and the candidate question, the generated search fragment is evaluated to obtain an evaluation result of the generated search fragment; In the case that the evaluation result of the generated search fragment does not meet the preset condition, iterative reflection is performed to continue evaluating and rewriting the candidate question until the evaluation result of the generated search fragment meets the preset condition.

4. The method of claim 3, wherein: The evaluation results include accuracy and / or relevance; The accuracy is used to characterize whether the search segment has no answer and / or whether the search segment is complete, and the relevance is used to characterize the relevance between the search segment and the question.

5. The method according to claim 3 or 4, wherein: The step of generating a first answer based on the candidate question and the generated search fragment using the large model includes: Based on the candidate questions and the generated search fragments, multiple large models are used to generate multiple candidate answers; The first answer is obtained from the multiple candidate answers.

6. The method of claim 5, wherein: The obtaining the first answer from the multiple candidate answers includes: Constructing third prompt information using a third prompt template, wherein the third prompt information is used to instruct the large model to score the multiple candidate answers on at least one scoring indicator dimension; the at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, and readability of the answer format; Based on the third prompt information, the multiple large models are used to score the multiple candidate answers on the at least one scoring indicator dimension to obtain scoring information of the multiple candidate answers; Based on the score information of the multiple candidate answers, the first answer is obtained from the multiple candidate answers.

7. The method of claim 1, wherein: The step of generating a corrected answer corresponding to the rewritten question using the large model based on the rewritten question and the corrected search fragment includes: Based on the rewritten question and the modified search fragment, multiple large models are used to generate multiple candidate answers; The corrected answer is obtained from the plurality of candidate answers.

8. The method of claim 7, wherein: The obtaining the corrected answer from the plurality of candidate answers comprises: The multiple large models are used to score the multiple candidate answers on at least one scoring indicator dimension to obtain scoring information of the multiple candidate answers; the at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the search fragment, completeness of the answer, and readability of the answer format; The corrected answer is obtained from the plurality of candidate answers based on the score information of the plurality of candidate answers.

9. A knowledge question answering model training method, comprising: Acquire training data, where the training data is generated based on the large model-based training data generation method according to any one of claims 1 to 8; The knowledge question answering model is trained based on the training data.

10. A training data generation device based on a large model, comprising: An acquisition module, configured to acquire N triple data based on historical operation data, where N is a positive integer; wherein each triple data includes a question, an answer corresponding to the question, and a search fragment; A screening module, used for screening out triple data with wrong answers from the N triple data as triple data to be corrected; A rewriting module, used for rewriting the questions in the triple data to be corrected based on the iterative reflection of the large model, and generating a corrected search fragment based on the rewritten questions and the large model; A generating module, configured to generate a corrected answer corresponding to the rewritten question using the large model based on the rewritten question and the corrected search fragment; An updating module is used to update the triple data to be corrected based on the rewritten question, the corrected search fragment and the corrected answer to obtain retrieval enhancement and generate RAG training data.

11. The device of claim 10, wherein: The screening module is used to: Based on the questions in the N triple data and the search segments corresponding to the questions, construct first prompt information using a first prompt template; Based on the first prompt information and the large model, obtaining a reference answer corresponding to the question; Based on the answer corresponding to the question and the reference answer corresponding to the question, the triple data with incorrect answers are screened out from the N triple data as the triple data to be corrected.

12. The device of claim 10, wherein: The rewriting module is used for: Evaluating the search fragment in the triple data to be corrected to obtain an evaluation result of the search fragment; Based on the triple data to be corrected and the evaluation result of the search fragment, construct second prompt information using a second prompt template; The second prompt information is used to indicate the large model rewriting problem; Rewrite the question in the to-be-corrected triple data based on the second prompt information and the large model to obtain a candidate question, and generate a search fragment based on the candidate question; Based on the candidate question and the generated search fragment, using the large model to generate a first answer; Based on the first answer generated by the large model and the candidate question, the generated search fragment is evaluated to obtain an evaluation result of the generated search fragment; In the case that the evaluation result of the generated search fragment does not meet the preset condition, iterative reflection is performed to continue evaluating and rewriting the candidate question until the evaluation result of the generated search fragment meets the preset condition.

13. The device of claim 12, wherein: The evaluation results include accuracy and / or relevance; The accuracy is used to characterize whether the search segment has no answer and / or whether the search segment is complete, and the relevance is used to characterize the relevance between the search segment and the question.

14. The device according to claim 12 or 13, wherein: The rewriting module is used for: Based on the candidate questions and the generated search fragments, multiple large models are used to generate multiple candidate answers; The first answer is obtained from the multiple candidate answers.

15. The device of claim 14, wherein: The rewriting module is used for: Constructing third prompt information using a third prompt template, wherein the third prompt information is used to instruct the large model to score the multiple candidate answers on at least one scoring indicator dimension; the at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the retrieval fragment, completeness of the answer, and readability of the answer format; Based on the third prompt information, the multiple large models are used to score the multiple candidate answers on the at least one scoring indicator dimension to obtain scoring information of the multiple candidate answers; Based on the score information of the multiple candidate answers, the first answer is obtained from the multiple candidate answers.

16. The device of claim 10, wherein: The generation module is used for: Based on the rewritten question and the modified search fragment, multiple large models are used to generate multiple candidate answers; The corrected answer is obtained from the plurality of candidate answers.

17. The device of claim 16, wherein: The generation module is used for: The multiple large models are used to score the multiple candidate answers on at least one scoring indicator dimension to obtain scoring information of the multiple candidate answers; the at least one scoring indicator dimension includes one or more of the following: answer accuracy, relevance of the answer to the search fragment, completeness of the answer, and readability of the answer format; The corrected answer is obtained from the plurality of candidate answers based on the score information of the plurality of candidate answers.

18. A knowledge question answering model training device, comprising: An acquisition module, used for acquiring training data, wherein the training data is generated based on the large model-based training data generation method according to any one of claims 1 to 8; A training module is used to train the knowledge question answering model based on the training data.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8 and 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8 and 9.

21. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 and 9 are implemented.

Images