Evaluation method and device for retrieval enhancement generation application, equipment and medium

Abstract

The embodiment of the invention provides an evaluation method and device for a retrieval enhancement generation application, equipment and a medium, and relates to the field of artificial intelligence. The evaluation method comprises the steps that input question information is input into the retrieval enhancement generation application, and output answer information is generated; inputting the output answer information and the input question information into an evaluation model to generate an evaluation score; the evaluation model is trained according to the following mode: constructing correct training data and defect training data based on the target domain knowledge base; generating preference training data based on the correct training data and the defect training data; and carrying out pre-training, supervision fine tuning and preference training on the first large language model to obtain an evaluation model. According to the method, the evaluation model is trained based on the training data, target domain knowledge is fused in the training stage, the evaluation model is kept consistent with human preferences through supervision fine tuning and preference training, and answers generated by the retrieval enhancement generation application can be better evaluated.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and more specifically, to an evaluation method, apparatus, device, and medium for retrieval enhancement generation applications. Background Art

[0002] The evaluation of generation tasks in natural language processing models has always been a major challenge, but the evaluation indicators (such as ROUGE, BLEU, etc.) cannot evaluate the Retrieval Augmented Generation (RAG) application well. On the one hand, some indicators are in the form of multiple-choice questions, which are actually used to evaluate the model's understanding ability; on the other hand, these indicators are often very different from the manual evaluation results and cannot accurately reflect the model performance.

[0003] Existing specialized evaluation methods for retrieval enhancement generation applications usually evaluate from three dimensions: fidelity, answer relevance, and context relevance. However, both fidelity and context relevance depend on the context, and obtaining accurate and comprehensive context itself is an item to be evaluated, which leads to poor evaluation results for retrieval enhancement generation applications. Therefore, how to better evaluate answers generated based on retrieval enhancement generation applications has become an urgent problem to be solved in this field. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and medium for evaluating a retrieval enhancement generation application, aiming to solve the problem of how to better evaluate answers generated based on the retrieval enhancement generation application.

[0005] A first aspect of an embodiment of the present application provides a method for evaluating a search enhancement generation application, the method comprising:

[0006] Inputting the input question information into the search enhancement generation application to generate output answer information;

[0007] Inputting the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information;

[0008] The evaluation model is trained as follows:

[0009] Build correct training data and defective training data based on the target domain knowledge base;

[0010] Generate preference training data based on the correct training data and the defective training data;

[0011] Pre-training the first language model based on the target domain knowledge base to obtain a first target model;

[0012] Based on the correct training data and the general training data, supervised fine-tuning is performed on the first target model to obtain a second target model;

[0013] Based on the preference training data, the second target model is trained to obtain the evaluation model.

[0014] In an optional implementation, constructing correct training data based on the target domain knowledge base includes:

[0015] Segmenting the text information in the target domain knowledge base according to the text format to obtain multiple text information;

[0016] Inputting the text information into the first language model, processing the text information based on the prompt word, and generating first question information and first answer information corresponding to the text information;

[0017] Combining each text information and the first question information and the first answer information corresponding to each text information into initial training data;

[0018] The initial training data is deduplicated to obtain the correct training data.

[0019] In an optional implementation, constructing defect training data based on the target domain knowledge base includes:

[0020] Based on the text information in the correct training data, construct first defective training data, where the first defective training data is used to represent erroneous training data generated due to unreasonable segmentation;

[0021] Based on the text information in the target domain knowledge base, construct second defect training data, where the second defect training data is used to represent erroneous training data generated by under-recall;

[0022] Based on the problem information in the correct training data and the text information in the target domain knowledge base, construct third defective training data, wherein the third defective training data is used to represent the erroneous training data generated due to the fact that the text information does not correspond to the problem information;

[0023] The first defect training data, the second defect training data, and the third defect training data are combined into the defect training data.

[0024] In an optional implementation manner, constructing the first defect training data based on the text information in the correct training data includes:

[0025] Randomly segmenting the text information in the correct training data to obtain a plurality of first text information;

[0026] Inputting the first text information into a second language model, processing the first text information based on a prompt word, and generating second question information and second answer information corresponding to the first text information;

[0027] Each first text information and the second question information and the second answer information corresponding to each first text information are combined into the first defect training data.

[0028] In an optional implementation, the constructing second defect training data based on the text information in the target domain knowledge base includes:

[0029] Segmenting the text information in the target domain knowledge base according to the text format, and randomly cutting the text information in the target domain knowledge base to obtain a plurality of second text information;

[0030] Inputting the second text information into a vector model, searching based on the question information in the correct training data, and obtaining third text information;

[0031] Inputting the question information in the correct training data and the third text information into the second largest language model to obtain third answer information;

[0032] The question information in the correct training data, the third text information and the third answer information are combined into the second defect training data.

[0033] In an optional implementation manner, constructing the third defect training data based on the problem information in the correct training data and the text information in the target domain knowledge base includes:

[0034] Inputting the question information in the correct training data into the second largest language model to generate fourth answer information; combining the question information in the correct training data and the fourth answer information into first sub-data;

[0035] Input the third question information and the fourth text information into the second language model to generate fifth answer information, and combine the third question information, the fourth text information and the fifth answer information into second sub-data, wherein the third question information is question information in any one of the correct training data, and the fourth text information is text information in any other correct training data that does not correspond to the third question information;

[0036] The first sub-data and the second sub-data are respectively screened based on the target keyword, and the screened first sub-data and the second sub-data are combined into the third defect training data.

[0037] In an optional implementation, generating the preference training data based on the correct training data and the defective training data includes:

[0038] Based on the correct training data, the defective training data is deduplicated so that question information of the defective training data is different from question information of the correct training data, and / or question information of the defective training data is different from answer information of the correct training data;

[0039] Combining different answer information corresponding to the target question information in all correct training data into a correct answer information set, and combining different answer information corresponding to the target question information in all incorrect training data into an incorrect answer information set, wherein the target question information is the question information in any correct training data;

[0040] The target question information, the correct answer information set, and the incorrect answer information set are combined into the preference training data.

[0041] A second aspect of an embodiment of the present application provides an evaluation device for a search enhancement generation application, the device comprising:

[0042] A search enhancement generation module, used for inputting input question information into a search enhancement generation application and generating output answer information;

[0043] An evaluation module, used to input the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information;

[0044] The evaluation model training module is used to train the evaluation model, wherein the evaluation model is trained in the following manner:

[0045] Build correct training data and defective training data based on the target domain knowledge base;

[0046] Generate preference training data based on the correct training data and the defective training data;

[0047] Pre-training the first language model based on the target domain knowledge base to obtain a first target model;

[0048] Based on the correct training data and the general training data, supervised fine-tuning is performed on the first target model to obtain a second target model;

[0049] Based on the preference training data, the second target model is trained to obtain the evaluation model.

[0050] In an optional implementation, the evaluation model training module includes:

[0051] The first segmentation module is used to segment the text information in the target domain knowledge base according to the text format to obtain multiple text information;

[0052] A first generating submodule, configured to input the text information into the first language model, process the text information based on a prompt word, and generate first question information and first answer information corresponding to the text information;

[0053] An initial training data submodule, used to combine each text information and the first question information and the first answer information corresponding to each text information into initial training data;

[0054] The correct training data submodule is used to perform deduplication processing on the initial training data to obtain the correct training data.

[0055] In an optional implementation, the evaluation model training module further includes:

[0056] A first defect training data submodule, used to construct first defect training data based on text information in the correct training data, wherein the first defect training data is used to represent erroneous training data generated due to unreasonable segmentation;

[0057] A second defect training data submodule, used to construct second defect training data based on text information in the target domain knowledge base, wherein the second defect training data is used to represent erroneous training data generated by under-recall;

[0058] A third defect training data submodule, for constructing third defect training data based on the problem information in the correct training data and the text information in the target domain knowledge base, wherein the third defect training data is used to represent the error training data generated due to the non-correspondence between the text information and the problem information;

[0059] The defect training data submodule is used to combine the first defect training data, the second defect training data and the third defect training data into the defect training data.

[0060] In an optional implementation, the first defect training data submodule includes:

[0061] A second segmentation unit, used for randomly segmenting the text information in the correct training data to obtain a plurality of first text information;

[0062] a second generating unit, configured to input the first text information into a second language model, process the first text information based on a prompt word, and generate second question information and second answer information corresponding to the first text information;

[0063] The first defect training data unit is used to combine each first text information and the second question information and the second answer information corresponding to each first text information into the first defect training data.

[0064] In an optional implementation, the second defect training data submodule includes:

[0065] A third segmentation unit is used to segment the text information in the target domain knowledge base according to the text format, and randomly cut the text information in the target domain knowledge base to obtain a plurality of second text information;

[0066] A third generating unit, configured to input the second text information into a vector model, and retrieve the question information in the correct training data to obtain third text information;

[0067] a fourth generating unit, configured to input the question information in the correct training data and the third text information into a second language model to obtain third answer information;

[0068] The second defect training data unit is used to combine the question information in the correct training data, the third text information and the third answer information into the second defect training data.

[0069] In an optional implementation, the third defect training data submodule includes:

[0070] A first sub-data unit is used to input the question information in the correct training data into a second language model to generate fourth answer information; and combine the question information in the correct training data and the fourth answer information into the first sub-data;

[0071] A second sub-data unit is used to input the third question information and the fourth text information into a second language model to generate fifth answer information, and combine the third question information, the fourth text information and the fifth answer information into second sub-data, wherein the third question information is question information in any one of the correct training data, and the fourth text information is text information in any other correct training data that does not correspond to the third question information;

[0072] The third defect training data unit is used to filter the first sub-data and the second sub-data based on target keywords, and combine the filtered first sub-data and the filtered second sub-data into the third defect training data.

[0073] In an optional implementation, the evaluation model training module further includes:

[0074] a deduplication submodule, configured to perform deduplication processing on the defective training data based on the correct training data, so that the question information of the defective training data is different from the question information of the correct training data, and / or the question information of the defective training data is different from the answer information of the correct training data;

[0075] An answer combination submodule is used to combine different answer information corresponding to the target question information in all correct training data into a correct answer information set, and to combine different answer information corresponding to the target question information in all incorrect training data into an incorrect answer information set, wherein the target question information is question information in any correct training data;

[0076] The preference training data submodule is used to combine the target question information, the correct answer information set, and the incorrect answer information set into the preference training data.

[0077] A third aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the evaluation method for the retrieval enhancement generation application described in any one of the first aspect of the embodiments of the present application.

[0078] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps in the evaluation method for the retrieval enhancement generation application described in any one of the first aspect of the embodiment of the present application are implemented.

[0079] Beneficial effects:

[0080] This application constructs training data by processing the target domain knowledge base, and obtains an evaluation model based on the training data. Since the target domain knowledge is integrated into the training phase, the evaluation model is kept consistent with human preferences through supervised fine-tuning and preference training, which can better evaluate the answers generated by the retrieval enhancement generation application, thereby reducing the workload required for the evaluation of the retrieval enhancement generation application and improving the efficiency of development iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0082] Figure 1 It is a flow chart of the training steps of an evaluation model in an evaluation method for a retrieval enhancement generation application proposed in an embodiment of the present application;

[0083] Figure 2 It is a structural schematic diagram of an evaluation device for a search enhancement generation application proposed in an embodiment of the present application;

[0084] Figure 3 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0086] In the drawings, the size of the constituent elements, the thickness of the layer or the area may be exaggerated for the sake of clarity. Therefore, any implementation of the present disclosure is not necessarily limited to the size shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect the true proportions. In addition, the drawings schematically show ideal examples, and any implementation of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0087] First, some professional terms involved in the embodiments of this application are explained:

[0088] Natural Language Processing (NLP) is a discipline that studies how to enable computers to understand, analyze, and generate human language. It involves processing text and speech data to achieve tasks such as text classification, sentiment analysis, machine translation, and question-answering systems.

[0089] Large Language Model (LLM) is a statistical model used to predict the probability distribution of the next word or character in a given context. LLM can be built by learning a large amount of text data and can capture the regularities and structures in the language. LLM is widely used in many NLP tasks, such as machine translation, speech recognition, automatic summarization, etc. By using LLM, we can generate coherent and fluent sentences, complete user input to provide better suggestions, and also evaluate whether a given sentence conforms to grammatical and semantic rules.

[0090] Question Answering (QA), a typical NLP application, is divided into extractive QA and generative QA. The answer to extractive QA appears in the original text, while the answer to generative QA requires the respondent to understand the semantics of the original text and organize the answer in his own language.

[0091] Retrieval Augmented Generation (RAG) is a paradigm for building knowledge-intensive QA. The original text provided by conventional QA is usually not very long, while RAG needs to obtain question-related content from a large amount of text, understand it, and generate answers. It is usually divided into two main steps, retrieval and generation. In the retrieval stage, the context related to the user's question is retrieved in the knowledge base through keywords, sentence vector similarity retrieval and other technologies. In the generation stage, the user's question and context are used to generate answers by LLM.

[0092] ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a metric used to evaluate the effect of generated text. It is usually used in automatic summarization. It measures the quality of automatically generated summaries by comparing the degree of overlap between the summaries generated by the system and the reference summaries.

[0093] BLEU (Bilingual Evaluation Understudy) is an indicator used to evaluate the quality of machine translation. It measures the accuracy of translation by comparing the similarity between the automatic translation result and the reference answer.

[0094] A knowledge base is a collection of standardized operating procedures, terminology libraries, knowledge documents, usage examples, etc. within an enterprise. Many application scenarios are based on the knowledge base. For example, in an intelligent customer service system, users may use a variety of different questioning methods to inquire about billing models, function descriptions, operating procedures, etc. in the knowledge base.

[0095] Context is a text snippet in the knowledge base used to support question answering in RAG applications. For example, when a user asks for instructions on a specific function of the system, any text snippet in the knowledge base that can be used to answer this question can be the context.

[0096] In related technologies, the evaluation of generation tasks in NLP has always been a major problem. Before the rise of large language models and related applications, the ROUGE series of indicators were often used to evaluate automatic summarization tasks, and BLEU was used to evaluate translation tasks. After the rise of large language models, Eval, C-Eval, MMLU, CMMLU, HELM and other indicators were usually used to evaluate LLM, but these indicators cannot evaluate RAG applications well. One reason is that some indicators are multiple-choice questions, such as MMLU, C-Eval, etc., which are actually used to evaluate the model's understanding ability. Indicators such as BLEU and ROUGE are often quite different from manual evaluation results and cannot accurately reflect model performance.

[0097] After the release of OpenAI's ChatGPT, it is usually evaluated with the help of OpenAI's API. There are two specific ways. The first way is for QA applications that already know the context. The questions, context, and answers generated by the model (sometimes also reference answers) are often submitted to OpenAI's API interface, and GPT scores the answers generated by the model; the second way is to directly submit the questions and reference answers to OpenAI's API for scoring. This evaluation method has two disadvantages. One is that it relies on the ability of ChatGPT. In some scenarios, ChatGPT may not be very consistent with human scoring; the other is that the data of some internal scenarios is confidential, and calling the open API may cause data leakage. For the first method, there is an additional disadvantage: such an evaluation is equivalent to assuming that the correct context has been obtained, but for RAG applications, this assumption is often not true. Many factors such as the retrieval method used, the size of the context text fragment, and whether there is an intersection between the fragments may affect the final result.

[0098] In addition, there are also special evaluation methods for RAG applications, such as RAGAS (Retrieval-Augmented Generation Assessment), which evaluates from the following three dimensions:

[0099] Faithfulness: Measures whether the generated answer is faithful to the context provided. Fidelity means that the factual claims in the answer can be inferred from the context, avoiding generating content that is inconsistent with the given information (i.e. "hallucinations").

[0100] Answer Relevance: Evaluates whether the generated answer directly and appropriately answers the question asked. This metric does not consider factual correctness, but focuses on whether the answer is complete, does not contain redundant information, and is closely related to the question.

[0101] Context Relevance: Measures the relevance of the retrieved context to answering the question. Ideally, the context should contain only the information needed to answer the question and avoid including too much irrelevant content.

[0102] From the above description, it can be seen that the RAGAS evaluation method, in which Faithfulness and Context Relevance are both dependent on the context, and obtaining accurate and comprehensive context itself is an item to be evaluated.

[0103] In view of the evaluation dilemma mentioned above, manual methods are often used to check the effect of RAG applications, but this usually consumes a lot of manpower costs and reduces the efficiency of iteration.

[0104] In view of this, an embodiment of the present application proposes an evaluation method for a retrieval enhancement generation application, the method comprising the following steps: inputting question information into the retrieval enhancement generation application to generate output answer information; inputting the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information. Figure 1 A flowchart of the training steps of the evaluation model in the evaluation method of the retrieval enhancement generation application is shown, as Figure 1 As shown, the evaluation model is trained according to the following steps:

[0105] S101. Construct correct training data and defective training data based on the target domain knowledge base.

[0106] When the step S101 is specifically implemented, firstly, based on the target domain knowledge base, training data for training the evaluation model is constructed. The target domain knowledge base refers to a data information base composed of knowledge documents in a field related to the question information when the retrieval enhancement generation application processes the input question information. The training data includes question information, answer information and text information, that is, forming<question,context,answer> The information construction of triples, wherein the text information refers to the context in the document in the target domain knowledge base (a text fragment in the knowledge base used to support question answering), and the question information and the answer information are corresponding questions and answers generated based on the text information.

[0107] In the embodiments of the present application, the training data includes correct training data. The correct training data refers to the training data composed of a triple of the answer information, the question information, and the text information when the generated answer information is correctly obtained based on the text information. Since the training of the model requires both positive samples and negative samples, the training data further includes defective training data. The defective training data refers to the training data composed of a triple of the answer information, the question information, and the text information when the answer information does not represent an answer to the question information based on the text information.

[0108] In some optional embodiments, the correct training data is obtained as follows: First, the text information in the target domain knowledge base is segmented according to the text format to obtain a plurality of text information. Here, the text format refers to the format processing for dividing logic and content based on writing habits. Exemplarily, the segmentation according to the text format includes segmenting the text information in the target domain knowledge base by using double line breaks, line breaks, logical paragraphs, chapters, etc. In the embodiments of the present application, by segmenting according to the text format, the relevant meanings and contents in the obtained text information can be segmented into the same text information, so that each text information can express a complete meaning, thereby helping to generate correct answer information.

[0109] After obtaining a plurality of text information, the text information is input into the first large language model, and the text information is processed based on the prompt word Prompt to generate the corresponding first question information and first answer information of the text information. Here, the first large language model is the large language model included in the retrieval-augmented generation application, and the prompt word is used to instruct the large language model to automatically generate a set of question information and answer information related to the input text information. Exemplarily, the prompt word may include the following content:

[0110] Please generate a question and an answer from the following text fragment, text fragment:

[0111] <text fragment>

[0112] ……

[0113] < / text fragment>

[0114] Please answer the question in JSON format, as follows:

[0115] {"question": "……", "answer": "……"}

[0116] After generating the question information and the answer information based on the prompt word, each text information and the first question information and the first answer information corresponding to each text information are combined into a Chosen <question,context,answer> The triples are used as initial training data. The initial training data is often used to optimize the retrieval model and the large language model in the retrieval enhancement generation application, so it is necessary to ensure accuracy as much as possible.

[0117] In addition, in the process of generating the initial training data, thousands or even tens of thousands of the initial training data are generally generated, and duplication may occur in the initial training data, which will affect the training effect of the evaluation model. Therefore, in the embodiment of the present application, after obtaining the initial training data, the initial training data is deduplicated to obtain the correct training data. Specifically, since the user does not need to manually input text information in the process of generating answer information based on the input question information processing of the large language model, the training process of the evaluation model does not need to pay attention to whether the text information in the initial training data is repeated. Therefore, when deduplicating, the question information and answer information of all the initial training data are deduplicated to ensure that there is only one identical data in the question information of all the initial training data, and there is only one identical data in the answer information of all the initial training data.

[0118] In an embodiment of the present application, the defect training data includes first defect training data, second defect training data, and third defect training data for characterizing defect answer information generated due to different reasons. Specifically, the first defect training data is used to characterize erroneous training data generated due to unreasonable segmentation. Because the documents of the target domain knowledge base are segmented in an unreasonable way, the complete context that can be used to answer the question information is cut off, and the answer information generated by the large language model based on the cut text information cannot correctly answer the question information; the second defect training data is used to characterize erroneous training data generated due to insufficient recall. The insufficient recall means that the text information is quite different from the target domain knowledge, and the model does not have enough knowledge of the target domain when generating the answer information. The answer information generated by the model based on the insufficiently recalled text information cannot correctly answer the question information; the third defect training data is used to characterize erroneous training data generated due to the fact that the text information does not correspond to the question information, that is, when the model cannot answer based on the text information, it does not refuse to answer, but constructs an illusion answer, that is, forcibly generates irrelevant answer information.

[0119] In some optional implementations, after obtaining the correct training data, the first defect training data is constructed based on the text information in the correct training data. <question,context,answer>) is randomly segmented to obtain multiple first text information, and the random segmentation is not restricted by the text format. Therefore, the first text information obtained after random segmentation includes a large amount of text content that cannot represent a complete and unified meaning, so that the answer information obtained based on the first text information is a defect answer; then, the first text information is input into the second largest language model, and the first text information is processed based on the prompt word to generate the second question information and the second answer information corresponding to the first text information, wherein the second largest language model is an arbitrary large language model, such as Qwen-7B-Chat and other large language models; finally, each first text information and the second question information and the second answer information corresponding to each first text information are combined into the first defect training data. Exemplarily, the prompt words used to generate the second question information and the second answer information may include the following content:

[0120] Please base on the following context ( <context>< / context> The part inside the package), answer the questions.

[0121] question:……

[0122] Context:

[0123]

[0124] Please answer:

[0125] In some optional implementations, after obtaining the correct training data, the second defect training data is constructed based on the text information in the target domain knowledge base. Specifically, firstly, the text information in the target domain knowledge base is segmented according to the text format, and the text information in the target domain knowledge base is randomly cut to obtain a plurality of second text information, that is, the second text information includes text content containing complete meaning and text content that cannot represent a complete and unified meaning; the second text information is input into a vector model, and the third text information is retrieved based on the question information in the correct training data to obtain the third text information, the vector model is a model that is not optimized based on the target domain knowledge, and is used to convert the second text information into a vector form and perform corresponding retrieval to generate new text information. Exemplarily, the vector model can be a BGE-Large-zh model; the question information in the correct training data and the third text information are input into the second largest language model, and the new third text information is directly answered based on the second largest language model to obtain the third answer information.

[0126] Finally, the question information, the third text information and the third answer information in the correct training data are combined into the second defective training data. Through the above steps, the second defective training data is constructed to address the problem that the vector model does not understand the target domain knowledge well enough, resulting in insufficient recall. It should be noted that the third answer information is not necessarily a defective answer, which depends on whether the difference between the target domain knowledge and the general knowledge is large enough. If the difference is not large, the third answer information may also be the correct answer.

[0127] In some optional implementations, after obtaining the correct training data, the third defective training data is constructed based on the question information in the correct training data and the text information in the target domain knowledge base. Specifically, the question information in the correct training data is input into the second large language model to generate fourth answer information; the question information in the correct training data and the fourth answer information are combined into the first sub-data. Since the second large language model does not receive the corresponding text information, but directly answers based on the question information, the fourth answer information is a defective answer fabricated by the large language model in the absence of context.

[0128] Subsequently, the third question information and the fourth text information are input into the second large language model to generate fifth answer information, and the third question information, the fourth text information and the fifth answer information are combined into the second sub-data, wherein the third question information is the question information in any one of the correct training data, and the fourth text information is the text information in any other correct training data that does not correspond to the third question information. In this step, since the fourth text information input into the large language model is not the corresponding context and question with the third question information, the generated fifth answer information is a defective answer fabricated by the large language model under the condition of wrong context.

[0129] After obtaining the first sub-data and the second sub-data, since the large language model will refuse to answer when the text information does not correspond to the question information, and generate answer information for refusing to answer, and these answer information for refusing to answer are correct answers, when constructing the answer information of the third defect training data, the answer information for refusing to answer needs to be screened out. Specifically, the first sub-data and the second sub-data are screened based on the target keyword, and the screened first sub-data and the second sub-data are combined into the third defect training data. Among them, the target keyword is the answer information that represents the meaning of refusing to answer. Exemplarily, the target keyword can be information containing similar content or words such as "Sorry...cannot answer".

[0130] Finally, after constructing the first defect training data, the second defect training data, and the third defect training data, the first defect training data, the second defect training data, and the third defect training data are combined into the defect training data Rejected <question,context,answer > .

[0131] S102: Generate preference training data based on the correct training data and the defective training data.

[0132] When step S102 is specifically implemented, since there may be training data in the defective training data that is repeated with the correct training data, after obtaining the defective training data, the defective training data is deduplicated based on the correct training data, and data in the defective training data that are identical to the question information and answer information in the correct training data are removed, so that the question information of the defective training data is different from the question information of the correct training data, and / or the question information of the defective training data is different from the answer information of the correct training data.

[0133] Then, the target question information is selected in all correct training data. <question,context,answer> The corresponding different answer information is sorted into a list and combined into the correct answer information set Chosen answers , and the target question information is rejected in all wrong training data <question,context,answer> The corresponding different answer information is sorted into a list and combined into a wrong answer information set Rejected answers , the target question information is the question information in any correct training data. Finally, the target question information, the correct answer information set, and the wrong answer information set are combined into the preference training data <question,chosen answers ,rejected answers >.

[0134] S103. Pre-training a first large language model based on the target domain knowledge base to obtain a first target model.

[0135] When the step S103 is specifically implemented, the first large language model is pre-trained based on the target domain knowledge base, the first large language model is the large language model in the retrieval enhancement generation application, and the first large language model is a model that has been pre-trained once based on general knowledge. In the embodiment of the present application, the first large language model is pre-trained twice, and the documents of the target domain knowledge base are organized into the data format required for pre-training and input into the first large language model to integrate the target domain knowledge into the first large language model. Optionally, the first large language model is a commonly used pre-trained large language model of a scale of 2B, 7B, etc. For example, the first large language model can be a Qwen-7B model.

[0136] In an embodiment of the present application, by performing secondary pre-training on the first language model based on the target domain knowledge, the evaluation model obtained by subsequent training can more accurately score the answers generated by the retrieval enhancement generation application based on the target domain knowledge, thereby improving the evaluation performance of the evaluation model.

[0137] S104: Based on the correct training data and the general training data, supervised fine-tuning is performed on the first target model to obtain a second target model.

[0138] When step S104 is specifically implemented, the correct training data is used to perform supervised fine-tuning on the first target model so that the first target model can provide guidance on the form and style of answering a question. When the generated evaluation model evaluates the answer generated by the retrieval enhancement generation application, the evaluation model can identify whether the model's answer form and style are appropriate for a given question.

[0139] In addition, in order to prevent the problem of overfitting in the supervision and fine-tuning process of the first target model, general training data is added to the correct training data, and the correct training data and the general training data are input into the first target model together.

[0140] S105. Based on the preference training data, the second target model is trained to obtain the evaluation model.

[0141] When the step S105 is implemented, the preference training data constructed in the step S101 is obtained. <question,chosen answers ,rejected answers>, train the second target model, wherein the overall model structure and training method are consistent with the reward model of the large language model RLHF training stage, to obtain the evaluation model. In an embodiment of the present application, training the second target model through the preference training data can enable the evaluation model to know what kind of answers are better and what kind of answers are not good when evaluating the answers generated by the retrieval enhancement generation application, so that the evaluation model obtained by training has the ability to evaluate the effects at different stages of the development of the retrieval enhancement generation application, and realizes automatic evaluation of the retrieval enhancement generation application, thereby reducing the workload required for the evaluation of the retrieval enhancement generation application and improving the development iteration efficiency.

[0142] An embodiment of the present application provides an evaluation method, apparatus, device and medium for a retrieval enhancement generation application, the method comprising: inputting input question information into a retrieval enhancement generation application to generate output answer information; inputting the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information; wherein the evaluation model is trained in the following manner: constructing correct training data and defective training data based on a target domain knowledge base; generating preference training data based on the correct training data and the defective training data; pre-training a first large language model based on the target domain knowledge base to obtain a first target model; supervising and fine-tuning the first target model based on the correct training data and the general training data to obtain a second target model; and training the second target model based on the preference training data to obtain the evaluation model.

[0143] This application constructs training data by processing the target domain knowledge base, and obtains an evaluation model based on the training data. Since the target domain knowledge is integrated into the training phase, the evaluation model is kept consistent with human preferences through supervised fine-tuning and preference training, which can better evaluate the answers generated by the retrieval enhancement generation application, thereby reducing the workload required for the evaluation of the retrieval enhancement generation application and improving the efficiency of development iteration.

[0144] Based on the same inventive concept, the embodiment of the present application discloses an evaluation device for search enhancement generation application. Figure 2 A schematic diagram of the structure of an evaluation device for a search enhancement generation application proposed in an embodiment of the present application is shown. Figure 2 As shown, the device comprises:

[0145] A search enhancement generation module, used for inputting input question information into a search enhancement generation application and generating output answer information;

[0146] An evaluation module, used to input the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information;

[0147] The evaluation model training module is used to train the evaluation model, wherein the evaluation model is trained in the following manner:

[0148] Build correct training data and defective training data based on the target domain knowledge base;

[0149] Generate preference training data based on the correct training data and the defective training data;

[0150] Pre-training the first language model based on the target domain knowledge base to obtain a first target model;

[0151] Based on the correct training data and the general training data, supervised fine-tuning is performed on the first target model to obtain a second target model;

[0152] Based on the preference training data, the second target model is trained to obtain the evaluation model.

[0153] In an optional implementation, the evaluation model training module includes:

[0154] The first segmentation module is used to segment the text information in the target domain knowledge base according to the text format to obtain multiple text information;

[0155] A first generating submodule, configured to input the text information into the first language model, process the text information based on a prompt word, and generate first question information and first answer information corresponding to the text information;

[0156] An initial training data submodule, used to combine each text information and the first question information and the first answer information corresponding to each text information into initial training data;

[0157] The correct training data submodule is used to perform deduplication processing on the initial training data to obtain the correct training data.

[0158] In an optional implementation, the evaluation model training module further includes:

[0159] A first defect training data submodule, used to construct first defect training data based on text information in the correct training data, wherein the first defect training data is used to represent erroneous training data generated due to unreasonable segmentation;

[0160] A second defect training data submodule, used to construct second defect training data based on text information in the target domain knowledge base, wherein the second defect training data is used to represent erroneous training data generated by under-recall;

[0161] A third defect training data submodule, for constructing third defect training data based on the problem information in the correct training data and the text information in the target domain knowledge base, wherein the third defect training data is used to represent the error training data generated due to the non-correspondence between the text information and the problem information;

[0162] The defect training data submodule is used to combine the first defect training data, the second defect training data and the third defect training data into the defect training data.

[0163] In an optional implementation, the first defect training data submodule includes:

[0164] A second segmentation unit, used for randomly segmenting the text information in the correct training data to obtain a plurality of first text information;

[0165] a second generating unit, configured to input the first text information into a second language model, process the first text information based on a prompt word, and generate second question information and second answer information corresponding to the first text information;

[0166] The first defect training data unit is used to combine each first text information and the second question information and the second answer information corresponding to each first text information into the first defect training data.

[0167] In an optional implementation, the second defect training data submodule includes:

[0168] A third segmentation unit is used to segment the text information in the target domain knowledge base according to the text format, and randomly cut the text information in the target domain knowledge base to obtain a plurality of second text information;

[0169] A third generating unit, configured to input the second text information into a vector model, and retrieve the question information in the correct training data to obtain third text information;

[0170] a fourth generating unit, configured to input the question information in the correct training data and the third text information into a second language model to obtain third answer information;

[0171] The second defect training data unit is used to combine the question information in the correct training data, the third text information and the third answer information into the second defect training data.

[0172] In an optional implementation, the third defect training data submodule includes:

[0173] A first sub-data unit is used to input the question information in the correct training data into a second language model to generate fourth answer information; and combine the question information in the correct training data and the fourth answer information into the first sub-data;

[0174] A second sub-data unit is used to input the third question information and the fourth text information into a second language model to generate fifth answer information, and combine the third question information, the fourth text information and the fifth answer information into second sub-data, wherein the third question information is question information in any one of the correct training data, and the fourth text information is text information in any other correct training data that does not correspond to the third question information;

[0175] The third defect training data unit is used to filter the first sub-data and the second sub-data based on target keywords, and combine the filtered first sub-data and the filtered second sub-data into the third defect training data.

[0176] In an optional implementation, the evaluation model training module further includes:

[0177] a deduplication submodule, configured to perform deduplication processing on the defective training data based on the correct training data, so that the question information of the defective training data is different from the question information of the correct training data, and / or the question information of the defective training data is different from the answer information of the correct training data;

[0178] An answer combination submodule is used to combine different answer information corresponding to the target question information in all correct training data into a correct answer information set, and to combine different answer information corresponding to the target question information in all incorrect training data into an incorrect answer information set, wherein the target question information is question information in any correct training data;

[0179] The preference training data submodule is used to combine the target question information, the correct answer information set, and the incorrect answer information set into the preference training data.

[0180] Based on the same inventive concept, the present application embodiment discloses an electronic device, Figure 3 A schematic diagram of an electronic device proposed in an embodiment of the present application is shown, such as Figure 3As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory 110 and the processor 120 are connected to each other through a bus communication, the memory 110 stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set can be loaded and run on the processor 120 to implement the steps in the evaluation method of the retrieval enhancement generation application disclosed in the embodiment of the present application.

[0181] Based on the same inventive concept, an embodiment of the present application discloses a computer-readable storage medium, on which is stored at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the steps in the evaluation method of the retrieval enhancement generation application disclosed in the embodiment of the present application.

[0182] Based on the same inventive concept, an embodiment of the present application discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the evaluation method for the retrieval enhancement generation application disclosed in the embodiment of the present application.

[0183] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other.

[0184] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0187] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0188] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0189] The above is a detailed introduction to the evaluation method, device, equipment and medium for a retrieval enhancement generation application provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for evaluating a search enhancement generation application, characterized in that: The method comprises: Inputting the input question information into the search enhancement generation application to generate output answer information; Inputting the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information; The evaluation model is trained as follows: Build correct training data and defective training data based on the target domain knowledge base; Generate preference training data based on the correct training data and the defective training data; Pre-training the first language model based on the target domain knowledge base to obtain a first target model; Based on the correct training data and the general training data, supervised fine-tuning is performed on the first target model to obtain a second target model; Based on the preference training data, the second target model is trained to obtain the evaluation model.

2. The evaluation method for search enhancement generation application according to claim 1, characterized in that: The correct training data is constructed based on the target domain knowledge base, including: Segmenting the text information in the target domain knowledge base according to the text format to obtain multiple text information; Inputting the text information into the first language model, processing the text information based on the prompt word, and generating first question information and first answer information corresponding to the text information; Combining each text information and the first question information and the first answer information corresponding to each text information into initial training data; The initial training data is deduplicated to obtain the correct training data.

3. The evaluation method for search enhancement generation application according to claim 1, characterized in that: The defect training data is constructed based on the target domain knowledge base, including: Based on the text information in the correct training data, construct first defective training data, where the first defective training data is used to represent erroneous training data generated due to unreasonable segmentation; Based on the text information in the target domain knowledge base, construct second defect training data, where the second defect training data is used to represent erroneous training data generated by under-recall; Based on the problem information in the correct training data and the text information in the target domain knowledge base, construct third defective training data, wherein the third defective training data is used to represent the erroneous training data generated due to the fact that the text information does not correspond to the problem information; The first defect training data, the second defect training data, and the third defect training data are combined into the defect training data.

4. The evaluation method for search enhancement generation application according to claim 3, characterized in that: The step of constructing first defect training data based on text information in the correct training data includes: Randomly segmenting the text information in the correct training data to obtain a plurality of first text information; Inputting the first text information into a second language model, processing the first text information based on a prompt word, and generating second question information and second answer information corresponding to the first text information; Each first text information and the second question information and the second answer information corresponding to each first text information are combined into the first defect training data.

5. The evaluation method for search enhancement generation application according to claim 3, characterized in that: The step of constructing second defect training data based on the text information in the target domain knowledge base includes: Segmenting the text information in the target domain knowledge base according to the text format, and randomly cutting the text information in the target domain knowledge base to obtain a plurality of second text information; Inputting the second text information into a vector model, searching based on the question information in the correct training data, and obtaining third text information; Inputting the question information in the correct training data and the third text information into the second largest language model to obtain third answer information; The question information in the correct training data, the third text information and the third answer information are combined into the second defect training data.

6. The evaluation method for search enhancement generation application according to claim 3, characterized in that: The step of constructing the third defect training data based on the problem information in the correct training data and the text information in the target domain knowledge base includes: Inputting the question information in the correct training data into the second largest language model to generate fourth answer information; combining the question information in the correct training data and the fourth answer information into first sub-data; Input the third question information and the fourth text information into the second language model to generate fifth answer information, and combine the third question information, the fourth text information and the fifth answer information into second sub-data, wherein the third question information is question information in any one of the correct training data, and the fourth text information is text information in any other correct training data that does not correspond to the third question information; The first sub-data and the second sub-data are respectively screened based on the target keyword, and the screened first sub-data and the second sub-data are combined into the third defect training data.

7. The evaluation method for search enhancement generation application according to claim 1, characterized in that: The generating the preference training data based on the correct training data and the defective training data comprises: Based on the correct training data, the defective training data is deduplicated so that question information of the defective training data is different from question information of the correct training data, and / or question information of the defective training data is different from answer information of the correct training data; Combining different answer information corresponding to the target question information in all correct training data into a correct answer information set, and combining different answer information corresponding to the target question information in all incorrect training data into an incorrect answer information set, wherein the target question information is the question information in any correct training data; The target question information, the correct answer information set, and the incorrect answer information set are combined into the preference training data.

8. An evaluation device for search enhancement generation application, characterized in that: The device comprises: A search enhancement generation module, used for inputting input question information into a search enhancement generation application and generating output answer information; An evaluation module, used to input the output answer information and the input question information into an evaluation model to generate an evaluation score corresponding to the output answer information; The evaluation model training module is used to train the evaluation model, wherein the evaluation model is trained in the following manner: Build correct training data and defective training data based on the target domain knowledge base; Generate preference training data based on the correct training data and the defective training data; Pre-training the first language model based on the target domain knowledge base to obtain a first target model; Based on the correct training data and the general training data, supervised fine-tuning is performed on the first target model to obtain a second target model; Based on the preference training data, the second target model is trained to obtain the evaluation model.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the evaluation method for the retrieval enhancement generation application according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps in the evaluation method for the retrieval enhancement generation application described in any one of claims 1 to 7 are implemented.

Images