Automatic evaluation method and system for retrieval enhancement generation system

Abstract

The invention belongs to the technical field of artificial intelligence. According to the automatic evaluation method and system for the retrieval enhancement generation system, different answer evaluation algorithms are adopted for evaluation for answers generated by the retrieval enhancement generation system and answers in an evaluation data set, and a plurality of algorithm evaluation indexes are obtained; and calculating a plurality of overall evaluation indexes of each retrieval enhancement generation system, performing principal component analysis on all the algorithm evaluation indexes and the overall evaluation indexes, and further evaluating the RAG system. According to the method, the automation problem of document analysis and question and answer data generation is solved, high-quality question and answer pairs are provided, the question and answer quality and retrieval quality of the RAG system are automatically evaluated, and a more accurate automatic evaluation result is obtained by combining multiple evaluation indexes with a principal component analysis method.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an automated evaluation method for a retrieval enhancement generation system, an automated evaluation system for a retrieval enhancement generation system, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] RAG (Retrieval-Augmented Generation) is a natural language processing method that combines information retrieval and text generation. Specifically, RAG first extracts documents or information fragments related to the input question from a large knowledge base or document library through a retrieval module; then, the retrieved information is input into the generation module, which uses this information to build more accurate and detailed answers. The advantage of RAG technology is that it can dynamically obtain the latest information, expand the knowledge scope of the model, and improve the accuracy and contextual relevance of the answers. Therefore, RAG technology performs particularly well in tasks that require real-time knowledge updates and complex information processing, such as question-answering systems, dialogue generation, and text summarization.

[0004] The benefit of RAG technology for large models is that it significantly improves the model's knowledge coverage and answer accuracy by combining the dual mechanisms of retrieval and generation. When generating text, large models can dynamically retrieve the latest and most relevant information, making up for the limitations of relying solely on the inherent knowledge of training data; in addition, RAG can reduce the parameter scale requirements of large models because the information in the knowledge base does not need to be fully embedded in the model parameters, but is obtained in real time through retrieval. This not only improves the performance of the model, but also reduces the consumption of storage and computing resources. In order to enhance the performance of RAG, many techniques have been proposed, such as block length and the order of retrieval blocks in context.

[0005] However, like all systems, these systems require objective metrics to measure the performance of the end-to-end system. Evaluating RAG systems is important because it ensures the effectiveness of the integration of retrieval methods and generative models. The challenge of evaluating RAG systems is to compare the factuality, relevance to the question, and semantic similarity of the generated answers with the real answers. In the entire end-to-end evaluation process, if it is necessary to manually generate relevant evaluation datasets, it will consume a lot of manpower. Even so, the manually generated document evaluation datasets may not cover a complete and wide range of documents.

[0006] The original approach to evaluating RAG consisted of repurposing metrics used in machine translation tasks, such as BLEU, ROUGE, or METEOR, while text generation was evaluated using BERT Score.

[0007] However, the above-mentioned RAG evaluation method has the following problems: (1) A complete RAG evaluation system has not been established. The RAG system is relatively complex and has three links: document processing, document retrieval, and answer generation. A complete RAG evaluation system needs to evaluate all links. (2) The evaluation data set is difficult to construct. Since it is an evaluation, there must be a corresponding data set. However, since the RAG system needs to change according to the processed documents, the data set construction is dynamic, so it is difficult to fix a single data set for evaluation. Summary of the invention

[0008] In order to address the deficiencies in the prior art, the present invention provides an automated evaluation method and system for a retrieval enhancement generation system, which realizes the automation of document parsing and question and answer data generation, and provides high-quality question and answer pairs for the evaluation of the RAG system. It can automatically evaluate the question and answer quality and retrieval quality of the RAG system, and uses a variety of evaluation indicators combined with the principal component analysis method to obtain more accurate automated evaluation results.

[0009] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides an automated evaluation method for a retrieval enhancement generation system.

[0010] An automated evaluation method for a retrieval enhancement generation system includes the following steps: Obtaining an evaluation data set for a search enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; Input each question in the evaluation data set into each retrieval enhancement generation system cyclically, so that the retrieval enhancement generation system generates a corresponding answer; The answers generated by the search enhancement generation system and the answers in the evaluation data set are evaluated using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; Calculate multiple overall evaluation indicators for each retrieval enhancement generation system, perform principal component analysis on all the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

[0011] As a further limitation of the first aspect of the present invention, the construction of the evaluation data set includes: Select the document set for which the dataset needs to be built , document set From n documents ,in composition, ; Convert all documents to txt type, split each txt document by paragraph, and Can be split into several paragraphs ,in , indicating the of the documents paragraphs, the set of paragraphs for each document , It is a document The number of paragraphs in the current document set D. ; Select S paragraphs from the split paragraphs to form a paragraph group, and generate N paragraph groups in total, where S is an integer greater than or equal to 1. , the N groups of paragraphs generated are expressed as ; Input N groups of content into the powerful large model N times, and use the powerful large model to generate possible questions and answers for the content in each group of paragraphs input. ; After question generation and answer generation are completed, the question-answer pairs with semantic duplication or poor quality of question or answer generation are screened out, and the question, current paragraph group, and answer are formed into a total data set and stored: .

[0012] As a further limitation of the first aspect of the present invention, ROUGE-1, ROUGE-2, ROUGE-L, BertScore, BLUE and Meteor are used for evaluation to obtain multiple algorithm evaluation indicators.

[0013] As a further limitation of the first aspect of the present invention, the overall evaluation indicators include: context recall, context precision, answer semantic similarity, loyalty and answer relevance.

[0014] As a further limitation of the first aspect of the present invention, principal component analysis is performed on all of the algorithm evaluation indicators and the overall evaluation indicators, including: Standardize all indicator data to obtain a standardized data matrix ; According to the data matrix Calculate the covariance matrix , the covariance matrix It is a symmetric matrix, which is used to describe the relationship between various indicators; For the covariance matrix Perform eigenvalue decomposition to obtain eigenvalues; Calculate the cumulative explained variance according to the eigenvalues, and select multiple principal components with higher cumulative explained variance ratios; The standardized data matrix Project to selected In the principal component space, a new low-dimensional data representation is obtained: ; The weighted sum of the selected principal components is used as the score of the retrieval enhancement generation system: ,in, It is principal components, is the corresponding weight; or the selected first principal component is used as the score of the retrieval enhancement generation system.

[0015] As a further limitation of the first aspect of the present invention, each of the question-answer pairs includes a question, a current paragraph group and an answer, and the retrieval enhancement generation system can output the answer and the question source paragraph based on the question.

[0016] In a second aspect, the present invention provides an automated evaluation system for a retrieval enhancement generation system.

[0017] An automatic evaluation system for a search enhancement generation system, comprising: The evaluation data set acquisition unit is configured to: acquire an evaluation data set of the retrieval enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; The evaluation answer generation unit is configured to: cyclically input each question in the evaluation data set into each retrieval enhancement generation system, so that the retrieval enhancement generation system generates a corresponding answer; The algorithm evaluation index generating unit is configured to: evaluate the answers generated by the retrieval enhancement generation system and the answers in the evaluation data set using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; The principal component analysis evaluation unit is configured to: calculate multiple overall evaluation indicators of each retrieval enhancement generation system, perform principal component analysis on all of the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

[0018] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method for automatically evaluating the retrieval enhancement generation system as described in the first aspect of the present invention is implemented.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the automatic evaluation method for the retrieval enhancement generation system as described in the first aspect of the present invention.

[0020] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the automatic evaluation method of the retrieval enhancement generation system as described in the first aspect of the present invention.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. Automated evaluation under data constraints. The RAG automated evaluation system of the present invention exhibits significant advantages under data constraints, especially when constructing and using special data sets for evaluation, it can provide great convenience and benefits. The objectivity and comprehensiveness of the RAG system evaluation depends on a complete data set. However, the current application scenarios of the RAG system are mostly concentrated on the processing of internal data and internal information. These data are usually highly confidential, which makes it difficult to easily build an open data set. Traditionally, relying on manual methods to build these data sets is not only costly, but also not feasible in terms of time and resources. The RAG evaluation system of the present invention assists users through automated tools, and can quickly and efficiently build a complete data set for any knowledge base file, thereby greatly reducing the threshold and cost of data construction. This capability not only improves the coverage and accuracy of the RAG system evaluation, but also ensures the feasibility and practicality of evaluation in a restricted data environment, ultimately providing users with a more flexible and scalable evaluation solution.

[0022] 2. A complete and comprehensive RAG evaluation system. The RAG evaluation system of the present invention provides a complete and comprehensive evaluation system, covering every key link in the entire RAG process, which is significantly different from the traditional evaluation system. Most of the RAG evaluation systems currently on the market usually only focus on the accuracy of the final answer for evaluation, ignoring other important links in the RAG process. This single evaluation method may cause certain problems of the system to be ignored, thereby affecting the overall performance and user experience. In contrast, the evaluation method of the present invention conducts a detailed evaluation of each step in the RAG process, including document segmentation, document retrieval, document sorting, and answer output, etc., to ensure that the output of each link can be accurately evaluated. This comprehensive evaluation method can not only reveal the shortcomings of the RAG system that may exist at each stage, but also help developers accurately locate the problem, so as to optimize and improve the system in a targeted manner. Through this comprehensive evaluation mechanism, the RAG system of the present invention can more effectively improve the overall performance, ensure that the best results can be achieved in each link, and ultimately provide users with a more reliable and efficient RAG solution.

[0023] 3. PCA evaluates the system more accurately. The RAG evaluation system of the present invention achieves a more accurate system evaluation through principal component analysis (PCA); first, the dimension reduction function of PCA can effectively reduce the redundant information in the data, so that the evaluation indicators are more concise and clear. This simplification not only helps to focus on the most influential factors, but also improves the efficiency of the evaluation process; secondly, the automation characteristics of PCA eliminate the need to manually set weights. Principal component analysis can automatically capture and extract the main information in the data to ensure the objectivity and accuracy of the evaluation process; finally, by extracting linearly unrelated principal components, PCA successfully eliminates the multicollinearity problem between the original evaluation indicators. This feature ensures the independence of each evaluation indicator and avoids the evaluation bias caused by the correlation between indicators, thereby making the overall evaluation results of the system more reliable and accurate. By streamlining data, automating evaluation and eliminating multicollinearity, the accuracy and credibility of system evaluation are significantly improved.

[0024] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 A schematic diagram of a data set generation process provided in Example 1 of the present invention; Figure 2 A schematic diagram of the RAG system evaluation process provided in Example 1 of the present invention; Figure 3 A schematic diagram of an automated evaluation system for a search enhancement generation system provided in Embodiment 2 of the present invention; Figure 4 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0030] Embodiment 1: This implementation proposes an automated evaluation method for a search enhancement generation system. The following is a brief introduction to the technical terms and related concepts involved in this processing solution, including: Large language model: a deep learning model trained based on massive text data; RAG (Retrieval-Augmented Generation): RAG is a natural language processing (NLP) technique that combines the capabilities of information retrieval and generative models. Specifically, the RAG model first uses a retrieval module to find the most relevant document fragments for the input question from a large document library, and then uses a generative model (such as GPT) to generate answers based on these retrieved documents. This approach enhances the model's performance in processing factual tasks while maintaining information accuracy and generation fluency. PCA (Principal Component Analysis): PCA is a data dimensionality reduction technique used to simplify complex data sets while retaining the main features of the data as much as possible. It maximizes the variance of the projected data on a new coordinate system by projecting the data onto a new coordinate system. These new coordinate axes are called principal components. Usually the first few principal components can explain most of the variation in the data, so that the original data can be represented with fewer dimensions.

[0031] The specific evaluation method of this implementation includes the following process: S1: Construction of RAG evaluation dataset.

[0032] S1.1: Select the document set for dataset construction , document set From n documents ,in composition, , the document set is followed by the knowledge source used for knowledge base / document base retrieval in RAG. Documents from multiple sources can be selected, such as pdf, word, txt, etc.

[0033] S1.2: Parse non-txt type documents such as PDF and Word into txt type. The parsing uses OCR recognition and other technologies, which will not be specifically described here.

[0034] S1.3: The parsed document is in txt format. At this time, each document d of txt type is split according to paragraphs. Can be split into several paragraphs ,in , indicating the of the documents The set of paragraphs for each document , It is a document The number of paragraphs in the current document set D. .

[0035] S1.4: Select 1 to 3 of the split paragraphs to form a paragraph group, and generate N paragraph groups in total. , the N groups of paragraphs generated are expressed as .

[0036] S1.5: Input N groups of content into a powerful large model, such as gpt4, N times, and use the powerful large model to generate possible questions and answers for the content in each group of paragraphs input: (1); S1.6: After question generation and answer generation are completed, they can be screened manually or automatically based on rules to remove some question-answer pairs with semantic duplication or low quality of question or answer generation.

[0037] S1.7: Finally, the question, current paragraph group, and answer are formed into a total data set and stored: (2).

[0038] S2: RAG system automated evaluation.

[0039] Requirements for the RAG system to be tested: Each complete RAG system will have document recognition and document segmentation functions. The RAG system to be tested here must be able to output the answer and the source paragraph of the question. Specifically, it includes the following processes: S2.1: Select the document for which the evaluation data set has been generated and submit it to the RAG system. The RAG system will perform document recognition and segmentation on its own.

[0040] S2.2: The evaluation dataset to be constructed for this document All questions The loop is entered into the RAG system, and the RAG system generates the answers and selected paragraphs for each question: (3); in: Indicates that the RAG system targets the problem The result of the answer; Represents the document fragment retrieved by the RAG system.

[0041] S2.3: Six answer evaluation algorithms are used to calculate the answers given by the RAG system and the answers of the previous evaluation data set. The specific evaluation algorithms are: ROUGE-1, ROUGE-2, ROUGE-L, Bert Score, BLUE, Meteor, get the precision P, recall R, and F1 value of each algorithm.

[0042] (1) ROUGE-1 (Recall-Oriented Understudy for Gisting Evaluation) evaluates the unigram overlap (i.e., single word) between the answer given by the RAG system and the reference answer. It measures the quality of the answer by counting the number of unigrams in the RAG answer that are the same as those in the reference answer. ROUGE-1 is mainly used to evaluate the coverage and consistency of the RAG system's answers: (4); (2) ROUGE-2 evaluates the bigram overlap (i.e., the combination of two consecutive words) between the RAG answer and the reference answer. It measures the quality of the answer by counting the number of bigrams in the RAG answer that are identical to those in the reference answer. ROUGE-2 focuses more on word order and phrase consistency than ROUGE-1: (5); (3) ROUGE-L uses the longest common subsequence (LCS) to evaluate the overlap between the generated text and the reference text. LCS takes into account the order of words and can therefore better reflect the semantic consistency of the text. The advantage of ROUGE-L is that it does not require strict word order matching and can tolerate some changes in word order: (6); (4) Bert Score uses the BERT (Bidirectional Encoder Representations from Transformers) model to calculate the similarity between the generated text and the reference text. BERT captures rich contextual information and semantic relationships through a pre-trained language model, enabling Bert Score to more accurately evaluate the semantic similarity of text: (7); in is the answer BERT embeddings for the words in is the answer given by the RAG system BERT embeddings in: (8); (9); (5) BLUE measures the quality of generated text by calculating the exact n-gram matches between the generated text and the reference text. BLEU usually combines multiple n-gram matches (such as unigrams, bigrams, trigrams, etc.) and uses the geometric mean to summarize these matches. It also prevents the generated text from being too short by penalizing short texts: (10); Among them, BP is the brevity penalty, which is calculated as follows: (11); is the weight of the nth n-gram. Usually, the weights of all n-grams are equal.

[0043] is the exact match rate of the n-th n-gram, defined as: (12); (6) Metero is a weighted average-based metric that takes into account overlapping words, synonym matches, and word order. It works by aligning the generated summary with the reference summary and then calculating the weighted average of the aligned parts: (13); in: It is the harmonic mean of precision and recall, defined as: (14); (15); (16); Penalty is a penalty item calculated based on the arrangement of matching words, which is used to reduce the matching score that is inconsistent with the word order of the reference text.

[0044] S2.4: Calculate the overall evaluation index of the RAG system.

[0045] (1) Context recall measures the ability of a large model to successfully retrieve all contextual information related to a question when answering it. This metric is used to evaluate the model's ability to accurately identify and extract fragments related to the question from a large amount of background information. The higher the context recall rate, the more comprehensive the model's ability to obtain information related to the question, ensuring that the basis for the answer is comprehensive and complete enough: (17); : A collection of context fragments retrieved by the model, : A collection of all context snippets related to the question.

[0046] (2) Contextual precision measures the accuracy of the content extracted by the large model when retrieving relevant contextual information. This indicator focuses on the model's ability to avoid retrieving information that is irrelevant to the question or misleading. High precision means that the contextual content retrieved by the model is highly relevant to the question, reducing unnecessary information interference: (18); (3) Answer semantic similarity evaluates the semantic similarity between the answers generated by the model and the standard answers. This indicator is used to measure whether the model can understand the question and generate answers that are semantically consistent with the standard answers, rather than just matching the surface vocabulary. High semantic similarity means that the model's answer can better express the same meaning as the standard answer: (19); is the semantic vector of the answer to the dataset, is the semantic vector of the answer given by RAG.

[0047] (4) The fidelity metric measures whether the large model is faithful to the contextual information it retrieves when answering questions. This metric is used to evaluate whether the model accurately answers based on the retrieved contextual information, rather than generating inaccurate or irrelevant content out of thin air. High fidelity means that the model's answers are strictly based on the provided contextual information, reducing the risk of incorrect or false information: (20); A collection of context-based information snippets in the model’s answer, The collection of all pieces of information in the model's answer.

[0048] (5) Answer relevance measures the direct correlation between the answer generated by the large model and the question asked. This indicator evaluates whether the model's answer hits the point of the question and avoids off-topic, redundant or irrelevant answers. A high correlation means that the model can accurately understand the question and provide a straightforward answer: (twenty one); Where N is the number of question-answer pairs, For the Question Corresponding to the answer given by RAG The correlation score between .

[0049] S2.5: Use PCA to calculate the final evaluation score of the RAG system.

[0050] For the 11 indicators obtained previously and multiple samples for multiple questions, the score of the entire RAG system is calculated comprehensively. In the comprehensive evaluation of the RAG system, PCA can help us reduce the redundancy between multiple indicators and convert these indicators into several principal components for comprehensive scoring, including: (1) Data standardization.

[0051] The first step of PCA is to standardize all indicator data. Since different indicators have different dimensions, standardization can ensure that each indicator contributes fairly to the principal component. The standardization formula is as follows: (twenty two); in, is the original data matrix Middle Row (sample) the value of the column (metric), It is The mean of the indicators, It is The standard deviation of the indicator, is the standardized value, the standardized data matrix The mean is 0 and the standard deviation is 1.

[0052] (2) Calculate the covariance matrix.

[0053] The standardized data matrix To calculate the covariance matrix , the covariance matrix reflects the linear correlation between the indicators. The calculation formula of the covariance matrix is ​​as follows: (twenty three); in, is the sample size, is the standardized data matrix, the covariance matrix It is an 11x11 symmetric matrix that describes the relationship between 11 indicators.

[0054] (3) Eigenvalue decomposition.

[0055] Covariance matrix Perform eigenvalue decomposition to obtain eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​represent the importance of the principal components, and the eigenvectors define the directions of the principal components: (twenty four); in: It is The eigenvalue represents the The importance of the principal components It is feature vector, indicating the The direction of the principal component.

[0056] (4) Select the principal components.

[0057] Select the main principal components according to the size of the eigenvalue. Usually, select the principal components with a higher cumulative explained variance ratio. The calculation formula for the explained variance ratio is as follows: (25); The cumulative explained variance ratio is used to determine how many principal components to retain, and the first few principal components are usually selected so that the cumulative explained variance reaches 80% to 95%.

[0058] Assume that the first K principal components are selected, then these principal components explain most of the variance of the original data.

[0059] (5) Projection into the principal component space.

[0060] The standardized data matrix Project to selected In the principal component space, a new low-dimensional data representation is obtained: (26); in, It is the projected data matrix, which contains the representation of all samples in the principal component space.

[0061] (6) Comprehensive score:

[0062] The principal components obtained by PCA can be directly used for the comprehensive score of the RAG system. Assuming that the first principal component explains most of the variance, it can be directly regarded as the main basis for the comprehensive score. The comprehensive score can also be obtained by weighted summation of the first few principal components. The comprehensive score calculation formula is: (27); in, It is principal components, is the corresponding weight. The weight can be determined according to the proportion of explained variance. Larger principal components will have higher weights, and equal weights can be selected.

[0063] Embodiment 2: like Figure 3 As shown, this implementation provides an automatic evaluation system for a search enhancement generation system, including: The evaluation data set acquisition unit is configured to: acquire an evaluation data set of the retrieval enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; The evaluation answer generation unit is configured to: cyclically input each question in the evaluation data set into each retrieval enhancement generation system, so that the retrieval enhancement generation system generates a corresponding answer; The algorithm evaluation index generating unit is configured to: evaluate the answers generated by the retrieval enhancement generation system and the answers in the evaluation data set using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; The principal component analysis evaluation unit is configured to: calculate multiple overall evaluation indicators of each retrieval enhancement generation system, perform principal component analysis on all of the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

[0064] The specific working process of each of the above units is described in Example 1 and will not be repeated here.

[0065] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the retrieval enhancement generation system automatic evaluation system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0066] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the automated evaluation method of the retrieval enhancement generation system of the embodiment of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0067] Embodiment 3: like Figure 4 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0068] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0069] The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0070] The processor 1001 is configured to execute the following process: Obtaining an evaluation data set for a search enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; Input each question in the evaluation data set into each retrieval enhancement generation system cyclically, so that the retrieval enhancement generation system generates a corresponding answer; The answers generated by the search enhancement generation system and the answers in the evaluation data set are evaluated using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; Calculate multiple overall evaluation indicators for each retrieval enhancement generation system, perform principal component analysis on all the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

[0071] The specific working process is described in Example 1 and will not be repeated here.

[0072] Embodiment 4: This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0073] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0074] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process: Obtaining an evaluation data set for a search enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; Input each question in the evaluation data set into each retrieval enhancement generation system cyclically, so that the retrieval enhancement generation system generates a corresponding answer; The answers generated by the search enhancement generation system and the answers in the evaluation data set are evaluated using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; Calculate multiple overall evaluation indicators for each retrieval enhancement generation system, perform principal component analysis on all the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

[0075] The specific working process is described in Example 1 and will not be repeated here.

[0076] Embodiment 5: The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process: Obtaining an evaluation data set for a search enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; Input each question in the evaluation data set into each retrieval enhancement generation system cyclically, so that the retrieval enhancement generation system generates a corresponding answer; The answers generated by the search enhancement generation system and the answers in the evaluation data set are evaluated using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; Calculate multiple overall evaluation indicators for each retrieval enhancement generation system, perform principal component analysis on all the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

[0077] The specific working process is described in Example 1 and will not be repeated here.

[0078] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0079] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for automatically evaluating a search enhancement generation system, characterized in that: The process includes: Obtaining an evaluation data set for a search enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; Input each question in the evaluation data set into each retrieval enhancement generation system cyclically, so that the retrieval enhancement generation system generates a corresponding answer; The answers generated by the search enhancement generation system and the answers in the evaluation data set are evaluated using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; Calculate multiple overall evaluation indicators for each retrieval enhancement generation system, perform principal component analysis on all the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

2. The automated evaluation method for the search enhancement generation system according to claim 1, characterized in that: The construction of the evaluation data set includes: Select the document set for which the dataset needs to be built , document set From n documents ,in composition, ; Convert all documents to txt type, split each txt document by paragraph, and Can be split into several paragraphs ,in , indicating the of the documents paragraphs, the set of paragraphs for each document , It is a document The number of paragraphs in the current document set D. ; Select S paragraphs from the split paragraphs to form a paragraph group, and generate N paragraph groups in total, where S is an integer greater than or equal to 1. , the N groups of paragraphs generated are expressed as ; Input N groups of content into the powerful large model N times, and use the powerful large model to generate possible questions and answers for the content in each group of paragraphs input. ; After question generation and answer generation are completed, the question-answer pairs with semantic duplication or poor quality of question or answer generation are screened out, and the question, current paragraph group, and answer are formed into a total data set and stored: .

3. The automated evaluation method for the search enhancement generation system according to claim 1, characterized in that: ROUGE-1, ROUGE-2, ROUGE-L, Bert Score, BLUE and Meteor were used for evaluation to obtain multiple algorithm evaluation indicators.

4. The automated evaluation method for the search enhancement generation system according to claim 1, characterized in that: The overall evaluation indicators include: context recall, context precision, answer semantic similarity, fidelity and answer relevance.

5. The automated evaluation method for the search enhancement generation system according to claim 1, characterized in that: Performing principal component analysis on all the algorithm evaluation indicators and the overall evaluation indicators includes: Standardize all indicator data to obtain a standardized data matrix ; According to the data matrix Calculate the covariance matrix , the covariance matrix It is a symmetric matrix, which is used to describe the relationship between various indicators; For the covariance matrix Perform eigenvalue decomposition to obtain eigenvalues; Calculate the cumulative explained variance according to the eigenvalues, and select multiple principal components with higher cumulative explained variance ratios; The standardized data matrix Project to selected In the principal component space, a new low-dimensional data representation is obtained: ; The weighted sum of the selected principal components is used as the score of the retrieval enhancement generation system: ,in, It is principal components, is the corresponding weight; or the selected first principal component is used as the score of the retrieval enhancement generation system.

6. The automated evaluation method for a search enhancement generation system according to any one of claims 1 to 5, characterized in that: Each of the question-answer pairs includes a question, a current paragraph group and an answer, and the retrieval enhancement generation system can output the answer and the source paragraph of the question according to the question.

7. An automated evaluation system for a search enhancement generation system, characterized in that: include: The evaluation data set acquisition unit is configured to: acquire an evaluation data set of the retrieval enhancement generation system, wherein the evaluation data set includes a plurality of question-answer pairs, each of which includes a question and an answer; The evaluation answer generation unit is configured to: cyclically input each question in the evaluation data set into each retrieval enhancement generation system, so that the retrieval enhancement generation system generates a corresponding answer; The algorithm evaluation index generating unit is configured to: evaluate the answers generated by the retrieval enhancement generation system and the answers in the evaluation data set using different answer evaluation algorithms to obtain multiple algorithm evaluation indicators; The principal component analysis evaluation unit is configured to: calculate multiple overall evaluation indicators of each retrieval enhancement generation system, perform principal component analysis on all of the algorithm evaluation indicators and the overall evaluation indicators, and use the weighted sum of the selected principal components as the score of the retrieval enhancement generation system, or use the selected first principal component as the score of the retrieval enhancement generation system.

8. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the automatic evaluation method for the retrieval enhancement generation system according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the automatic evaluation method for a retrieval enhancement generation system according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the automatic evaluation method of the retrieval enhancement generation system according to any one of claims 1 to 6.

Images