Evaluation method and device for retrieval enhancement generation model and computing equipment

Abstract

The embodiment of the invention provides an evaluation method and device for a retrieval enhancement generation model and computing equipment, and the method comprises the steps that a preset text is obtained, the preset text comprises at least one question and answer statement pair, and the question and answer statement pair comprises question and answer statements and reference answers corresponding to the question and answer statements; the preset text is processed through the retrieval enhancement generation model, a standard evaluation index and at least one special evaluation index of the retrieval enhancement generation model are obtained, and the special evaluation index is used for representing the performance of the retrieval enhancement generation model in the target dimension; and determining an evaluation result according to the standard evaluation index and the special evaluation index. According to the method, the applicability of the model evaluation method can be improved, and an accurate improvement direction is provided for model optimization in different application scenes.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an evaluation method, apparatus and computing device for a retrieval enhancement generative model. Background Technology

[0002] With the development of artificial intelligence technology, Retrieval Augmented Generation (RAG) has become one of the most popular architectures in Large Language Model (LLM) systems, especially in dialogue-based products. How to evaluate the suitability of RAG solutions for specific scenarios and determine the optimization direction of the model has become a common problem in the industry.

[0003] In related technologies, retrieval enhancement generative models are evaluated using standard evaluation metrics, including loyalty, answer relevance, and contextual relevance. Loyalty measures the consistency between the answer text and the retrieval context; answer relevance measures the relevance between the answer text and the question-and-answer statements; and contextual relevance measures the relevance between the retrieval context and the question-and-answer statements. However, these methods suffer from poor adaptability. Summary of the Invention

[0004] This application provides an evaluation method, apparatus, and computing device for retrieval enhancement generative models, which addresses the problem of poor adaptability of retrieval enhancement generative model evaluation methods in related technologies.

[0005] In a first aspect, embodiments of this application provide an evaluation method for a retrieval-enhanced generative model, comprising:

[0006] Obtain a preset text, which includes at least one question-and-answer pair, the question-and-answer pair including a question and answer statement and a corresponding reference answer;

[0007] By processing the preset text using a retrieval enhancement generation model, a standard evaluation index and at least one specific evaluation index of the retrieval enhancement generation model are obtained, wherein the specific evaluation index is used to characterize the performance of the retrieval enhancement generation model in the target dimension.

[0008] The evaluation results are determined based on the standard evaluation indicators and the specific evaluation indicators.

[0009] In the above scheme, by constructing a hierarchical evaluation system that includes standard evaluation indicators and at least one special evaluation indicator, a multi-dimensional and scenario-customizable fine-grained performance evaluation of the retrieval enhancement generation model is achieved. The special evaluation indicator can evaluate the performance of the retrieval enhancement generation model in the target dimension based on the requirements of the target application scenario, thereby improving the adaptability of the model evaluation method and providing precise improvement directions for model optimization in different application scenarios.

[0010] In one possible implementation, the preset text is processed by a retrieval-enhanced generation model to obtain a standard evaluation metric and at least one specific evaluation metric for the retrieval-enhanced generation model, including:

[0011] Based on the preset text, a retrieval context is determined in the document database, the retrieval context including multiple text segments related to the question and answer statement;

[0012] Based on the retrieval context and the question-and-answer statements, generate the answer text using a large language model;

[0013] The standard evaluation metrics are determined based on the search context, the answer text, and the question-and-answer statements.

[0014] Obtain the target performance of the large language model when generating text answers, and determine the specific evaluation index based on the target performance.

[0015] In the above scheme, the preset text can be processed by the retrieval enhancement generation model, the standard evaluation index can be obtained based on the processing result, and the target performance can be obtained during the processing to obtain the specific evaluation index, so that the evaluation result can be further determined based on the standard evaluation index and the target evaluation index.

[0016] In one possible implementation, the standard evaluation metrics include loyalty, answer relevance, and contextual relevance. The standard evaluation metrics are determined based on the retrieval context, the answer text, and the question-and-answer statements, including:

[0017] Based on a large language model, the answer text is decomposed into multiple atomic statements, and the loyalty is determined according to whether the multiple atomic statements can be supported by the retrieval context.

[0018] Based on a large language model, multiple potential questions are generated from the answer text, and the relevance of the answer is determined based on the semantic similarity between the potential questions and the question-and-answer statements.

[0019] Based on a large language model, multiple text segments related to the question and answer statement are determined from the retrieval context. The context relevance is determined by comparing the number of text segments related to the question and answer statement with the total number of text segments in the retrieval context.

[0020] In the above scheme, the consistency between the answer text and the retrieval context is measured by the loyalty index, the relevance between the answer text and the question and answer statements is measured by the answer relevance index, and the relevance between the retrieval context and the question and answer statements is measured by the context relevance index. This improves the credibility of the model output results and ensures that the model can generate more reliable and relevant answers when dealing with complex queries.

[0021] In one possible implementation, the specific evaluation metrics include response time. Determining the specific evaluation metrics for the retrieval enhancement generation model includes:

[0022] Based on the retrieval enhancement generation model, a first time and a second time are determined. The first time is used to indicate the time when the retrieval enhancement generation model receives the preset text, and the second time is used to indicate the time when the answer text is generated.

[0023] The response duration is determined based on the difference between the first time point and the second time point.

[0024] In the above scheme, by establishing a timestamp-based response duration quantification mechanism, the quantifiable tracking of model processing speed is realized, which can provide data support for model efficiency optimization and improve the adaptability of the evaluation framework in application scenarios where efficiency is required.

[0025] In one possible implementation, the specific evaluation indicators include resource utilization rate. Determining the specific evaluation indicators for the retrieval enhancement generation model includes:

[0026] The device monitoring system is used to monitor the CPU and GPU of the computing device and obtain monitoring data, including CPU utilization and GPU utilization.

[0027] The resource utilization rate is determined based on the monitoring data of the computing device between the first time point and the second time point.

[0028] In the above solution, by monitoring and quantifying resource usage in real time, data support can be provided for system optimization, such as improving resource utilization efficiency by adjusting model complexity or hardware configuration.

[0029] In one possible implementation, the specific evaluation indicators include user reviews. Determining the specific evaluation indicators for the retrieval enhancement generation model includes:

[0030] Obtain user evaluation information for the answer text, including user downvote / dislike actions on the answer text, and / or user rating information for the answer text;

[0031] The user rating is determined based on the evaluation information.

[0032] In the above scheme, by collecting user evaluation information through the user evaluation mechanism, the model evaluation can be more comprehensive and closer to the actual use scenario. User evaluation can promote model optimization and improve user experience.

[0033] In one possible implementation, the evaluation result is determined based on the standard evaluation indicators and the specific evaluation indicators, including:

[0034] Based on the standard evaluation indicators, the evaluation values ​​corresponding to the standard evaluation indicators are determined. The evaluation values ​​corresponding to the standard evaluation indicators include loyalty evaluation value, answer relevance evaluation value, and context relevance evaluation value.

[0035] Based on the specific evaluation indicators, the evaluation values ​​corresponding to the specific evaluation indicators are determined. The evaluation values ​​corresponding to the specific evaluation indicators include resource utilization rate evaluation value, response time evaluation value, and user evaluation value.

[0036] The evaluation result is determined based on the evaluation value corresponding to the standard evaluation indicator, the evaluation value corresponding to the special evaluation indicator, and the weights corresponding to the standard evaluation indicator and the special evaluation indicator, respectively.

[0037] The above solution integrates standard evaluation indicators with special evaluation indicators in multiple dimensions and introduces configurable weighting strategies to achieve a dynamic balance between comprehensive performance and scenario-specific performance evaluation, effectively improving the adaptability of evaluation results to actual business needs.

[0038] In one possible implementation, the preset text includes at least two question-and-answer pairs. Based on the specific evaluation indicators, the evaluation value corresponding to each indicator is determined, including:

[0039] Based on the multiple question-and-answer statements in the preset text, generate the retrieval context and answer text corresponding to each question-and-answer statement;

[0040] Based on the multiple question-and-answer statements, as well as the retrieval context and answer text corresponding to each question-and-answer statement, the resource utilization rate, response time, and user rating corresponding to each question-and-answer statement are determined.

[0041] Based on the resource utilization rate, response time, and user evaluation corresponding to each question and answer statement, determine the resource utilization rate evaluation value, response time evaluation value, and user evaluation evaluation value.

[0042] In the above scheme, by using multiple question-and-answer statements to obtain the corresponding standard evaluation indicators and special evaluation indicators respectively, the evaluation bias caused by improper selection of a single question-and-answer statement can be reduced, thereby improving the accuracy of the evaluation results.

[0043] Secondly, embodiments of this application provide an evaluation apparatus for a retrieval-enhanced generative model, comprising:

[0044] The acquisition module is used to acquire preset text, which includes at least one question-and-answer statement pair, and the question-and-answer statement pair includes a question-and-answer statement and a corresponding reference answer.

[0045] The first processing module is used to process the preset text through a retrieval enhancement generation model to obtain a standard evaluation index and at least one specific evaluation index of the retrieval enhancement generation model, wherein the specific evaluation index is used to characterize the performance of the retrieval enhancement generation model in the target dimension.

[0046] The second processing module is used to determine the evaluation result based on the standard evaluation indicators and the specific evaluation indicators.

[0047] In the above scheme, by constructing a hierarchical evaluation system that includes standard evaluation indicators and at least one special evaluation indicator, a multi-dimensional and scenario-customizable fine-grained performance evaluation of the retrieval enhancement generation model is achieved. The special evaluation indicator can evaluate the performance of the retrieval enhancement generation model in the target dimension based on the requirements of the target application scenario, thereby improving the adaptability of the model evaluation method and providing precise improvement directions for model optimization in different application scenarios.

[0048] In one possible implementation, the first processing module is specifically used for,

[0049] Based on the preset text, a retrieval context is determined in the document database, the retrieval context including multiple text segments related to the question and answer statement;

[0050] Based on the retrieval context and the question-and-answer statements, generate the answer text using a large language model;

[0051] The standard evaluation metrics are determined based on the search context, the answer text, and the question-and-answer statements.

[0052] Obtain the target performance of the large language model when generating text answers, and determine the specific evaluation index based on the target performance.

[0053] In the above scheme, the preset text can be processed by the retrieval enhancement generation model, the standard evaluation index can be obtained based on the processing result, and the target performance can be obtained during the processing to obtain the specific evaluation index, so that the evaluation result can be further determined based on the standard evaluation index and the target evaluation index.

[0054] In one possible implementation, the first processing module is specifically used for,

[0055] Based on a large language model, the answer text is decomposed into multiple atomic statements, and the loyalty is determined according to whether the multiple atomic statements can be supported by the retrieval context.

[0056] Based on a large language model, multiple potential questions are generated from the answer text, and the relevance of the answer is determined based on the semantic similarity between the potential questions and the question-and-answer statements.

[0057] Based on a large language model, multiple text segments related to the question and answer statement are determined from the retrieval context. The context relevance is determined by comparing the number of text segments related to the question and answer statement with the total number of text segments in the retrieval context.

[0058] In the above scheme, the consistency between the answer text and the retrieval context is measured by the loyalty index, the relevance between the answer text and the question and answer statements is measured by the answer relevance index, and the relevance between the retrieval context and the question and answer statements is measured by the context relevance index. This improves the credibility of the model output results and ensures that the model can generate more reliable and relevant answers when dealing with complex queries.

[0059] In one possible implementation, the first processing module is specifically used for,

[0060] Based on the retrieval enhancement generation model, a first time and a second time are determined. The first time is used to indicate the time when the retrieval enhancement generation model receives the preset text, and the second time is used to indicate the time when the answer text is generated.

[0061] The response duration is determined based on the difference between the first time point and the second time point.

[0062] In the above scheme, by establishing a timestamp-based response duration quantification mechanism, the quantifiable tracking of model processing speed is realized, which can provide data support for model efficiency optimization and improve the adaptability of the evaluation framework in application scenarios where efficiency is required.

[0063] In one possible implementation, the first processing module is specifically used for,

[0064] The device monitoring system is used to monitor the CPU and GPU of the computing device and obtain monitoring data, including CPU utilization and GPU utilization.

[0065] The resource utilization rate is determined based on the monitoring data of the computing device between the first time point and the second time point.

[0066] In the above solution, by monitoring and quantifying resource usage in real time, data support can be provided for system optimization, such as improving resource utilization efficiency by adjusting model complexity or hardware configuration.

[0067] In one possible implementation, the first processing module is specifically used for,

[0068] Obtain user evaluation information for the answer text, including user downvote / dislike actions on the answer text, and / or user rating information for the answer text;

[0069] The user rating is determined based on the evaluation information.

[0070] In the above scheme, by collecting user evaluation information through the user evaluation mechanism, the model evaluation can be more comprehensive and closer to the actual use scenario. User evaluation can promote model optimization and improve user experience.

[0071] In one possible implementation, the second processing module is specifically used for,

[0072] Based on the standard evaluation indicators, the evaluation values ​​corresponding to the standard evaluation indicators are determined. The evaluation values ​​corresponding to the standard evaluation indicators include loyalty evaluation value, answer relevance evaluation value, and context relevance evaluation value.

[0073] Based on the specific evaluation indicators, the evaluation values ​​corresponding to the specific evaluation indicators are determined. The evaluation values ​​corresponding to the specific evaluation indicators include resource utilization rate evaluation value, response time evaluation value, and user evaluation value.

[0074] The evaluation result is determined based on the evaluation value corresponding to the standard evaluation indicator, the evaluation value corresponding to the special evaluation indicator, and the weights corresponding to the standard evaluation indicator and the special evaluation indicator, respectively.

[0075] The above solution integrates standard evaluation indicators with special evaluation indicators in multiple dimensions and introduces configurable weighting strategies to achieve a dynamic balance between comprehensive performance and scenario-specific performance evaluation, effectively improving the adaptability of evaluation results to actual business needs.

[0076] In one possible implementation, the second processing module is specifically used for,

[0077] Based on the multiple question-and-answer statements in the preset text, generate the retrieval context and answer text corresponding to each question-and-answer statement;

[0078] Based on the multiple question-and-answer statements, as well as the retrieval context and answer text corresponding to each question-and-answer statement, the resource utilization rate, response time, and user rating corresponding to each question-and-answer statement are determined.

[0079] Based on the resource utilization rate, response time, and user evaluation corresponding to each question and answer statement, determine the resource utilization rate evaluation value, response time evaluation value, and user evaluation evaluation value.

[0080] In the above scheme, by using multiple question-and-answer statements to obtain the corresponding standard evaluation indicators and special evaluation indicators respectively, the evaluation bias caused by improper selection of a single question-and-answer statement can be reduced, thereby improving the accuracy of the evaluation results.

[0081] Thirdly, embodiments of this application provide a computing device, including: a memory and a processor;

[0082] The memory stores computer-executed instructions;

[0083] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0084] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0085] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0086] The retrieval enhancement generation model evaluation method, apparatus, and computing device provided in this application include: acquiring preset text, which includes at least one question-and-answer statement pair, each containing a question-and-answer statement and a corresponding reference answer; processing the preset text using the retrieval enhancement generation model to obtain a standard evaluation index and at least one specific evaluation index for the retrieval enhancement generation model, wherein the specific evaluation index characterizes the performance of the retrieval enhancement generation model in the target dimension; and determining the evaluation result based on the standard evaluation index and the specific evaluation index. In the above method, by constructing a hierarchical evaluation system including a standard evaluation index and at least one specific evaluation index, a multi-dimensional, scenario-customizable fine-grained performance evaluation of the retrieval enhancement generation model is achieved. The specific evaluation index can evaluate the performance of the retrieval enhancement generation model in the target dimension based on the requirements of the target application scenario, thereby improving the adaptability of the model evaluation method and providing precise improvement directions for model optimization in different application scenarios. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0089] Figure 2 A flowchart illustrating an evaluation method for a retrieval-enhanced generative model provided in this application embodiment. Figure 1 ;

[0090] Figure 3 for Figure 2 A flowchart illustrating the specific implementation of S102 in the illustrated embodiment;

[0091] Figure 4 A schematic diagram illustrating the determination of standard evaluation indicators provided in the embodiments of this application;

[0092] Figure 5 A flowchart illustrating an evaluation method for a retrieval-enhanced generative model provided in this application embodiment. Figure 2 ;

[0093] Figure 6 A schematic diagram of the structure of an evaluation device for a retrieval enhancement generative model provided in an embodiment of this application;

[0094] Figure 7This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0095] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0096] To facilitate understanding, the following will be combined with Figure 1 The application scenarios involved in the embodiments of this application will be described.

[0097] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Please refer to [link / reference]. Figure 1 The system includes a client 11 and a server 12. The evaluation method for the retrieval enhancement generation model provided in this application embodiment can be applied to interactive question-and-answer or dialogue systems, in scenarios where the retrieval enhancement generation model needs to be evaluated. For example, in an intelligent customer service system, the retrieval enhancement generation model can generate corresponding answer text based on the user's question and answer statements. To measure whether the answer text can effectively solve the user's problem, the retrieval enhancement generation model needs to be evaluated. Based on the evaluation results, the retrieval enhancement generation scheme can be further optimized to better meet user needs. A communication connection can be established between the client 11 and the server 12 to realize data transmission between the client 11 and the server 12.

[0098] Client 11 can be various electronic devices with a display screen, including but not limited to smartphones, tablets, desktop computers, and laptops. Client 11 can obtain user-inputted question-and-answer statements and send them to server 12.

[0099] Server 12 is equipped with a retrieval-enhanced generative model. Server 12 can receive question-and-answer statements sent by client 11, retrieve text data related to the question-and-answer statements from the document database, generate answer text based on a large language model, and transmit the answer text to client 11.

[0100] With the development of artificial intelligence technology, retrieval-enhanced generation (RAG) has become one of the most popular architectures in large language model systems, especially in dialogue-based products. How to quickly evaluate the suitability of RAG solutions for specific scenarios and determine the direction of model optimization has become a common problem in the industry.

[0101] In related technologies, retrieval enhancement generation models are evaluated using standard evaluation metrics, including loyalty, answer relevance, and contextual relevance. Loyalty measures the consistency between the answer text and the retrieval context, answer relevance measures the relevance between the answer text and the question-and-answer statements, and contextual relevance measures the relevance between the retrieval context and the question-and-answer statements.

[0102] However, the dimensions of focus for model performance vary greatly in different application scenarios, and the above evaluation methods cannot reflect the model requirements in different application scenarios, resulting in poor adaptability.

[0103] To address the aforementioned issues, this application proposes an evaluation method for retrieval enhancement generation models. By constructing a hierarchical evaluation system that includes standard evaluation metrics and at least one specific evaluation metric, it achieves multi-dimensional and scenario-customizable fine-grained performance evaluation of retrieval enhancement generation models. The specific evaluation metric can evaluate the performance of the retrieval enhancement generation model in the target dimension based on the requirements of the target application scenario, thereby improving the adaptability of the model evaluation method and providing precise improvement directions for model optimization in different application scenarios.

[0104] The technical solutions of the embodiments of this application will be described in detail below with specific examples. These specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0105] Figure 2 A flowchart illustrating an evaluation method for a retrieval-enhanced generative model provided in this application embodiment. Figure 1 The execution subject of this method can be a computing device or a processor located within a computing device. For example, the computing device can be a server. The processor within the computing device can be a CPU or GPU in a server. For ease of understanding, the following explanation will use a computing device as the execution subject. Please see [link to documentation]. Figure 2 The method may include:

[0106] S101. Obtain the preset text.

[0107] The preset text is a pre-constructed collection of texts containing domain-specific knowledge or scenario-based question-and-answer pairs. Each preset text includes at least one question-and-answer statement pair, with each pair consisting of a question and answer statement and a corresponding reference answer. The question and answer statements simulate real user queries, while the reference answers provide verified and accurate information to evaluate whether the responses generated by the retrieval enhancement generative model meet expectations.

[0108] The pre-defined text content can differ across different fields. For example, in the medical field, question-and-answer statements may include questions related to medication; in the financial field, they may include questions related to claims procedures. Pre-defined text can be constructed based on professional documents, historical dialogue records, and human annotations. For instance, in the medical field, pre-defined documents can be constructed using a large language model by obtaining information such as drug instructions, medical guidelines, historical consultation records, and doctor annotations.

[0109] S102. Process the preset text through the retrieval enhancement generation model to obtain at least one standard evaluation index and at least one special evaluation index of the retrieval enhancement generation model.

[0110] The retrieval-enhanced generative model is the model to be evaluated. Based on information retrieval technology, the retrieval-enhanced generative model can retrieve information related to the question and answer statements from the text database, generate retrieval context, and use text generation technology to generate answer text based on the retrieval context.

[0111] Standard evaluation metrics are a set of general indicators used to evaluate retrieval augmentation generative models. These metrics, such as those obtained based on the RAGAS framework, include loyalty, answer relevance, and contextual relevance. Loyalty measures the consistency between the answer text and the retrieval context; answer relevance measures the relevance between the answer text and the question-and-answer statements; and contextual relevance measures the relevance between the retrieval context and the question-and-answer statements.

[0112] Specific evaluation metrics are used to assess the performance of the retrieval enhancement generation model in the target dimension or scenario, such as response time, resource utilization, and user evaluation. Response time includes the time required from receiving the question-and-answer statement to generating the answer text; resource utilization includes the CPU and GPU usage of the retrieval generation model when processing the question-and-answer statement; and user evaluation includes collecting user satisfaction ratings for the answer text generated by the retrieval enhancement generation model.

[0113] The number of standard evaluation metrics and specific evaluation metrics can be multiple. For any given question-and-answer pair, there is a corresponding set of standard and specific evaluation metrics. For example, question-and-answer pair 'a' can yield answer text 'a', as well as the standard evaluation metric 'a' and specific evaluation metric 'a' corresponding to the retrieval enhancement generation model; question-and-answer pair 'b' can yield answer text '2', as well as the standard evaluation metric 'b' and specific evaluation metric 'b' corresponding to the retrieval enhancement generation model, and so on. By using multiple question-and-answer pairs to obtain the corresponding standard and specific evaluation metrics, the evaluation bias caused by inappropriate selection of a single question-and-answer pair can be reduced, thus improving the accuracy of the evaluation results.

[0114] For example, such as Figure 3As shown, the specific implementation of S102 includes:

[0115] S1021. Determine the retrieval context in the document database based on the preset text.

[0116] A document database is a system that stores a large amount of text data related to a preset text domain. For example, a document database may include documents in the form of articles, reports, manuals, etc.

[0117] Retrieval-enhanced generative models can retrieve relevant information fragments from document databases based on question-and-answer statements using information retrieval techniques, thus obtaining retrieval context. This retrieval context provides the necessary background knowledge and relevant information for the retrieval-enhanced generative model to generate the answer text.

[0118] Determining the search context can include steps such as preprocessing, key information extraction, document database retrieval, and search context determination.

[0119] Preprocessing can include segmentation, semantic expansion, and standardization to facilitate subsequent analysis and processing. For example, documents in a document database can be segmented into multiple text blocks, making it easier to extract text blocks related to question-and-answer statements and avoid introducing excessive redundant information into the retrieval context. For example, question-and-answer pairs can be converted into vector representations to facilitate the determination of values ​​for various standard evaluation metrics. For example, query statements can be expanded through synonym replacement, entity recognition, and query rewriting to enhance semantic understanding and avoid missing relevant information during retrieval. For example, text blocks in a document database can be converted into vectorized representations to facilitate the determination of the retrieval context based on the relevance between text blocks and question-and-answer statements.

[0120] Key information extraction can include entity recognition and keyword extraction. For example, key entities such as names of people, places, and organizations can be extracted from question-and-answer statements to facilitate quick location of relevant information in document databases. Keywords can also be extracted from question-and-answer statements, reflecting their theme and core content, thus improving the relevance between different text segments and the question-and-answer statements within the retrieval context.

[0121] When searching a document database, an appropriate retrieval algorithm can be selected based on the scenario to obtain preliminary search results. For example, in queries with explicit terms, preliminary search results can be determined based on keyword matching, such as the BM25 algorithm; for semantically complex queries, the query statement and text blocks in the document database can be transformed into vectors, and the cosine similarity between the query statement and each text block can be calculated to obtain preliminary search results, such as Sentence Transformers, BERT series algorithms, etc.

[0122] Context determination is used to extract highly relevant text blocks from the initial search results and integrate these extracted text blocks to obtain the search context. For example, multiple text blocks with higher similarity can be extracted from the initial search results, and duplicate content in the text blocks can be removed, related paragraphs can be merged, etc., to obtain the search context. This can avoid including too much redundant information in the search context, which would affect the quality and efficiency of the generated answer text.

[0123] S1022. Generate answer text based on the retrieval context and question-and-answer statements using a large language model.

[0124] Generating answer text based on a large language model can be achieved through the following process: integrating the retrieval context with the question and answer statements to form the input of the large language model; transforming the integrated data into a format that the large language model can process, such as a text sequence; and then, based on the input, performing in-depth analysis of the question and answer statements to generate the answer text.

[0125] S1023. Determine standard evaluation metrics based on the search context, answer text, and question-and-answer statements.

[0126] Standard evaluation metrics may include loyalty, answer relevance, and contextual relevance. Figure 4 A schematic diagram illustrating the determination of standard evaluation indicators provided in the embodiments of this application, such as... Figure 4 As shown, the various indicators in the standard evaluation metrics can be determined based on the search context, answer text, and question and answer statements.

[0127] Loyalty measures the consistency between the answer text and the retrieval context. Loyalty can be determined as follows: Based on a large language model, the answer text is decomposed into multiple atomic statements. Loyalty is determined based on whether these atomic statements are supported by the retrieval context. Specifically, the large language model identifies sentence boundaries and logical breakpoints in the answer text, breaking it down into atomic statements (the smallest semantic units). For each atomic statement, direct or indirect supporting evidence is searched in the retrieval context to determine loyalty. Whether an atomic statement is supported by the retrieval context can be determined by calculating the semantic similarity between the text segment and the atomic statement in the retrieval context. When the semantic similarity is greater than or equal to a similarity threshold, it is considered supported; otherwise, it is considered unsupported. The similarity threshold can be preset, such as 0.85. The loyalty value is the number of supported atomic statements divided by the total number of atomic statements, ranging from [0, 1]. For example, if the answer text contains a total of 4 atomic statements, and 2 of them are supported, the loyalty is 0.5.

[0128] Answer relevance measures the relevance between the answer text and the question-and-answer statements. It can be determined as follows: Based on a large language model, multiple potential questions are generated from the answer text, and the answer relevance is determined based on the semantic similarity between the potential questions and the question-and-answer statements. Specifically, using a large language model, multiple potential questions are generated from the answer text, and the semantic similarity between each potential question and the original question-and-answer statements is calculated. The average of these semantic similarities is determined as the answer relevance, which ranges from [0, 1].

[0129] Context relevance measures the relevance between the retrieval context and the question-and-answer statement. It can be determined as follows: Based on a large language model, multiple text segments related to the question-and-answer statement are identified from the retrieval context. The context relevance is determined by comparing the number of related text segments with the total number of text segments in the retrieval context. Specifically, using a large language model, text segments in the retrieval context are divided into two categories: relevant and irrelevant. Text segments containing keywords, synonyms, or semantic extensions from the question-and-answer statement are marked as relevant; otherwise, they are marked as irrelevant. The context relevance is calculated by dividing the number of relevant text segments by the total number of text segments in the retrieval context, and the value ranges from 0 to 1. For example, if the retrieval context includes 10 text segments, and 7 of them are marked as relevant, the context relevance value is 0.7.

[0130] S1024. Obtain the target performance of the large language model when generating text answers, and determine specific evaluation indicators based on the target performance.

[0131] Specialized evaluation indicators may include response time, resource utilization, and user feedback.

[0132] The response time can be determined as follows: Based on the retrieval enhancement generation model, determine the first time and the second time. The first time indicates the time when the retrieval enhancement generation model receives the preset text, and the second time indicates the time when the answer text is generated. The response time is determined based on the difference between the first time and the second time.

[0133] Specifically, the first moment can be triggered by the timestamp record when the retrieval enhancement generation model receives the preset text. After the first moment, the retrieval enhancement generation model determines the retrieval context based on the preset text, generates the answer text based on the retrieval context, and records the timestamp of the time when the answer text is generated. The second moment is determined by subtracting the first moment from the second moment. The response duration is a real number greater than 0, and the unit is seconds (S). For example, the response duration is 2 seconds.

[0134] Resource utilization can be determined as follows: Use a device monitoring system to monitor the CPU and GPU of the computing device and obtain monitoring data, including CPU utilization and GPU utilization; determine resource utilization based on the monitoring data of the computing device between the first and second time points.

[0135] Specifically, a device monitoring system can be a hardware system deployed on a computing device, which collects the operating status information of the computing device in real time through sensors or hardware interfaces; a device monitoring system can also be a software system installed on an operating system, which obtains monitoring data by calling APIs or third-party libraries.

[0136] The equipment monitoring system can periodically (e.g., once per second) collect CPU and GPU utilization data from computing devices. The utilization data includes multiple data points, and the timestamp corresponding to each data point is recorded. The collected monitoring data is stored in a local database. The monitoring data is preprocessed (e.g., data cleaning, normalization, etc.). Based on the timestamps of the first and second moments, the time window of the monitoring data (i.e., the time period between the first and second moments) is determined. The resource utilization rate is obtained based on the monitoring data within the time window. The value range of the resource utilization rate is [0,1], for example, the resource utilization rate is 80%.

[0137] Resource utilization can be the maximum value among all data points; or, resource utilization can be the average value among all data points; or, different weights can be assigned to CPU utilization and GPU utilization, and resource utilization can be the weighted average of the average values ​​of the data points in CPU utilization and the average values ​​of the data points in GPU utilization, and so on.

[0138] User reviews can be determined as follows: obtain user evaluation information on the answer text, including user click-down actions on the answer text, and / or user rating information on the answer text; determine user reviews based on the evaluation information.

[0139] Specifically, user evaluation information refers to users' subjective feedback on the quality of the answer text, reflecting their satisfaction and approval of the answer. Dislike / disapproval information refers to information indicating negative feedback from users through specific interactive behaviors (such as clicking the "Dissatisfied" button). Rating information refers to information where users quantitatively evaluate the quality of the answer text through continuous numerical input (such as star ratings or numerical ratings). The user interface can include evaluation entry points, such as "Satisfied" / "Dissatisfied" buttons or rating components. The computing device receives the evaluation information and stores it in a local database. Based on the timestamps of the evaluation information, it aggregates the information into the corresponding time windows to obtain the user evaluation.

[0140] User reviews can be disliked or rejected, such as "dissatisfied" or "satisfied"; or, user reviews can be star ratings, such as "three stars" or "four stars"; or, user reviews can be specific scores, such as "98 points"; or, user reviews can be any combination of the above forms, and this application does not impose any restrictions on this.

[0141] It should be noted that the execution order of S1023 and S1024 in the above embodiments is not strictly limited; they can be executed sequentially, alternately, or at least partially in parallel. For example, in one embodiment, S1024 is executed after S1023 is completed; in another embodiment, some operations of S1024 can be started before the completion of S1023; furthermore, S1023 and S1024 can also achieve synchronous processing by sharing computing resources.

[0142] S103. Determine the evaluation results based on the standard evaluation indicators and the specific evaluation indicators.

[0143] The evaluation results can be determined by weighted averaging the evaluation values ​​corresponding to the standard evaluation indicators and the specific evaluation indicators. Specifically, since the dimensions corresponding to the values ​​of each indicator are different—for example, loyalty, answer relevance, context relevance, and resource utilization range from [0,1], response time is a real number greater than or equal to 0, and user ratings can be a binary array (positive or negative) or a real number greater than or equal to 0—it is necessary to standardize the values ​​of each indicator. The values ​​in the standard evaluation indicators and the specific evaluation indicators can be converted into corresponding standardized evaluation values. For example, after conversion, the range of each evaluation value can be [0,100], facilitating the weighted averaging of the evaluation values.

[0144] Furthermore, different weights can be assigned to each indicator based on the evaluation objectives and scenarios. By adjusting the weights of each indicator, the effectiveness and applicability of the evaluation results can be optimized. For example, in a medical auxiliary diagnosis scenario, the relevance of the answer text may be more important, and the weight of answer relevance may be relatively higher; in a real-time interaction scenario, response time may be more important, and the weight of response time may be relatively higher.

[0145] The evaluation method for a retrieval enhancement generation model provided in this application involves obtaining preset text, which includes at least one question-and-answer pair, comprising a question and answer statement and a corresponding reference answer. The preset text is then processed by the retrieval enhancement generation model to obtain a standard evaluation metric and at least one specific evaluation metric. The specific evaluation metric characterizes the performance of the retrieval enhancement generation model in the target dimension. The evaluation result is determined based on the standard and specific evaluation metrics. This method, by constructing a hierarchical evaluation system including a standard evaluation metric and at least one specific evaluation metric, achieves multi-dimensional, scenario-customizable, fine-grained performance evaluation of the retrieval enhancement generation model. The specific evaluation metric can evaluate the performance of the retrieval enhancement generation model in the target dimension based on the requirements of the target application scenario, thereby improving the adaptability of the model evaluation method and providing precise improvement directions for model optimization in different application scenarios.

[0146] Figure 5 A flowchart illustrating an evaluation method for a retrieval-enhanced generative model provided in this application embodiment. Figure 2 .like Figure 5 As shown, it includes:

[0147] S201. Obtain the preset text.

[0148] S202. Process the preset text through the retrieval enhancement generation model to obtain at least one standard evaluation index and at least one special evaluation index of the retrieval enhancement generation model.

[0149] S203. Determine the evaluation value corresponding to the standard evaluation index based on the standard evaluation index.

[0150] The evaluation values ​​corresponding to the standard evaluation indicators include loyalty evaluation value, answer relevance evaluation value, and context relevance evaluation value.

[0151] Because the dimensions corresponding to the values ​​of each indicator in the standard evaluation indicators and those in the specific evaluation indicators are different, it is necessary to standardize the values ​​of each indicator. The values ​​in both the standard and specific evaluation indicators can be converted into corresponding standardized evaluation values. For example, the range of each evaluation value after conversion can be [0, 100], which facilitates weighted averaging of the evaluation values. Specifically, since the range of loyalty, answer relevance, and context relevance is [0, 1], the values ​​of loyalty, answer relevance, and context relevance can be multiplied by 100 respectively to obtain the loyalty evaluation value, answer relevance evaluation value, and context relevance evaluation value.

[0152] S204. Determine the evaluation value corresponding to the special evaluation indicator based on the special evaluation indicator.

[0153] The evaluation values ​​corresponding to the special evaluation indicators include resource utilization rate evaluation value, response time evaluation value, and user evaluation value.

[0154] The resource utilization rate ranges from [0,1]. When the transformed evaluation values ​​range from [0,100], the resource utilization rate can be standardized based on a utilization rate threshold. Specifically, a higher resource utilization rate results in higher throughput for the large language model. However, excessively high resource utilization can lead to insufficient memory, affecting the normal operation of the retrieval enhancement generation model. Therefore, a utilization rate threshold is pre-set. The resource utilization rate indicated by the utilization rate threshold ensures the normal operation of the model. The resource utilization rate evaluation value corresponding to the utilization rate threshold can be 100. When the resource utilization rate is less than the utilization rate threshold, the resource utilization rate evaluation value can be 100. When the resource utilization rate is greater than the utilization rate threshold, the resource utilization rate evaluation value can be proportionally reduced from 100 based on the portion exceeding the utilization rate threshold, until it drops to 0. For example, for every 1% increase in resource utilization rate, the evaluation value decreases by 25 points.

[0155] The response duration is a real number greater than or equal to 0. When the range of each evaluation value after conversion is [0, 100], the response duration can be standardized according to a time threshold. Specifically, the time threshold can be preset and dynamically adjusted according to the real-time interaction requirements of the application scenario. For example, the time threshold can be 2 seconds. When the response duration is less than or equal to the time threshold, the response duration evaluation value can be 100. When the response duration is greater than the time threshold, the response duration evaluation value can be proportionally reduced from 100 based on the portion exceeding the time threshold until it drops to 0. For example, for every second the response duration exceeds the threshold, the evaluation value decreases by 25. That is, when the response duration is 3 seconds and the time threshold is 2, the evaluation value corresponding to this response duration is 75.

[0156] User review data can come in various forms, and can be standardized based on the proportion of different data types. For example, when user reviews are a binary array, they can be standardized to 0 and 100, such as "dissatisfied" corresponding to 0 and "satisfied" corresponding to "100"; when user reviews are star ratings, such as 0 to 5 stars, one star can correspond to 20, and 5 stars can correspond to 100; when user reviews are specific scores, the scores can be converted according to a proportion, such as multiplying the original score by 10 in a 10-point rating system to obtain the evaluation value, and in a 100-point rating system, the original score is the evaluation value.

[0157] S205. Determine the evaluation results based on the evaluation values ​​corresponding to the standard evaluation indicators, the evaluation values ​​corresponding to the special evaluation indicators, and the weights corresponding to the standard evaluation indicators and the special evaluation indicators, respectively.

[0158] Weights can be assigned to each evaluation indicator based on the assessment needs and application scenarios to measure the importance of that indicator in the current application scenario. For example, in a medical auxiliary diagnosis scenario, the relevance of the answer text may be more important, and the weight of answer relevance is relatively higher; in a real-time interaction scenario, response time may be more important, and the weight of response time is relatively higher, and so on. For example, in a medical auxiliary diagnosis scenario, the weights assigned to each evaluation indicator are as follows: loyalty 15%, answer relevance 30%, context relevance 15%, response time 15%, resource utilization 5%, and user rating 20%. Then the evaluation result is determined as: loyalty evaluation value × 15% + answer relevance evaluation value × 30% + context relevance evaluation value × 15% + response time evaluation value × 15% + resource utilization evaluation value × 5% + user rating evaluation value × 20%.

[0159] In one possible implementation, the preset text may include multiple question-and-answer pairs. For any given question-and-answer pair, processing is performed based on a retrieval-enhanced generative model to obtain the aforementioned evaluation metrics. Therefore, each evaluation metric may have multiple values. In this case, the evaluation value corresponding to each metric can be determined in the following way.

[0160] For each evaluation indicator, it can be determined by averaging. The following explanation uses loyalty evaluation as an example: Multiple loyalty values ​​are averaged to obtain an average loyalty value. This average loyalty value is then standardized, for example, converting the average loyalty value in the range [0,1] into a loyalty evaluation value in the range [0,100]. The conversion process is as described in S301 and will not be repeated here. Optionally, the loyalty values ​​can be standardized first to obtain multiple loyalty evaluation values, and then the average of these multiple loyalty evaluation values ​​can be obtained to obtain the final loyalty evaluation value. This embodiment of the application does not impose specific limitations on this approach.

[0161] Optionally, the response time evaluation value can also be determined as follows: obtain the 95th percentile of each response time, that is, sort each response time in ascending order, and 95% of the response times are less than or equal to this value. If this value is less than or equal to the time threshold, the response time evaluation value corresponds to 100. If this value is greater than the time threshold, the response time evaluation value is reduced by 25 for every second it exceeds 100, up to a maximum of 0.

[0162] Optionally, the user evaluation value can also be determined in the following way: When the user evaluation includes downvote information, the downvote rate can be determined by the ratio of the number of downvotes (i.e. the number of times the user selects "dissatisfied") to the total number of evaluations. The positive review rate is obtained by subtracting the downvote rate from 1. The positive review rate is then multiplied by 100 for standardization to obtain the user evaluation value.

[0163] The evaluation method for the retrieval enhancement generative model provided in this application standardizes various evaluation indicators, mapping the evaluation values ​​of each indicator to values ​​within a unified range. This facilitates further calculation of each evaluation value. By assigning different weights to each evaluation indicator, the weights can be adjusted according to different application scenarios, thereby optimizing the effectiveness and applicability of the evaluation results. Furthermore, by using multiple sets of question-and-answer statements to obtain multiple sets of evaluation indicators, the evaluation bias caused by inappropriate selection of a single question-and-answer statement can be reduced, improving the accuracy of the evaluation results.

[0164] Figure 6 This is a schematic diagram of the structure of an evaluation device for a retrieval-enhanced generative model provided in an embodiment of this application. Figure 6 As shown, the evaluation device 20 for retrieval-enhanced generative models includes: an acquisition module 21, a first processing module 22, and a second processing module 23, wherein,

[0165] The acquisition module 21 is used to acquire preset text, which includes at least one question-and-answer statement pair, and the question-and-answer statement pair includes a question-and-answer statement and a reference answer corresponding to the question-and-answer statement;

[0166] The first processing module 22 is used to process the preset text through the retrieval enhancement generation model to obtain the standard evaluation index and at least one specific evaluation index of the retrieval enhancement generation model, wherein the specific evaluation index is used to characterize the performance of the retrieval enhancement generation model in the target dimension.

[0167] The second processing module 23 is used to determine the evaluation result based on the standard evaluation indicators and the specific evaluation indicators.

[0168] In the above scheme, by constructing a hierarchical evaluation system that includes standard evaluation indicators and at least one special evaluation indicator, a multi-dimensional and scenario-customizable fine-grained performance evaluation of the retrieval enhancement generation model is achieved. The special evaluation indicator can evaluate the performance of the retrieval enhancement generation model in the target dimension based on the requirements of the target application scenario, thereby improving the adaptability of the model evaluation method and providing precise improvement directions for model optimization in different application scenarios.

[0169] In one possible implementation, the first processing module 22 is specifically used for,

[0170] Based on the preset text, a retrieval context is determined in the document database, the retrieval context including multiple text segments related to the question and answer statement;

[0171] Based on the retrieval context and the question-and-answer statements, generate the answer text using a large language model;

[0172] The standard evaluation metrics are determined based on the search context, the answer text, and the question-and-answer statements.

[0173] Obtain the target performance of the large language model when generating text answers, and determine the specific evaluation index based on the target performance.

[0174] In the above scheme, the preset text can be processed by the retrieval enhancement generation model, the standard evaluation index can be obtained based on the processing result, and the target performance can be obtained during the processing to obtain the specific evaluation index, so that the evaluation result can be further determined based on the standard evaluation index and the target evaluation index.

[0175] In one possible implementation, the first processing module 22 is specifically used for,

[0176] Based on a large language model, the answer text is decomposed into multiple atomic statements, and the loyalty is determined according to whether the multiple atomic statements can be supported by the retrieval context.

[0177] Based on a large language model, multiple potential questions are generated from the answer text, and the relevance of the answer is determined based on the semantic similarity between the potential questions and the question-and-answer statements.

[0178] Based on a large language model, multiple text segments related to the question and answer statement are determined from the retrieval context. The context relevance is determined by comparing the number of text segments related to the question and answer statement with the total number of text segments in the retrieval context.

[0179] In the above scheme, the consistency between the answer text and the retrieval context is measured by the loyalty index, the relevance between the answer text and the question and answer statements is measured by the answer relevance index, and the relevance between the retrieval context and the question and answer statements is measured by the context relevance index. This improves the credibility of the model output results and ensures that the model can generate more reliable and relevant answers when dealing with complex queries.

[0180] In one possible implementation, the first processing module 22 is specifically used for,

[0181] Based on the retrieval enhancement generation model, a first time and a second time are determined. The first time is used to indicate the time when the retrieval enhancement generation model receives the preset text, and the second time is used to indicate the time when the answer text is generated.

[0182] The response duration is determined based on the difference between the first time point and the second time point.

[0183] In the above scheme, by establishing a timestamp-based response duration quantification mechanism, the quantifiable tracking of model processing speed is realized, which can provide data support for model efficiency optimization and improve the adaptability of the evaluation framework in application scenarios where efficiency is required.

[0184] In one possible implementation, the first processing module 22 is specifically used for,

[0185] The device monitoring system is used to monitor the CPU and GPU of the computing device and obtain monitoring data, including CPU utilization and GPU utilization.

[0186] The resource utilization rate is determined based on the monitoring data of the computing device between the first time point and the second time point.

[0187] In the above solution, by monitoring and quantifying resource usage in real time, data support can be provided for system optimization, such as improving resource utilization efficiency by adjusting model complexity or hardware configuration.

[0188] In one possible implementation, the first processing module 22 is specifically used for,

[0189] Obtain user evaluation information for the answer text, including user downvote / dislike actions on the answer text, and / or user rating information for the answer text;

[0190] The user rating is determined based on the evaluation information.

[0191] In the above scheme, by collecting user evaluation information through the user evaluation mechanism, the model evaluation can be more comprehensive and closer to the actual use scenario. User evaluation can promote model optimization and improve user experience.

[0192] In one possible implementation, the second processing module 23 is specifically used for,

[0193] Based on the standard evaluation indicators, the evaluation values ​​corresponding to the standard evaluation indicators are determined. The evaluation values ​​corresponding to the standard evaluation indicators include loyalty evaluation value, answer relevance evaluation value, and context relevance evaluation value.

[0194] Based on the specific evaluation indicators, the evaluation values ​​corresponding to the specific evaluation indicators are determined. The evaluation values ​​corresponding to the specific evaluation indicators include resource utilization rate evaluation value, response time evaluation value, and user evaluation value.

[0195] The evaluation result is determined based on the evaluation value corresponding to the standard evaluation indicator, the evaluation value corresponding to the special evaluation indicator, and the weights corresponding to the standard evaluation indicator and the special evaluation indicator, respectively.

[0196] The above solution integrates standard evaluation indicators with special evaluation indicators in multiple dimensions and introduces configurable weighting strategies to achieve a dynamic balance between comprehensive performance and scenario-specific performance evaluation, effectively improving the adaptability of evaluation results to actual business needs.

[0197] In one possible implementation, the second processing module 23 is specifically used for,

[0198] Based on the multiple question-and-answer statements in the preset text, generate the retrieval context and answer text corresponding to each question-and-answer statement;

[0199] Based on the multiple question-and-answer statements, as well as the retrieval context and answer text corresponding to each question-and-answer statement, the resource utilization rate, response time, and user rating corresponding to each question-and-answer statement are determined.

[0200] Based on the resource utilization rate, response time, and user evaluation corresponding to each question and answer statement, determine the resource utilization rate evaluation value, response time evaluation value, and user evaluation evaluation value.

[0201] In the above scheme, by using multiple question-and-answer statements to obtain the corresponding standard evaluation indicators and special evaluation indicators respectively, the evaluation bias caused by improper selection of a single question-and-answer statement can be reduced, thereby improving the accuracy of the evaluation results.

[0202] The evaluation device for the retrieval enhancement generative model provided in this embodiment can execute the method shown in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0203] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 7 As shown, the computing device 30 may include a processor 31 and a memory 32, wherein the processor 31 and the memory 32 can communicate; for example, the processor 31 and the memory 32 communicate via a communication bus 33, the memory 32 is used to store computer execution instructions, and the processor 31 is used to call the computer execution instructions in the memory to execute the evaluation method of the retrieval enhancement generative model shown in any of the above method embodiments.

[0204] Optionally, the computing device 30 may also include a communication interface, which may include a transmitter and / or a receiver.

[0205] The computing device 30 can be the computing device shown in any of the above method embodiments, and can execute the evaluation method of the retrieval enhancement generative model shown in any of the above method embodiments.

[0206] Optionally, the processor mentioned above can be a CPU, or a GPU, a Baseboard Management Controller (BMC), other general-purpose processors, a Digital Signal Processor (DSP), or an Application Specific Integrated Circuit (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0207] This application provides a computer-readable storage medium storing computer-executable instructions; the computer-executable instructions are used to implement the text replacement method as described in any of the above embodiments.

[0208] This application provides a computer program product, which includes a computer program that, when executed, causes a computer to perform the above-described text replacement method.

[0209] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0210] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable terminal device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0211] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable terminal device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0213] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, the embodiments of this application are also intended to include these modifications and variations.

[0214] In the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. The term "or" and its variations can mean "and / or." In the embodiments of this application, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In the embodiments of this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0215] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the invention disclosed in the specification and in practice. The embodiments of this application are intended to cover any variations, uses, or adaptations of the embodiments of this application that follow the general principles of the embodiments of this application and include common knowledge or customary technical means in the art not disclosed in the embodiments of this application.

Claims

1. An evaluation method for a retrieval-enhanced generative model, characterized in that, include: Obtain a preset text, which includes at least one question-and-answer pair, the question-and-answer pair including a question and answer statement and a corresponding reference answer; By processing the preset text using a retrieval enhancement generation model, a standard evaluation index and at least one specific evaluation index of the retrieval enhancement generation model are obtained, wherein the specific evaluation index is used to characterize the performance of the retrieval enhancement generation model in the target dimension. The evaluation results are determined based on the standard evaluation indicators and the specific evaluation indicators.

2. The method according to claim 1, characterized in that, By processing the preset text using a retrieval-enhanced generation model, a standard evaluation metric and at least one specific evaluation metric for the retrieval-enhanced generation model are obtained, including: Based on the preset text, a retrieval context is determined in the document database, the retrieval context including multiple text segments related to the question and answer statement; Based on the retrieval context and the question-and-answer statements, generate the answer text using a large language model; The standard evaluation metrics are determined based on the search context, the answer text, and the question-and-answer statements. Obtain the target performance of the large language model when generating text answers, and determine the specific evaluation index based on the target performance.

3. The method according to claim 2, characterized in that, The standard evaluation metrics include loyalty, answer relevance, and contextual relevance. These metrics are determined based on the retrieval context, the answer text, and the question-and-answer statements, and include: Based on a large language model, the answer text is decomposed into multiple atomic statements, and the loyalty is determined according to whether the multiple atomic statements can be supported by the retrieval context. Based on a large language model, multiple potential questions are generated from the answer text, and the relevance of the answer is determined based on the semantic similarity between the potential questions and the question-and-answer statements. Based on a large language model, multiple text segments related to the question and answer statement are determined from the retrieval context. The context relevance is determined by comparing the number of text segments related to the question and answer statement with the total number of text segments in the retrieval context.

4. The method according to any one of claims 1-2, characterized in that, The specific evaluation indicators include response time. The specific evaluation indicators for determining the retrieval enhancement generation model include: Based on the retrieval enhancement generation model, a first time and a second time are determined. The first time is used to indicate the time when the retrieval enhancement generation model receives the preset text, and the second time is used to indicate the time when the answer text is generated. The response duration is determined based on the difference between the first time point and the second time point.

5. The method according to any one of claims 1-4, characterized in that, The specific evaluation indicators include resource utilization rate. The specific evaluation indicators for determining the retrieval enhancement generation model include: The device monitoring system is used to monitor the CPU and GPU of the computing device and obtain monitoring data, including CPU utilization and GPU utilization. The resource utilization rate is determined based on the monitoring data of the computing device between the first time point and the second time point.

6. The method according to any one of claims 1-2, characterized in that, The specific evaluation indicators include user reviews. The specific evaluation indicators for determining the retrieval enhancement generation model include: Obtain user evaluation information for the answer text, including user downvote / dislike actions on the answer text, and / or user rating information for the answer text; The user rating is determined based on the evaluation information.

7. The method according to any one of claims 1-3, characterized in that, The evaluation results are determined based on the standard evaluation indicators and the specific evaluation indicators, including: Based on the standard evaluation indicators, the evaluation values ​​corresponding to the standard evaluation indicators are determined. The evaluation values ​​corresponding to the standard evaluation indicators include loyalty evaluation value, answer relevance evaluation value, and context relevance evaluation value. Based on the specific evaluation indicators, the evaluation values ​​corresponding to the specific evaluation indicators are determined. The evaluation values ​​corresponding to the specific evaluation indicators include resource utilization rate evaluation value, response time evaluation value, and user evaluation value. The evaluation result is determined based on the evaluation value corresponding to the standard evaluation indicator, the evaluation value corresponding to the special evaluation indicator, and the weights corresponding to the standard evaluation indicator and the special evaluation indicator, respectively.

8. The method according to claim 7, characterized in that, The preset text includes at least two question-and-answer pairs. Based on the specific evaluation indicators, the evaluation values ​​corresponding to the specific evaluation indicators are determined, including: Based on the multiple question-and-answer statements in the preset text, generate the retrieval context and answer text corresponding to each question-and-answer statement; Based on the multiple question-and-answer statements, as well as the retrieval context and answer text corresponding to each question-and-answer statement, the resource utilization rate, response time, and user rating corresponding to each question-and-answer statement are determined. Based on the resource utilization rate, response time, and user evaluation corresponding to each question and answer statement, determine the resource utilization rate evaluation value, response time evaluation value, and user evaluation evaluation value.

9. An evaluation apparatus for a retrieval-enhanced generative model, characterized in that, include: The acquisition module is used to acquire preset text, which includes at least one question-and-answer statement pair, and the question-and-answer statement pair includes a question-and-answer statement and a corresponding reference answer. The first processing module is used to process the preset text through a retrieval enhancement generation model to obtain a standard evaluation index and at least one specific evaluation index of the retrieval enhancement generation model, wherein the specific evaluation index is used to characterize the performance of the retrieval enhancement generation model in the target dimension. The second processing module is used to determine the evaluation result based on the standard evaluation indicators and the specific evaluation indicators.

10. A computing device, characterized in that, include: Memory, processor; The memory is used to store computer-executed instructions; The processor executes computer instructions stored in the memory to cause the computing device to implement the method as described in any one of claims 1-8.

Images