System quality evaluation method and device, storage medium and program product

Abstract

The embodiment of the invention provides a system quality evaluation method and device, a storage medium and a program product. The method comprises the steps of obtaining a plurality of query statements and standard reply statements corresponding to the query statements respectively, inputting the query statements into a to-be-evaluated system, obtaining to-be-evaluated reply statements corresponding to the query statements respectively, and aiming at each query statement, obtaining a to-be-evaluated reply statement corresponding to the to-be-evaluated reply statement corresponding to the to-be-evaluated reply statement; inputting a standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model to obtain a corresponding target evaluation result, the target evaluation result being used for indicating a quality change condition between the to-be-evaluated reply statement and the standard reply statement, and determining a system quality evaluation result according to the target evaluation results corresponding to the plurality of query statements. According to the method provided by the embodiment of the invention, the effectiveness of system evaluation can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and in particular to a system quality evaluation method and device, a storage medium and a program product. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) systems directly affect decision reliability and user experience, and need to be verified through systematic evaluation to ensure the accuracy of their output, information relevance, and business scenario adaptability, and to ensure the actual value of technology landing.

[0003] In related technologies, the similarity between the reply statements output by the RAG system and the manually written reply statements can be calculated to determine the evaluation result.

[0004] However, during the implementation of the present application, the inventors found that at least the following problem exists: the effectiveness of system evaluation in the above manner is poor. SUMMARY

[0005] Embodiments of the present application provide a system quality evaluation method, device, storage medium and program product to improve the effectiveness of RAG system evaluation.

[0006] In a first aspect, embodiments of the present application provide a system quality evaluation method, comprising:

[0007] obtaining a plurality of query statements and a plurality of standard reply statements corresponding to the plurality of query statements, respectively;

[0008] inputting the plurality of query statements into a system to be evaluated to obtain a plurality of to-be-evaluated reply statements corresponding to the plurality of query statements, respectively;

[0009] for each of the query statements, inputting the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model to obtain a corresponding target evaluation result; the target evaluation result is used to indicate the quality change between the to-be-evaluated reply statement and the standard reply statement;

[0010] determining the system quality evaluation result according to the target evaluation result corresponding to each of the plurality of query statements.

[0011] In a possible design, the obtaining of the plurality of standard reply statements corresponding to the plurality of query statements, respectively, comprises:

[0012] inputting the plurality of query statements into a second model to obtain a plurality of initial reply statements corresponding to the plurality of query statements, respectively;

[0013] The standard reply statement corresponding to each of the query statements is determined according to an initial reply statement corresponding to each of the query statements.

[0014] In a possible design, the target evaluation result is obtained by inputting the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model.

[0015] The target evaluation result is obtained by inputting the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model.

[0016] The target evaluation result is obtained by inputting the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model.

[0017] The target evaluation result is obtained by fusing the positive evaluation result and the negative evaluation result.

[0018] In a possible design, the target evaluation result is obtained by fusing the positive evaluation result and the negative evaluation result.

[0019] The new positive result is obtained by performing negation on the negative evaluation result.

[0020] The target evaluation result is obtained by performing statistics on the positive evaluation result and the new positive result.

[0021] In a possible design, the system quality evaluation result is determined according to the target evaluation results corresponding to the query statements.

[0022] The first quantity of target evaluation results corresponding to the query statements that are improved, the second quantity of target evaluation results corresponding to the query statements that are unchanged, and the third quantity of target evaluation results corresponding to the query statements that are deteriorated are determined.

[0023] The system quality evaluation result is determined according to the first quantity, the second quantity, and the third quantity.

[0024] In a possible design, the system quality evaluation result is determined according to the first quantity, the second quantity, and the third quantity.

[0025] If there is a unique maximum value in the first quantity, the second quantity, and the third quantity, the target evaluation result corresponding to the unique maximum value is determined as the system quality evaluation result.

[0026] If two of the first number, the second number and the third number are equal and are the maximum, re-evaluation is performed.

[0027] In a possible design, the method further includes:

[0028] For each query statement in which the target evaluation result is improved, the corresponding standard reply statement of the query statement is updated by the corresponding to-be-evaluated reply statement, to obtain an updated standard reply statement.

[0029] In a second aspect, an embodiment of the present application provides a system quality evaluation device, including:

[0030] The acquisition module is configured to acquire a plurality of query statements and a plurality of standard reply statements corresponding to the query statements respectively;

[0031] The input module is configured to input the plurality of query statements into a to-be-evaluated system, to obtain a plurality of to-be-evaluated reply statements corresponding to the query statements respectively;

[0032] The input module is further configured to, for each query statement, input the corresponding standard reply statement and to-be-evaluated reply statement of the query statement into a first model, to obtain a corresponding target evaluation result; the target evaluation result is used to indicate a quality change between the to-be-evaluated reply statement and the standard reply statement;

[0033] The determination module is configured to determine the system quality evaluation result according to the target evaluation results corresponding to the plurality of query statements respectively.

[0034] In a third aspect, an embodiment of the present application provides a system quality evaluation device, including at least one processor and a memory;

[0035] The memory stores computer-executed instructions;

[0036] The at least one processor executes the computer-executed instructions stored in the memory, so that the at least one processor executes the method in the first aspect and various possible designs of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores computer-executed instructions, when a processor executes the computer-executed instructions, the method in the first aspect and various possible designs of the first aspect is implemented.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, when the computer program is executed by a processor, the method in the first aspect and various possible designs of the first aspect is implemented.

[0039] The system quality evaluation method, device, storage medium and program product provided by the embodiment, the method comprises: obtaining a plurality of query statements and a plurality of standard reply statements corresponding to the plurality of query statements respectively; inputting the plurality of query statements into a system to be evaluated to obtain a plurality of to-be-evaluated reply statements corresponding to the plurality of query statements respectively; for each query statement, inputting the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model to obtain a corresponding target evaluation result, the target evaluation result being used to indicate a quality change between the to-be-evaluated reply statement and the standard reply statement; and determining a system quality evaluation result according to the target evaluation result corresponding to each query statement. The method provided by the embodiment of the application can input the to-be-evaluated reply statement output by the to-be-evaluated system and the standard reply statement into a large language model at the same time, evaluate the change between the two reply statements by the large language model, obtain explicit evaluation results such as getting better and getting worse, and then evaluate the to-be-evaluated system based on the specific results, so as to effectively quantify the effect, improve the effectiveness of the evaluation, and automatically evaluate by using the large language model, so as to reduce the labor cost and improve the stability of the evaluation result. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0041] Figure 1 The application scenario diagram of the system quality evaluation method provided by the embodiment of the application is shown in the following figure.

[0042] Figure 2 The flowchart of the system quality evaluation method provided by the embodiment of the application is shown in the following figure. Figure 1

[0043] Figure 3 The principle diagram of the system quality evaluation method provided by the embodiment of the application is shown in the following figure.

[0044] Figure 4 The structure diagram of the system quality evaluation device provided by the embodiment of the application is shown in the following figure.

[0045] Figure 5 The hardware structure diagram of the electronic device provided by the embodiment of the application is shown in the following figure.

[0046] The specific embodiments of the application have been shown in the above-described figures, and will be described in more detail hereinafter. These figures and the written description are not intended to limit the scope of the inventive concept in any way, but to illustrate the inventive concept to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0047] ​In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0048] It should be noted that the system quality evaluation method, device, storage medium and program product provided by the present application can be used in the field of artificial intelligence, and can also be used in any field other than the field of artificial intelligence. The application field of the system quality evaluation method, device, storage medium and program product provided by the present application is not limited.

[0049] Retrieval-Augmented Generation (RAG) system effectively alleviates the illusion and knowledge timeliness problem of traditional large models by integrating information retrieval and large language model generation capabilities, dynamically obtaining relevant information from external knowledge base and generating targeted answers. Since the system directly affects the reliability of decision-making and user experience, it is necessary to verify the accuracy of its output, information relevance and business scenario adaptability through systematic evaluation to ensure the actual value of technology landing.

[0050] In the related art, an end-to-end evaluation method can be used, that is, without considering the intermediate links of the RAG system, the input query and the final model enhanced answer are directly evaluated, including fidelity, relevance, harmlessness, etc. For example, 100 query answer sentences are artificially prepared. When evaluating the RAG system, the similarity between the 100 query answer sentences output by the RAG system and the 100 query answer sentences artificially prepared is calculated to calculate the accuracy. However, in this method, the similarity calculated is usually a decimal less than 1, and it is difficult to know the advantages and disadvantages of the current version of the RAG system based on this decimal, and the effectiveness of the evaluation is poor.

[0051] To solve the above technical problems, the present inventors have found that by using a large language model (LLM) for each query, the difference between the answer sentence output by the RAG system and the standard answer sentence can be output to obtain a more explicit quantitative result, thereby improving the effectiveness of system evaluation. Based on this, the present application provides a system quality evaluation method.

[0052] Figure 1 The application scenario diagram of the system quality evaluation method provided by the present application is shown in FIG. 1. Figure 1The terminal device 101 is in communication connection with the server 102. The terminal device 101 is loaded with a first model (for example, a large language model) and a system to be evaluated (for example, an RAG system). The RAG system can be used for insurance professional consultation (for example, medical insurance, car insurance, claim settlement, underwriting, etc.), that is, to meet various inquiries of users on insurance products, insurance clauses. For the RAG system, it is necessary to quickly, frequently, and accurately iterate and optimize multiple online modules. The purpose of the present application is to scientifically, reasonably, and efficiently evaluate the end-to-end results during the optimization and online operation phase to ensure system stability.

[0053] In the specific implementation process, the server 102 collects a plurality of query sentences, the terminal device 101 obtains the plurality of query sentences from the server 102, and then obtains a plurality of standard reply sentences corresponding to the plurality of query sentences based on the plurality of query sentences, inputs the plurality of query sentences into the system to be evaluated, obtains a plurality of evaluated reply sentences corresponding to the plurality of query sentences, for each query sentence, inputs the standard reply sentence corresponding to the query sentence and the evaluated reply sentence into the first model, and obtains a corresponding target evaluation result; the target evaluation result is used to indicate the quality change between the evaluated reply sentence and the standard reply sentence, and the system quality evaluation result is determined according to the target evaluation result corresponding to the plurality of query sentences. The system quality evaluation method provided in the embodiment of the present application inputs the evaluated reply sentence output by the system to be evaluated and the standard reply sentence into the large language model at the same time, evaluates the change between the two kinds of reply sentences through the large language model, obtains clear evaluation results such as getting better and getting worse, and then evaluates the system to be evaluated based on these specific results, realizes effective quantification of the effect, improves the effectiveness of the evaluation, and automatically evaluates through the use of the large language model, which can reduce the labor cost and improve the stability of the evaluation result.

[0054] It should be noted that, Figure 1 The scene schematic diagram shown is only an example, and the system quality evaluation method and the scene described in the embodiment of the present application are used to more clearly illustrate the technical solutions of the embodiment of the present application, and do not constitute a limitation on the technical solutions provided by the embodiment of the present application. Those skilled in the art can know that, with the evolution of the system and the appearance of new business scenarios, the technical solutions provided by the embodiment of the present application are also applicable to similar technical problems.

[0055] The technical solutions of the present application will be described in detail in the following specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0056] Figure 2 Flowchart of the system quality evaluation method provided in the embodiment of the present application Figure 1 As shown in Figure 2As shown, the method comprises:

[0057] 201, acquire a plurality of query statements, and a plurality of query statements correspond to standard reply statements respectively.

[0058] The execution subject of the embodiment is a terminal device or a server, for example Figure 1 The terminal device shown.

[0059] In the embodiment, the standard reply statement refers to a statement used as a correct label value.

[0060] Specifically, a plurality of query statements can be collected to construct a test set. Specifically, the sources of query statements mainly include collecting online logs (for example, actual questions issued by users online, such as “How to reimburse out-of-town medical insurance”), expert preparation (experts prepare query statements according to existing knowledge base and different business scenarios), and automatic generation based on large models, so as to ensure that the test set covers the core capabilities of the system.

[0061] For example, the collection method of online logs can include: first, data screening can be performed: selecting actual user queries in the past 3 months (for example, sample size 100,000+). Second, the online logs obtained by screening can be cleaned, for example, sensitive information can be removed, and garbage / invalid queries (such as garbled codes, advertisements, etc.) can be filtered, and then normalized processing (combining different expressions of repeated semantics) is performed. Finally, the distribution ratio can be counted according to the dimensions of intent type, length, complexity, etc.

[0062] For the collection method of expert preparation, it can include: first, based on multi-dimensional functions such as function coverage matrix (core function × user role × use case), language expression variants (dialect, colloquial, professional terms, etc.), and boundary test cases (long text, special symbols, multi-language mixing, etc.), writing can be performed; second, the reply statements written can be controlled by establishing a labeling specification document (including positive and negative example standards) and implementing a cross-checking mechanism (expert mutual review + sampling review).

[0063] For the collection method of large model generation, it can include: adopting semantic expansion based on seed query (synonym replacement / sentence conversion, adversarial generation (inducing the model to produce easy-to-mistake cases), and long-tail scene completion (such as low-resource field query) generation strategies, by setting diversity constraints (controlling lexical diversity >0.6), establishing a reality checking mechanism (manual review ratio is recommended to be ≥20%) to ensure quality.

[0064] In some embodiments, obtaining the standard reply statement corresponding to each of the plurality of query statements can include: inputting the plurality of query statements into a second model to obtain an initial reply statement corresponding to each of the plurality of query statements; and determining the standard reply statement corresponding to each of the plurality of query statements according to the initial reply statement corresponding to each of the plurality of query statements. This embodiment realizes the automation and standardization of reply generation by obtaining the standard reply statement corresponding to the query statement, which brings significant beneficial effects. First, by inputting the query statement into the second model to generate the initial reply statement, manual intervention is reduced, efficiency and consistency are improved. Then, the standard reply statement is further determined according to the initial reply statement, which ensures the accuracy and quality of the reply. This process not only corrects the bias in the initial generation, but also provides a unified user experience, especially when processing a large number of queries. In addition, the automated process reduces the dependence on human resources, saves operating costs, and improves resource utilization efficiency. Finally, accurate and consistent replies improve user trust and satisfaction, enhancing overall user experience and system performance.

[0065] In this embodiment, the second model can be a large language model.

[0066] Specifically, the initial label label of the test set can be obtained as the standard reply statement. Obtaining the initial label is to obtain the answer corresponding to all queries in the test set, which is a cold start problem, that is, a batch of queries are given, how to obtain the initial answer. A simple way is to input the queries in the test set into a general large model in batches. The general large model lacks professional knowledge base in the insurance field, and the accuracy and pertinence of its answers may be lacking in some specific problems, but this has little effect here. This step only initializes the label of the test set.

[0067] Considering that the higher the quality of the initial label is, the more beneficial it is to improve the effectiveness and accuracy of the evaluation results, therefore, in order to improve the quality of the initial label, the quality of the label can be controlled from three aspects of generation strategy, quality control and optimization iteration.

[0068] Regarding the generation strategy, firstly, prompt engineering can be implemented to strengthen the prompts. Specifically, phased guidance can be used, for example, asking you, as an insurance expert, to answer in the following steps: 1. Analyze the core demands and potential implicit needs in the query; 2. Identify semantic points that may cause ambiguity; 3. The structured output includes: the core answer (no more than 3 sentences), key evidence (citing authoritative sources), and precautions (common misunderstandings or risk warnings). Secondly, adversarial constraints can be implemented, such as requiring answers to meet the following conditions: 1. If the information is uncertain, it must be clearly stated; 2. If numerical calculations are designed, the derivation process must be shown; 3. When multiple possibilities exist, a probability assessment should be provided. Thirdly, generation parameters can be optimized: for example, reducing randomness, maintaining a certain degree of diversity, controlling output length, avoiding repetition, using cluster retrieval to improve coherence, and generating deterministic results. Finally, multi-model collaborative validation can be implemented, for example, using the consensus results of multiple models.

[0069] Regarding quality control (post-processing and verification), firstly, automated quality filtering can be implemented: for example, building a quality filtering pipeline including factual consistency checks, logical coherence analysis, and security compliance reviews. Secondly, knowledge tracing verification can be performed, such as extracting assertive statements and comparing them with a knowledge base.

[0070] For optimization iteration, error pattern analysis, dynamic prompt word optimization (prompt iteration based on error analysis), and reinforcement learning fine-tuning (reinforcement learning from human feedback (RLHF) process) can be performed.

[0071] 202. Input multiple query statements into the system to be evaluated, and obtain the corresponding response statements to be evaluated for each query statement.

[0072] In this embodiment, the system to be evaluated can be a RAG system used for insurance consulting.

[0073] Specifically, after obtaining multiple query statements and their corresponding standard response statements, the multiple query statements can be input into the system to be evaluated, thereby obtaining the response statements to be evaluated corresponding to the multiple query statements output by the system to be evaluated.

[0074] 203. For each query statement, input the standard response statement and the response statement to be evaluated into the first model to obtain the corresponding target evaluation result; the target evaluation result is used to indicate the quality change between the response statement to be evaluated and the standard response statement.

[0075] In this embodiment, the first model can be a large language model.

[0076] Specifically, after obtaining the to-be-evaluated reply statement and the standard reply statement corresponding to each query statement, the two can be combined and input into a large language model to obtain a target evaluation result, which can explicitly indicate the advantages and disadvantages between the to-be-evaluated reply statement and the standard reply statement.

[0077] In some embodiments, there are various ways to obtain the target evaluation result, for example, the target evaluation result can be obtained by using a forward evaluation method, the target evaluation result can also be obtained by using a reverse evaluation method, and the results obtained by the forward evaluation method and the reverse evaluation method can be fused to obtain the target evaluation result.

[0078] Specifically, in this embodiment, the forward and reverse evaluation methods are taken as examples for illustration. In this embodiment, the standard reply statement and the to-be-evaluated reply statement corresponding to the query statement are input into the first model to obtain the corresponding target evaluation result, which can include: inputting the standard reply statement and the to-be-evaluated reply statement corresponding to the query statement into the first model to obtain the corresponding forward evaluation result; the forward evaluation result is used to indicate the quality change of the to-be-evaluated reply statement relative to the standard reply statement; inputting the to-be-evaluated reply statement and the standard reply statement corresponding to the query statement into the first model to obtain the corresponding reverse evaluation result; the reverse evaluation result is used to indicate the quality change of the standard reply statement relative to the to-be-evaluated reply statement; and the forward evaluation result and the reverse evaluation result are fused to obtain the target evaluation result. This embodiment performs quality evaluation on the reply statement through the forward and reverse evaluation methods, which brings significant beneficial effects. First, the forward and reverse evaluation provides a double verification mechanism, ensuring the reliability and accuracy of the evaluation result from two directions. Secondly, by fusing the forward and reverse evaluation results, the system obtains a more comprehensive quality evaluation perspective, eliminating the bias that may be caused by a single evaluation. Avoiding instability caused by hallucinations of the first model.

[0079] For example, in the forward evaluation process, the standard reply statement can be referred to as label, and the to-be-evaluated statement can be referred to as response. The label and the response of the same query are combined and input into a large language model for evaluation. The input prompt word Prompt can be as follows:

[0080] You as an evaluation expert of the RAG system evaluate the following results.

[0081] ## User input: {query}

[0082] ## Initialization answer: {label}

[0083] ## Updated answer: {response}

[0084] Please evaluate the initialized answer (i.e., the standard reply statement) and the updated answer (i.e., the reply statement to be evaluated) from the perspectives of authority, timeliness, conciseness, and readability. Provide the changes in the updated answer compared to the initialized answer and your evaluation reasons.

[0085] Wherein, the change can be represented by good, same, and bad, corresponding to improvement, no change, and deterioration, respectively.

[0086] In addition, the large model can output the reasoning process to describe why the evaluation result is good, bad, etc. This helps to improve the accuracy of the final conclusion. It also helps to analyze and summarize the reasons for bad results into several categories (such as newly added documents in the knowledge base leading to enhanced answers) to determine the optimization direction.

[0087] Collect the evaluation results of multiple queries in the test set, extract the change from the output results of the large model, and obtain good:same:bad (hereinafter referred to as GSB). It should be noted that, in order to avoid unstable output of the evaluation model, N requests can be made for each query.

[0088] In the reverse evaluation process, the label and response in the prompt used in the forward evaluation can be interchanged to obtain the reverse evaluation results of each query statement.

[0089] In some embodiments, the forward evaluation results and the reverse evaluation results are fused to obtain target evaluation results, which can include: inverting the reverse evaluation results to obtain new forward results; and statistically combining the forward evaluation results and the new forward results to obtain the target evaluation results. This embodiment fuses the forward and reverse evaluation results to determine the target evaluation results, which brings significant beneficial effects. First, by inverting the reverse evaluation results and fusing them with the forward results, a more comprehensive and objective evaluation perspective is provided, improving the accuracy of the evaluation.

[0090] Specifically, in the process of fusing the forward evaluation results and the reverse evaluation results, the forward evaluation results and the reverse evaluation results can be merged according to the query, and the reverse evaluation results are inverted, i.e., G becomes B, S remains unchanged, and B becomes G. After merging, there will be 2N results for each query, including forward and reverse results.

[0091] 204、According to the target evaluation results corresponding to the plurality of query statements, determine the system quality evaluation result.

[0092] Specifically, after obtaining the target evaluation results, the system quality evaluation result can be obtained by statistically analyzing the different values of the target evaluation results of each query statement.

[0093] In some embodiments, determining the system quality evaluation result according to the target evaluation results corresponding to the plurality of query statements can include: determining a first number of results that are better, a second number of results that are unchanged, and a third number of results that are worse in the target evaluation results corresponding to the plurality of query statements; and determining the system quality evaluation result according to the first number, the second number, and the third number. This embodiment optimizes the system quality evaluation method by analyzing the target evaluation results of the plurality of query statements, bringing significant beneficial effects. First, by quantifying the number of better, unchanged, and worse results, a more comprehensive and objective evaluation perspective is provided, improving the accuracy and reliability of the evaluation. Second, this method enhances decision support capabilities, allowing managers to more clearly understand system performance changes and make more targeted improvement decisions. In addition, the system can dynamically adjust strategies and resource allocation based on evaluation results, improving adaptability. Ultimately, this continuous monitoring and evaluation mechanism helps to identify and address potential problems in a timely manner, improving user experience and satisfaction.

[0094] For example, assuming 20 query statements, the target evaluation results of the first, second, and third query statements are worse (bad), the target evaluation results of the fourth and fifth query statements are unchanged (same), and the target evaluation results of the remaining query statements are better (good). Then, the first number is 15, the second number is 2, and the third number is 3. Optionally, to improve accuracy, each query statement can be input multiple times, such as 20 times, and the total results obtained by inputting the 20 query statements 20 times can be evaluated based on this strategy.

[0095] In some embodiments, determining the system quality evaluation result according to the first number, the second number, and the third number can include: if there is a unique maximum value among the first number, the second number, and the third number, determining the target evaluation result corresponding to the unique maximum value as the system quality evaluation result; and if there are two equal maximum values among the first number, the second number, and the third number, re-evaluating. This embodiment optimizes the method of determining the system quality evaluation result, significantly improving the accuracy and efficiency of the evaluation. Specifically, by identifying the unique maximum value to directly determine the evaluation result, the decision-making process is simplified, complex calculations are reduced, and the clarity and credibility of the evaluation are improved. At the same time, when there are two equal maximum values, the system will re-evaluate to ensure the reliability of the results and enhance the robustness of the system. This mechanism not only improves resource utilization efficiency but also reduces unnecessary resource consumption.

[0096] For example, as Figure 3As shown, a specific strategy can be used to determine the system quality evaluation result based on the target evaluation result. For example, assuming that the number of evaluation results is x, this strategy can be that when x > argmax{G, S, B} except x, x, it is considered that the query evaluation is valid and the result is x; that is, x needs to be strictly greater than the maximum value in the parameter set except itself. Assuming that the first number is 15, the second number is 2, and the third number is 3, then the first number 15 is the strict maximum value, and it can be determined that the system quality evaluation result is getting better, and the publishing can be performed.

[0097] Assuming that GSB = 7:6:7, that is, the first number and the third number are equal and are the maximum value, then at this time the query evaluation is invalid, and subsequent manual intervention is required.

[0098] Alternatively, when the evaluation is invalid, m times of requests can be performed to obtain a valid evaluation result.

[0099] By obtaining the response to the original reference label, the GSB result can be completely and accurately evaluated.

[0100] The system quality evaluation method provided in this embodiment can input the to-be-evaluated reply statement output by the to-be-evaluated system and the standard reply statement into the large language model at the same time, evaluate the change between the two reply statements through the large language model, obtain explicit evaluation results such as getting better and getting worse, and then evaluate the to-be-evaluated system based on these specific results, effectively quantify the effect, improve the effectiveness of the evaluation, and automatically evaluate by using the large language model, which can reduce the labor cost and improve the stability of the evaluation result.

[0101] In some embodiments, in order to improve the accuracy of the standard reply statement and avoid resource consumption caused by manual preparation, the standard reply statement can be automatically updated based on the evaluation result, so as to obtain a standard reply statement with higher accuracy. Specifically, the method further comprises: for each query statement in the plurality of query statements with a target evaluation result of getting better, updating the standard reply statement corresponding to the to-be-evaluated reply statement corresponding to the query statement, to obtain an updated standard reply statement. In this embodiment, the method of automatically updating the standard reply statement significantly improves the accuracy and efficiency of the reply. The automatic updating mechanism reduces the dependence on manual preparation and saves the labor and time costs.

[0102] Specifically, after obtaining the target evaluation result, the to-be-evaluated reply statement corresponding to the good result can be used to update the label of the original test set to form a new version of the evaluation test set. For example, Figure 3As shown, after obtaining the target evaluation result by the first model, the good-to-be-evaluated reply statement in the target evaluation result can be updated to the standard reply statement.

[0103] In some embodiments, in order to ensure accuracy, the good result can be manually detected or cross-validated by multiple models before updating. The good result can also be further corrected before updating.

[0104] Considering that the initial label is not a real label in the actual sense, it is initialized by using a general large model and does not have the knowledge of the rag knowledge base, and the accuracy, authority, etc. There is still a gap from the real label. The higher the quality of the evaluation set, the better the evaluation effect. The way of manually writing the evaluation set is huge, so here a gradual way is adopted to get a better evaluation set.

[0105] In some embodiments, only the good result can be updated, or the entire result of the winning to-be-evaluated system can be updated. The difference between the two methods is that the update range is different. The update range of the former is larger, and it is easier to get a high-quality evaluation set, and the subsequent model is more difficult to go online. The update range of the latter is smaller, which reduces the difficulty of the subsequent model going online. The update strategy can be adopted according to the actual situation. For example, the first few iterations can use a large-scale update strategy. When the overall effect tends to be stable, a small-scale update strategy can be used to reduce the difficulty of going online while reflecting the subtle effect of improvement more obviously. This embodiment does not limit this, and the actual needs can be determined.

[0106] Figure 4 The structural schematic diagram of the system quality evaluation device provided by the embodiment of the application is shown in FIG. 1. Figure 4 As shown, the system quality evaluation device 40 includes an acquisition module 401, an input module 402, and a determination module 403.

[0107] The acquisition module 401 is configured to acquire a plurality of query statements and a plurality of standard reply statements corresponding to the plurality of query statements respectively.

[0108] The input module 402 is configured to input the plurality of query statements into a to-be-evaluated system to obtain a plurality of to-be-evaluated reply statements corresponding to the plurality of query statements respectively.

[0109] The input module 402 is further configured to input, for each query statement, the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into a first model to obtain a corresponding target evaluation result. The target evaluation result is used to indicate a quality change between the to-be-evaluated reply statement and the standard reply statement.

[0110] The determination module 403 is configured to determine a system quality evaluation result according to the target evaluation results corresponding to the plurality of query statements respectively.

[0111] The system quality evaluation device provided by the embodiment of the application can input the to-be-evaluated reply statement and the standard reply statement output by the to-be-evaluated system into the large language model at the same time, evaluate the change between the two reply statements through the large language model, obtain clear evaluation results such as getting better and getting worse, and then evaluate the to-be-evaluated system based on the specific results, effectively quantify the effect, improve the effectiveness of the evaluation, and automatically evaluate by using the large language model, which can reduce the labor cost and improve the stability of the evaluation results.

[0112] In some embodiments, the acquisition module 401 is specifically configured to: input the plurality of query statements into the second model to obtain initial reply statements corresponding to the plurality of query statements respectively; and determine standard reply statements corresponding to the plurality of query statements respectively according to the initial reply statements corresponding to the plurality of query statements respectively.

[0113] In some embodiments, the input module 402 is specifically configured to: input the standard reply statement corresponding to the query statement and the to-be-evaluated reply statement into the first model to obtain a corresponding positive evaluation result; the positive evaluation result is used to indicate the quality change of the to-be-evaluated reply statement relative to the standard reply statement; input the to-be-evaluated reply statement corresponding to the query statement and the standard reply statement into the first model to obtain a corresponding reverse evaluation result; the reverse evaluation result is used to indicate the quality change of the standard reply statement relative to the to-be-evaluated reply statement; and fuse the positive evaluation result and the reverse evaluation result to obtain a target evaluation result.

[0114] In some embodiments, the input module 402 is specifically configured to: perform negation on the reverse evaluation result to obtain a new positive result; and perform statistics on the positive evaluation result and the new positive result to obtain the target evaluation result.

[0115] In some embodiments, the determination module 403 is specifically configured to: determine a first number of results that get better, a second number of results that remain unchanged, and a third number of results that get worse in the target evaluation results corresponding to the plurality of query statements respectively; and determine a system quality evaluation result according to the first number, the second number, and the third number.

[0116] In some embodiments, the determination module 403 is specifically configured to: if there is a unique maximum value in the first number, the second number, and the third number, determine the target evaluation result corresponding to the unique maximum value as the system quality evaluation result; and if there are two equal maximum values in the first number, the second number, and the third number, re-evaluate.

[0117] In some embodiments, the determining module 403 is further configured to: for each query statement in the plurality of query statements whose target evaluation result is improved, update the corresponding standard response statement to the standard response statement corresponding to the query statement, and obtain the updated standard response statement.

[0118] The system quality assessment device provided in this application embodiment can be used to execute the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0119] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0120] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0121] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0122] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0123] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0124] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0125] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described above.

[0126] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0127] The readable storage medium described above can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0128] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0129] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0131] In addition, each functional unit in various embodiments of the application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0132] If the function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiment methods of the application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0133] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0134] Finally, it should be noted that those skilled in the art, after considering the specification and practicing the application disclosed herein, will easily think of other embodiments of the application. The application is intended to cover any variations, uses, or adaptations of the application that follow the general principles of the application and include common knowledge or conventional techniques in the art that are not disclosed by the application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the application is only limited by the appended claims.

Claims

1. A system quality assessment method, characterized in that, include: Obtain multiple query statements, and the standard response statements corresponding to each of the query statements; Input multiple query statements into the system to be evaluated to obtain the response statements to be evaluated corresponding to each of the query statements. For each query statement, the standard response statement and the response statement to be evaluated corresponding to the query statement are input into the first model to obtain the corresponding target evaluation result; The target evaluation result is used to indicate the quality change between the response statement to be evaluated and the standard response statement; The system quality assessment result is determined based on the target assessment results corresponding to the multiple query statements.

2. The method according to claim 1, characterized in that, The step of obtaining the standard response statements corresponding to the multiple query statements includes: Input multiple query statements into the second model to obtain initial response statements corresponding to each of the query statements; Based on the initial response statements corresponding to the multiple query statements, the standard response statements corresponding to the multiple query statements are determined.

3. The method according to claim 1, characterized in that, The step of inputting the standard response statement and the response statement to be evaluated corresponding to the query statement into the first model to obtain the corresponding target evaluation result includes: The standard response statement and the response statement to be evaluated corresponding to the query statement are input into the first model to obtain the corresponding positive evaluation result; the positive evaluation result is used to indicate the quality change of the response statement to be evaluated relative to the standard response statement; The query statement and the standard response statement are input into the first model to obtain the corresponding reverse evaluation result; the reverse evaluation result is used to indicate the quality change of the standard response statement relative to the query statement. The positive evaluation results and the negative evaluation results are fused to obtain the target evaluation result.

4. The method according to claim 3, characterized in that, The step of fusing the positive evaluation result and the reverse evaluation result to obtain the target evaluation result includes: The reverse evaluation result is inverted to obtain a new positive result; The positive evaluation results and the new positive results are statistically analyzed to obtain the target evaluation result.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the system quality assessment result based on the target assessment results corresponding to the multiple query statements includes: Among the target evaluation results corresponding to the multiple query statements, the first number is the one that improves, the second number is the one that remains unchanged, and the third number is the one that worsens. The system quality assessment result is determined based on the first quantity, the second quantity, and the third quantity.

6. The method according to claim 5, characterized in that, Determining the system quality assessment result based on the first quantity, the second quantity, and the third quantity includes: If there is a unique maximum value among the first quantity, the second quantity, and the third quantity, then the target evaluation result corresponding to the unique maximum value is determined as the system quality evaluation result; If two of the first quantity, the second quantity, and the third quantity are equal and represent the maximum value, then the evaluation is repeated.

7. The method according to claim 5, characterized in that, The method further includes: For each query statement whose target evaluation result is "improved" among the multiple query statements, the corresponding standard response statement is updated by updating the response statement to be evaluated for the query statement to obtain the updated standard response statement.

8. A system quality assessment device, characterized in that, include: The acquisition module is used to acquire multiple query statements and the standard response statements corresponding to each of the multiple query statements; The input module is used to input multiple query statements into the system to be evaluated and obtain the response statements to be evaluated corresponding to the multiple query statements respectively. The input module is further configured to input the standard response statement and the response statement to be evaluated corresponding to each query statement into the first model to obtain the corresponding target evaluation result; the target evaluation result is used to indicate the quality change between the response statement to be evaluated and the standard response statement. The determination module is used to determine the system quality assessment result based on the target assessment results corresponding to the multiple query statements.

9. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the system quality assessment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the system quality assessment method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the system quality assessment method according to any one of claims 1 to 7.

Images