Model reply quality evaluation method and related equipment

Abstract

The invention provides a model reply quality evaluation method and related equipment. The method comprises the following steps: acquiring a question text and a plurality of reply texts obtained by replying the question text by using a plurality of target models; performing reply quality evaluation on the plurality of reply texts through a plurality of different role models, and determining a plurality of multi-role evaluation results corresponding to the plurality of reply texts; sequencing the reply qualities of the reply texts from a plurality of different dimensions, and determining a multi-dimensional sequencing result of the reply texts; and according to the multiple multi-role evaluation results and the multi-dimensional sorting result, determining quality evaluation results of the multiple reply texts. According to the method, prejudice problems in the evaluation process can be reduced, fine-grained evaluation is realized, and the quality evaluation accuracy of model reply is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for evaluating model response quality, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of computer technology, language models have emerged. Language models can be applied to the processing of different natural language tasks. Language models can analyze question texts and output corresponding reply texts.

[0003] In order to evaluate the natural language processing capabilities of different language models, in related technologies, the response quality of different language models under the same question text is usually evaluated based on a voting mechanism.

[0004] However, the above-mentioned evaluation method of reply text based on voting mechanism can only provide overall and macro quality evaluation results. Bias problems are prone to occur in the evaluation process, and the accuracy of the quality evaluation results is low. Summary of the invention

[0005] The present application provides a method for evaluating the quality of model responses. The method can reduce bias problems in the evaluation process, achieve fine-grained evaluation, and improve the accuracy of response text evaluation. The present application also provides a device, electronic device, computer-readable storage medium, and computer program product corresponding to the above method.

[0006] In a first aspect, the present application provides a method for evaluating the quality of a model response, the method comprising:

[0007] Obtaining a question text and a plurality of reply texts obtained by respectively replying to the question text using a plurality of target models;

[0008] Performing response quality evaluation on the multiple response texts respectively through multiple different role models, and determining multiple multi-role evaluation results corresponding to the multiple response texts respectively;

[0009] Sorting the reply qualities of the multiple reply texts from multiple different dimensions to determine a multi-dimensional sorting result of the multiple reply texts;

[0010] The quality evaluation results of the multiple reply texts are determined according to the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0011] In a second aspect, the present application provides a device for evaluating model response quality, the device comprising:

[0012] An acquisition module, used for acquiring a question text and a plurality of reply texts obtained by respectively replying to the question text using a plurality of target models;

[0013] A first determination module is used to respectively evaluate the response quality of the plurality of response texts through a plurality of different role models, and determine a plurality of multi-role evaluation results respectively corresponding to the plurality of response texts;

[0014] A second determination module is used to sort the reply qualities of the multiple reply texts from multiple different dimensions to determine a multi-dimensional sorting result of the multiple reply texts;

[0015] The third determination module is used to determine the quality evaluation results of the multiple reply texts according to the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0016] In a third aspect, the present application provides an electronic device, the electronic device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory, so that the electronic device performs the method for evaluating the model response quality in the first aspect or any implementation of the first aspect.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct an electronic device to execute the method for evaluating the model response quality described in the first aspect or any one of the implementations of the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method for evaluating the model response quality described in the first aspect or any one of the implementations of the first aspect.

[0019] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

[0020] It can be seen from the above technical solutions that this application has the following advantages:

[0021] The present application provides a method for evaluating the quality of model responses. The method first obtains a question text and multiple response texts obtained by respectively responding to the question text using multiple target models, then evaluates the response quality of the multiple response texts using multiple different role models, determines multiple multi-role evaluation results corresponding to the multiple response texts, sorts the response qualities of the multiple response texts from multiple different dimensions, determines the multi-dimensional sorting results of the multiple response texts, and then determines the quality evaluation results of the multiple response texts based on the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0022] In this method, in the scenario of quality assessment of model responses, the results of multi-role assessment and multi-dimensional assessment are combined to determine the final quality assessment results of multiple model responses. Since multi-role assessment involves multiple different roles performing quality assessment on the response text separately, it can reduce bias in the assessment process. Since multi-dimensional assessment involves performing quality assessment on the response text separately from different dimensions, it can achieve fine-grained assessment, thereby improving the accuracy of quality assessment of model responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0024] Figure 1 A flowchart of a method for evaluating model response quality provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for evaluating model response quality provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a device for evaluating model response quality provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0029] First, some technical terms and application scenarios involved in the embodiments of this application are introduced.

[0030] With the continuous development of computer technology, language models have emerged. Language models can be applied to the processing of different natural language tasks. For example, language models can be applied to question-answering tasks. Users can input question text, and the language model analyzes the question text and outputs the corresponding reply text.

[0031] Due to the large number and types of language models, in order to evaluate the natural language processing capabilities of different language models, in related technologies, the reply quality of different language models under the same question text is usually evaluated based on a voting mechanism. Specifically, multiple other language models (i.e., multiple language models that are not involved in the question text reply) are used to analyze multiple reply texts respectively, output the evaluation results of multiple reply texts, and then the final quality evaluation result is obtained based on the voting mechanism according to the evaluation results of multiple reply texts output by other multiple language models.

[0032] For example, multiple reply texts include reply text A and reply text B. Language model A, language model B, and language model C are used to evaluate the reply texts. The evaluation results of the multiple reply texts output by language model A indicate that "the reply quality of reply text A is better than the reply quality of reply text B", the evaluation results of the multiple reply texts output by language model B indicate that "the reply quality of reply text A is better than the reply quality of reply text B", and the evaluation results of the multiple reply texts output by language model C indicate that "the reply quality of reply text B is better than the reply quality of reply text A". Since the evaluation results of the multiple reply texts output by two language models indicate that "the reply quality of reply text A is better than the reply quality of reply text B", based on the voting mechanism, the final quality evaluation result is determined to be "the reply quality of reply text A is better than the reply quality of reply text B".

[0033] However, the above method has the following defects: First, the method of evaluating reply texts based on voting mechanisms often lacks transparency and explanation, and the final quality evaluation results only rely on the evaluation results of multiple reply texts output by multiple language models. In addition, the method of evaluating reply texts based on voting mechanisms can only provide overall and macro quality evaluation results, and it is difficult to provide a more detailed analysis, and the accuracy of the quality evaluation results is low. Moreover, if multiple language models have common bias problems, the method of evaluating reply texts based on voting mechanisms cannot eliminate the bias, and may even amplify the impact of the bias, ultimately leading to inaccurate quality evaluation results.

[0034] In view of this, the present application provides a method for evaluating the quality of model responses, which first obtains a question text and multiple response texts obtained by respectively responding to the question text using multiple target models, then evaluates the response quality of the multiple response texts through multiple different role models, determines multiple multi-role evaluation results corresponding to the multiple response texts, sorts the response qualities of the multiple response texts from multiple different dimensions, determines the multi-dimensional sorting results of the multiple response texts, and then determines the quality evaluation results of the multiple response texts based on the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0035] In this method, in the scenario of quality assessment of model responses, the results of multi-role assessment and multi-dimensional assessment are combined to determine the final quality assessment results of multiple model responses. Since multi-role assessment involves multiple different roles performing quality assessment on the response text separately, it can reduce bias in the assessment process. Since multi-dimensional assessment involves performing quality assessment on the response text separately from different dimensions, it can achieve fine-grained assessment, thereby improving the accuracy of quality assessment of model responses.

[0036] To facilitate understanding of the technical solution provided by the embodiments of the present application, the following will be described in conjunction with the accompanying drawings. Figure 1 A flow chart of a method for evaluating the quality of a model response is shown, and the method specifically includes:

[0037] S101: Obtain a question text and a plurality of reply texts obtained by respectively replying to the question text using a plurality of target models.

[0038] The question text may be understood as a text representing a question, and the reply text may be understood as a text including an answer to the question text. In other words, the question text may be used to ask a question, and the reply text may be used to answer the question.

[0039] In the embodiment of the present application, since it is necessary to perform quality assessment on the reply texts of different models, there may be multiple reply texts. In addition, both the question text and the reply text represent natural language content, in other words, both the question text and the reply text are described in natural language.

[0040] Specifically, the multiple reply texts may be reply texts generated after multiple target models reply to the question texts. The target model may be a model with question reply capability. In some embodiments, the target model may be a language model with natural language processing capability, capable of understanding the meaning of natural language, and processing different types of natural language tasks. For example, the target model may be a deep learning model trained using text data.

[0041] For example, the question text is sent to multiple target models respectively, and the multiple target models analyze the question text, respectively generate and return reply texts corresponding to the question text, and obtain multiple reply texts.

[0042] The multiple target models are different models, such as language models provided by different service providers, language models with different model parameters, or language models of different versions. In this way, for the same question text, multiple different target models are used to answer respectively, and multiple reply texts are obtained. The reply quality of the multiple reply texts is subsequently evaluated to achieve the evaluation of the reasoning ability of the multiple target models.

[0043] S102: Perform reply quality evaluation on multiple reply texts respectively through multiple different role models, and determine multiple multi-role evaluation results corresponding to the multiple reply texts respectively.

[0044] Among them, one multi-role evaluation result corresponds to one reply text, that is, for each reply text in the multiple reply texts, the corresponding multi-role evaluation result is determined respectively, and the multi-role evaluation result of each reply text in the multiple reply texts constitutes multiple multi-role evaluation results.

[0045] A multi-role evaluation result is determined by evaluating the response quality of a response text corresponding to the multi-role evaluation result through multiple different role models. That is, for each response text in the multiple response texts, the response quality of the response text is evaluated by simulating multiple different roles to determine the multi-role evaluation result corresponding to the response text.

[0046] Among them, the role model can be understood as a model for characterizing different roles and used for reply quality assessment. In the embodiment of the present application, the role can be understood as an identity. Different roles are different identities. Multiple different roles have knowledge in different knowledge fields, different behavior patterns and ways of thinking. For example, multiple different roles can be university teachers, computer practitioners, physics professors, middle school students, etc. By evaluating the reply quality of the same reply text through multiple different role models, the reply quality of the reply text can be more comprehensively evaluated from the perspectives of different roles.

[0047] It should be noted that a role model may correspond to one role, or a role model may correspond to multiple different roles, which is not limited in the embodiments of the present application.

[0048] In specific implementation, the quality of each reply text is evaluated through multiple different role models to determine multiple multi-role evaluation sub-results of each reply text, and the multi-role evaluation result of each reply text is determined based on the multiple multi-role evaluation sub-results of each reply text, where one multi-role evaluation sub-result corresponds to one role.

[0049] That is to say, the process of determining the multi-role evaluation result of each reply text in the multiple reply texts is divided into two steps: first, for each of the multiple roles, the corresponding multi-role evaluation sub-result is determined respectively, and the multi-role evaluation sub-result of each of the multiple roles constitutes the multiple multi-role evaluation sub-results of the reply text. Then, the multi-role evaluation result of the reply text is determined by combining the multiple multi-role evaluation sub-results of the reply text.

[0050] For example, the reply text includes reply text A and reply text B, and the multiple roles include role A and role B. For reply text A, the role model corresponding to role A evaluates the reply quality of reply text A to obtain the multi-role evaluation sub-result of role A, the role model corresponding to role B evaluates the reply quality of reply text A to obtain the multi-role evaluation sub-result of role B, and the multi-role evaluation sub-result of role A and the multi-role evaluation sub-result of role B are combined to determine the multi-role evaluation result of reply text A. Similarly, for reply text B, the role model corresponding to role A evaluates the reply quality of reply text B to obtain the multi-role evaluation sub-result of role A, the role model corresponding to role B evaluates the reply quality of reply text B to obtain the multi-role evaluation sub-result of role B, and the multi-role evaluation sub-result of role A and the multi-role evaluation sub-result of role B are combined to determine the multi-role evaluation result of reply text B, and the multi-role evaluation result of reply text A and the multi-role evaluation sub-result of role B are combined to determine the multi-role evaluation result of reply text B, and the multi-role evaluation result of reply text A and the multi-role evaluation result of reply text B constitute multiple multi-role evaluation results to achieve multi-role labeling.

[0051] In some embodiments, multiple different role models can be implemented with the help of a language model, thereby determining multiple multi-role evaluation results for each reply text. The language model can simulate multiple different roles based on prompt learning, thereby determining multiple multi-role evaluation results for each reply text. Among them, prompt words can be used to guide the language model to perform specific outputs in generative tasks (such as text generation tasks, question-and-answer tasks, and dialogue tasks). By configuring prompt words, the language model is helped to understand the background and requirements of the task, so that the language model can handle different types of natural language processing tasks without the need to retrain the language model, thereby increasing the scalability and flexibility of the language model.

[0052] In specific implementation, for each reply text among the multiple reply texts, the following steps are performed: multiple first prompt words are generated, the multiple first prompt words are sent to at least one first language model respectively, and multiple multi-role evaluation sub-results of the reply text returned by at least one first language model are received.

[0053] Among them, the first language model can be understood as a language model used to simulate multiple different roles and determine multi-role evaluation sub-results. A first prompt word corresponds to a role, and a first prompt word may include: question text, the reply text, first information used to indicate the role corresponding to the first prompt word, and second information used to indicate a reply quality assessment of the reply text in combination with the question text.

[0054] That is to say, for each reply text among the multiple reply texts, a first prompt word corresponding to a different role is generated respectively, and the multiple first prompt words are sent to the first language model respectively. The natural language processing capability of the first language model is used to simulate the reply quality evaluation process of the role for the reply text, and the multi-role evaluation sub-results under the role are output, thereby determining the multiple multi-role evaluation sub-results of the reply text.

[0055] Since the first prompt word includes the question text, the reply text, the first information indicating the role corresponding to the first prompt word, and the second information indicating the reply quality evaluation of the reply text in combination with the question text, the first language model can simulate the behavior pattern and thinking mode of the role based on the prompt ability of the first prompt word, and evaluate the reply quality of the reply text in combination with the question text, and then output the reply text and the multi-role evaluation sub-result of the role. In this way, for each reply text in the multiple reply texts, by sending the multiple first prompt words to the first language model respectively, receiving the multi-role evaluation sub-results of different roles output by the first language model respectively, the multiple multi-role evaluation sub-results of the reply text are determined.

[0056] The embodiments of the present application do not limit the first language model. For example, for each reply text in multiple reply texts, multiple first prompt words can be sent to the same first language model respectively, and the same first language model can be used to simulate different roles. Alternatively, multiple first prompt words can be sent to different first language models respectively, and different first language models can be used to simulate different roles.

[0057] After obtaining multiple multi-role evaluation sub-results of each reply text, the multi-role evaluation result of each reply text can be determined. In some embodiments, the multi-role evaluation result of each reply text can be determined by the average value of multiple multi-role evaluation sub-results of each reply text. For example, the multi-role evaluation sub-result may include the evaluation grade of the reply text under the role, such as the grade of "excellent", "medium", "poor", or the score of "3 points", "2 points", "1 point", the reply text includes reply text A and reply text B, the multiple multi-role evaluation sub-results of reply text A are "3 points" and "2 points", the multiple multi-role evaluation sub-results of reply text B are "2 points" and "1 point", then the multi-role evaluation result of reply text A can be "2.5 points", and the multi-role evaluation result of reply text B can be "1.5 points".

[0058] In other embodiments, different roles may also be configured with different weights. For example, role A has a knowledge reserve in the same knowledge field as the question text, role A has a higher weight (e.g., 2), role B has a lower weight (e.g., 1), the reply text includes reply text A and reply text B, for reply text A, role A's multi-role evaluation sub-result is "3 points", role B's multi-role evaluation sub-result is "2 points", for reply text B, role A's multi-role evaluation sub-result is "2 points", role B's multi-role evaluation sub-result is "1 point", then the multi-role evaluation result of reply text A can be "2.67 points", and the multi-role evaluation result of reply text B can be "1.67 points".

[0059] S103: Sort the reply qualities of the multiple reply texts from multiple different dimensions to determine a multi-dimensional sorting result of the multiple reply texts.

[0060] In an embodiment of the present application, the reply quality of multiple reply texts is sorted under multiple different dimensions to determine a multi-dimensional sorting result. Among them, dimensions can be understood as levels and aspects, and different dimensions are different levels and different aspects. In other words, by evaluating the reply quality of multiple reply texts from multiple different dimensions, and then combining the results of evaluating the reply quality of multiple reply texts from multiple different dimensions, a multi-dimensional sorting result is determined. For example, in the multi-dimensional sorting result, the reply texts with high reply quality are ranked first, and the reply texts with low reply quality are ranked last.

[0061] In some embodiments, the multiple different dimensions may include at least two of the following: a dimension for determining whether the reply text meets the question and answer requirements, a dimension for determining whether there are errors in the reply text, a dimension for determining whether the reply text responds from different angles, a dimension for determining whether the reply text contains repeated content, and a dimension for determining whether the text structure of the reply text is reasonable.

[0062] Among them, the dimension of judging whether the reply text meets the question and answer requirements can be used to examine whether the reply text answers the question and answer requirements of the question text; the dimension of judging whether there are errors in the reply text can be used to examine whether there are errors in the reply text that are inconsistent with objective facts, public consensus, etc.; the dimension of judging whether the reply text replies from different angles can be used to examine whether the reply text answers the question text comprehensively; the dimension of judging whether there is repeated content in the reply text can be used to examine whether the content of the reply text is refined; and the dimension of judging whether the text structure of the reply text is reasonable can be used to examine whether reading the reply text is convenient.

[0063] In this way, by combining multiple different dimensions to sort the reply qualities of multiple reply texts, it is possible to compare the reply qualities of multiple reply texts in a fine-grained manner, thereby improving the level of detail in evaluating the reply qualities of the reply texts.

[0064] In specific implementation, the response quality of multiple response texts is evaluated from multiple different dimensions, the multi-dimensional evaluation sub-result corresponding to each dimension is determined, the multiple response texts are sorted according to the multi-dimensional evaluation sub-result corresponding to each dimension, and the multi-dimensional sorting results of the multiple response texts are determined.

[0065] That is to say, the process of determining the multi-dimensional ranking results of the multiple reply texts is divided into two steps: first, for each dimension of the multiple dimensions, the corresponding multi-dimensional evaluation sub-results are determined respectively. Then, the multiple reply texts are sorted according to the multi-dimensional evaluation results corresponding to each dimension to determine the multi-dimensional ranking results.

[0066] For example, the reply text includes reply text A and reply text B, and the multiple dimensions include dimension A, dimension B, and dimension C. The reply quality of reply text A and reply text B is evaluated from dimension A to determine the multi-dimensional evaluation sub-result corresponding to dimension A, the reply quality of reply text A and reply text B is evaluated from dimension B to determine the multi-dimensional evaluation sub-result corresponding to dimension B, the reply quality of reply text A and reply text B is evaluated from dimension C to determine the multi-dimensional evaluation sub-result corresponding to dimension C, and after the reply quality of reply text A and reply text B is evaluated from dimensions A, B, and C, the reply text A and reply text B are sorted in combination with the multi-dimensional evaluation sub-result corresponding to dimension A, the multi-dimensional evaluation sub-result corresponding to dimension B, and the multi-dimensional evaluation sub-result corresponding to dimension C to obtain a multi-dimensional sorting result, thereby realizing hierarchical labeling.

[0067] In some embodiments, the multi-dimensional evaluation sub-result corresponding to each dimension can be determined with the help of a language model. Similarly, the language model can determine the multi-dimensional evaluation sub-result corresponding to each dimension based on prompt learning. In specific implementation, for each dimension in a plurality of different dimensions, the following steps are performed: generating a second prompt word, sending the second prompt word to a second language model, and receiving the multi-dimensional evaluation sub-result corresponding to the dimension returned by the second language model.

[0068] Among them, the second language model can be understood as a language model used to determine the multi-dimensional evaluation sub-result corresponding to each dimension, and the second prompt word can include: question text, multiple reply texts, third information used to indicate the dimension, and fourth information used to indicate that the reply quality of multiple reply texts is evaluated from this dimension in combination with the question text.

[0069] That is to say, for each dimension of multiple different dimensions, a corresponding second prompt word is generated, and the second prompt word is sent to the second language model. The natural language processing capability of the second language model is used to evaluate the reply quality of multiple reply texts from this dimension, and a multi-dimensional evaluation sub-result after the reply quality evaluation of the multiple reply texts from this dimension is output.

[0070] Since the second prompt word includes the question text, multiple reply texts, the third information indicating the dimension, and the fourth information indicating that the reply quality of the multiple reply texts is evaluated from the dimension in combination with the question text, the second language model can evaluate the reply quality of the multiple reply texts from the dimension in combination with the question text based on the prompt capability of the second prompt word, and then output the multi-dimensional evaluation sub-result of the dimension. In this way, for each of the multiple dimensions, by sending the second prompt words corresponding to different dimensions to the second language model, receiving the multi-dimensional evaluation sub-results of different dimensions output by the second language model, the multi-dimensional evaluation sub-result corresponding to each dimension is determined.

[0071] Furthermore, considering that the second prompt word includes multiple reply texts, in order to avoid the influence of the order of the multiple reply texts on the second language model, the order of the multiple reply texts in the second prompt word can be randomly arranged.

[0072] For example, an isolation mark is added between multiple reply texts of the second prompt word to distinguish different reply texts. Then, the order of the multiple reply texts is randomly disrupted so that the order of the multiple reply texts in the second prompt word is not fixed, thereby reducing the position bias of the second language model.

[0073] In some embodiments, in order to enhance the degree of association between the multidimensional evaluation sub-results and the question text, at least one response point associated with the question text can also be determined. In this case, the second prompt word also includes at least one response point associated with the question text, and the third information in the second prompt word for indicating the dimension can be the third information for indicating the dimension related to at least one response point associated with the question text.

[0074] That is to say, by determining at least one response point associated with the question text, multiple different dimensions can be related to the response point. In this way, when the third language model evaluates the response quality of multiple response texts from multiple different dimensions, it can conduct targeted evaluation in combination with the response points associated with the question text, thereby improving the accuracy of the response text quality evaluation.

[0075] The process of determining at least one reply key point associated with the question text can also be implemented with the help of a language model. In specific implementation, a fourth prompt word is generated, the fourth prompt word is sent to a fourth language model, and at least one reply key point associated with the question text returned by the fourth language model is received.

[0076] The fourth language model may be understood as a language model used to extract the key points of the reply in combination with the question text, and the fourth prompt word may include: the question text and sixth information used to indicate the key points of the reply generated from the question text.

[0077] Since the fourth prompt word includes the question text and the sixth information for indicating the key points of the response to the generated question text, the fourth language model can analyze the question text based on the prompting ability of the fourth prompt word and extract at least one key point of the response associated with the question text. In this way, it is convenient to combine the key points of the response associated with the question text and evaluate the response quality of multiple response texts in a targeted and accurate manner from multiple different dimensions.

[0078] After obtaining the multi-dimensional evaluation sub-result corresponding to each dimension, the multi-dimensional sorting results of the multiple reply texts can be determined. In some possible implementations, the multi-dimensional sorting results of the multiple reply texts can be determined with the help of a language model. In a specific implementation, a third prompt word is generated, the third prompt word is sent to a third language model, and the multi-dimensional sorting results of the multiple reply texts returned by the third language model are received.

[0079] Among them, the third language model can be understood as a language model used to sort multiple reply texts in combination with the multidimensional evaluation sub-results corresponding to each dimension, and then determine the multidimensional sorting results of the multiple reply texts. The third prompt word may include: question text, multiple reply texts, multidimensional evaluation sub-results corresponding to each dimension, and fifth information used to indicate that multiple reply texts are sorted in combination with the multidimensional evaluation sub-results corresponding to each dimension.

[0080] Since the third prompt word includes the question text, multiple reply texts, the multidimensional evaluation sub-result corresponding to each dimension, and the fifth information for indicating that the multiple reply texts are to be sorted in combination with the multidimensional evaluation sub-result corresponding to each dimension, the third language model can sort the reply qualities of the multiple reply texts based on the prompt ability of the third prompt word, in combination with the question text and the multidimensional evaluation sub-result corresponding to each dimension, and then output the multidimensional sorting result.

[0081] Similarly, considering that the third prompt word includes multiple reply texts, in order to avoid the influence of the order of the multiple reply texts on the third language model, the order of the multiple reply texts in the third prompt word can be randomly arranged.

[0082] For example, an isolation mark is added between multiple reply texts of the third prompt word to distinguish different reply texts. Then, the order of the multiple reply texts is randomly disrupted so that the order of the multiple reply texts in the third prompt word is not fixed, thereby reducing the position bias of the third language model.

[0083] In addition, considering that the third prompt word includes the multidimensional evaluation sub-results corresponding to each dimension, in order to avoid the influence of the order of the multidimensional evaluation sub-results corresponding to each dimension on the third language model, the order of the multidimensional evaluation sub-results corresponding to each dimension in the third prompt word can also be randomly arranged.

[0084] In this way, the third language model is prevented from overly relying on the multi-dimensional evaluation sub-result corresponding to a certain dimension (for example, a dimension at the front or the back) to sort multiple reply texts, thereby improving the accuracy of the multi-dimensional sorting results.

[0085] It should be noted that the embodiment of the present application does not limit the execution order of S102 and S103. For example, S102 and S103 can be executed simultaneously, or S102 can be executed first and then S103, or S103 can be executed first and then S102.

[0086] S104: Determine quality evaluation results of multiple reply texts based on multiple multi-role evaluation results and multi-dimensional sorting results.

[0087] In the embodiment of the present application, multiple multi-role evaluation results can respectively characterize the reply quality of each reply text, and the multi-dimensional sorting result can uniformly characterize the order relationship between the reply qualities of multiple reply texts by sorting. Therefore, by judging whether the multiple multi-role evaluation results and the multi-dimensional sorting results are consistent, the results obtained by the multi-role evaluation and the multi-dimensional evaluation are verified to determine the final quality evaluation result.

[0088] In some possible implementations, it is determined whether the reply qualities of multiple reply texts represented by multiple multi-role evaluation results are consistent with the reply qualities of multiple reply texts represented by multi-dimensional sorting results. In response to the reply qualities of multiple reply texts represented by multiple multi-role evaluation results being consistent with the reply qualities of multiple reply texts represented by multi-dimensional sorting results, the multiple multi-role evaluation results and the multi-dimensional sorting results are determined as quality evaluation results of the multiple reply texts.

[0089] That is to say, when the results obtained by the multi-role evaluation and the multi-dimensional evaluation are consistent, the multiple multi-role evaluation results and the multi-dimensional sorting results are directly used as the final quality evaluation results. For example, the multi-role evaluation results include the evaluation level (such as the score) of the reply text corresponding to the multi-role evaluation result. After sorting the multiple reply texts according to the multiple multi-role evaluation results (such as the score), if they are the same as the multi-dimensional sorting results, it means that the reply text with better reply quality identified in the multi-role evaluation is still identified as having better reply quality in the multi-dimensional evaluation, and the evaluation results of the two methods are the same. The quality evaluation results of the multiple reply texts include the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0090] In response to the inconsistency between the reply quality of multiple reply texts represented by multiple multi-role evaluation results and the reply quality of multiple reply texts represented by multi-dimensional sorting results, at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results is modified, and the modified multiple multi-role evaluation results and multi-dimensional sorting results are determined as quality evaluation results of the multiple reply texts.

[0091] That is to say, when the results obtained by the multi-role evaluation and the multi-dimensional evaluation are inconsistent, the inconsistent content is modified, and the modified multiple multi-role evaluation results and / or multi-dimensional sorting results are used as the final quality evaluation results. In this way, the verification of multiple multi-role evaluation results and multi-dimensional sorting results is completed to ensure that there are no contradictions in the quality evaluation results of the multiple reply texts obtained in the end.

[0092] In some embodiments, the multi-role evaluation result includes an evaluation grade (e.g., a score) of a reply text corresponding to the multi-role evaluation result, and the multi-dimensional sorting result is determined based on the multi-dimensional evaluation sub-result corresponding to each dimension, and the multi-dimensional evaluation sub-result corresponding to each dimension includes the evaluation grade (e.g., a score) of the multiple reply texts under each dimension. In this case, the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results and the reply quality of the multiple reply texts represented by the multi-dimensional sorting result can be verified according to the set verification rules. In response to the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results and the reply quality of the multiple reply texts represented by the multi-dimensional sorting result satisfying the verification rules, it is determined that the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results is consistent with the reply quality of the multiple reply texts represented by the multi-dimensional sorting result, otherwise, it is determined that the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results is inconsistent with the reply quality of the multiple reply texts represented by the multi-dimensional sorting result.

[0093] In other words, verification rules are pre-configured, and when the reply qualities of multiple reply texts represented by multiple multi-role evaluation results and the reply qualities of multiple reply texts represented by multi-dimensional sorting results both meet the verification rules, it indicates that the results obtained by the multi-role evaluation and the multi-dimensional evaluation are consistent; otherwise, it indicates that the results obtained by the multi-role evaluation and the multi-dimensional evaluation are inconsistent.

[0094] The verification rules may include at least one of the following: there is no contradiction between the evaluation levels of the multiple reply texts in the multiple multi-role evaluation results and the order of the multiple reply texts in the multi-dimensional sorting result, there is no contradiction between the order of the multiple reply texts in the multi-dimensional sorting result and the evaluation levels of the multiple reply texts in the multi-dimensional evaluation sub-results corresponding to each dimension, and there is no contradiction between the order of the multiple reply texts in the multi-dimensional sorting result and the order of the multiple reply texts in the multi-dimensional sorting result of the multiple reply texts determined again.

[0095] Among them, the multi-dimensional sorting result of the multiple reply texts that are re-determined can be understood as randomly rearranging the order of the multiple reply texts in the third prompt word, and / or randomly rearranging the order of the multi-dimensional evaluation sub-results corresponding to each dimension in the third prompt word, sending the third prompt word including the multiple reply texts that are randomly arranged again and / or the multi-dimensional evaluation sub-results corresponding to each dimension that are randomly arranged again to the third language model, and receiving the multi-dimensional sorting result of the multiple reply texts that are re-determined and returned by the third language model.

[0096] Taking the case where multiple multi-role evaluation results are presented in the form of scores, the multi-dimensional evaluation sub-results corresponding to each dimension are presented in the form of scores, and in the multi-dimensional sorting results, the order of the reply text with high reply quality is at the front, and the order of the reply text with low reply quality is at the back, as an example, there is no contradiction between the evaluation levels of multiple reply texts in the multiple multi-role evaluation results and the order of multiple reply texts in the multi-dimensional sorting results. It can be understood that: the reply text with a higher evaluation level (i.e., score) in the multi-role evaluation results is also at the front in the multi-dimensional sorting results. For example, the score of reply text A in the multi-role evaluation results is 3 points, the score of reply text B in the multi-role evaluation results is 4 points, and the multi-dimensional sorting result is "reply text A>reply text B". At this time, there is a contradiction between the evaluation levels of multiple reply texts in the multiple multi-role evaluation results and the order of multiple reply texts in the multi-dimensional sorting results.

[0097] There is no contradiction between the order of multiple reply texts in the multi-dimensional sorting result and the evaluation levels of multiple reply texts in the multi-dimensional evaluation sub-results corresponding to each dimension. It can be understood that: the reply text with a higher evaluation level (i.e., score) in the multi-dimensional evaluation sub-results corresponding to each dimension is also ranked higher in the multi-dimensional sorting result. For example, the scores of reply text A in 5 dimensions are [4, 4, 3, 3, 2], and the scores of reply text B in 5 dimensions are [3, 3, 3, 3, 1]. The multi-dimensional sorting result is "reply text A < reply text B". At this time, there is a contradiction between the order of multiple reply texts in the multi-dimensional sorting result and the evaluation levels of multiple reply texts in the multi-dimensional evaluation sub-results corresponding to each dimension.

[0098] The order of the multiple reply texts in the multi-dimensional sorting result does not contradict the order of the multiple reply texts in the multi-dimensional sorting result of the multiple reply texts that are determined again. It can be understood that: the reply texts that are ranked higher in the multi-dimensional sorting result are also ranked higher in the multi-dimensional sorting result of the multiple reply texts that are determined again. For example, the multi-dimensional sorting result is "reply text A < reply text B", and the multi-dimensional sorting result that is determined again is "reply text A > reply text B". At this time, the order of the multiple reply texts in the multi-dimensional sorting result contradicts the order of the multiple reply texts in the multi-dimensional sorting result of the multiple reply texts that are determined again.

[0099] When the results obtained by the multi-role evaluation and the multi-dimensional evaluation are inconsistent, at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results can be modified. In some embodiments, at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results can be modified with the help of a language model. In specific implementation, in response to the inconsistency between the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results and the reply quality of the multiple reply texts represented by the multi-dimensional sorting results, a fifth prompt word is generated, the fifth prompt word is sent to the fifth language model, and the modified multiple multi-role evaluation results and the multi-dimensional sorting results returned by the fifth language model are received.

[0100] Among them, the fifth language model can be understood as a language model used to modify at least one of multiple multi-role evaluation results and multi-dimensional sorting results, and the fifth prompt word can include: question text, multiple reply texts, multiple multi-role evaluation results, multi-dimensional sorting results and seventh information used to indicate the modification of at least one of multiple multi-role evaluation results and multi-dimensional sorting results.

[0101] Since the fifth prompt word includes the question text, multiple reply texts, multiple multi-role evaluation results, multi-dimensional sorting results, and the seventh information for indicating the modification of at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results, the fifth language model can modify the inconsistent content between the multiple multi-role evaluation results and the multi-dimensional sorting results based on the prompting ability of the fifth prompt word, combined with the question text and the multiple reply texts, and then output the modified multiple multi-role evaluation results and the multi-dimensional sorting results. In this way, the inconsistent content between the multiple multi-role evaluation results and the multi-dimensional sorting results is eliminated, and the quality evaluation results of the multiple reply texts include the modified multiple multi-role evaluation results and the multi-dimensional sorting results.

[0102] It should be noted that the embodiments of the present application do not limit the first language model, the second language model, the third language model, the fourth language model, and the fifth language model. For example, the first language model, the second language model, the third language model, the fourth language model, and the fifth language model may be the same language model, or the first language model, the second language model, the third language model, the fourth language model, and the fifth language model may be different language models.

[0103] In this method, in the scenario of quality assessment of model responses, the results of multi-role assessment and multi-dimensional assessment are combined to determine the final quality assessment results of multiple model responses. Since multi-role assessment involves multiple different roles performing quality assessment on the response text separately, it can reduce bias in the assessment process. Since multi-dimensional assessment involves performing quality assessment on the response text separately from different dimensions, it can achieve fine-grained assessment, thereby improving the accuracy of quality assessment of model responses.

[0104] The previous article describes the evaluation method for the response text provided by this application. Figure 2 The evaluation process of the response text is described in detail.

[0105] See also Figure 2 First, obtain the question text and multiple response texts associated with the question text. In some examples, the question text is "Does the light of the moon have temperature?", and the response text A is "The moon itself does not emit light. The moonlight we see is the sunlight reflected from the surface of the moon. After reaching the moon, the sunlight is reflected back to the earth by the surface of the moon. In this process, the energy of the light is mainly transmitted in the form of photons, which carry energy but do not have temperature in the traditional sense. Therefore, the light reflected by the moon itself has no temperature", and the response text B is "The moon itself does not emit light. The light of the moon we see is actually the sunlight reflected from the surface of the moon. Therefore, the light reflected by the moon itself has no temperature, but the sunlight has temperature".

[0106] In the multi-role evaluation, multiple different roles correspond to different multi-role evaluation sub-results. When determining multiple multi-role evaluation sub-results with the help of the first language model, the first prompt word can be as follows:

[0107] "You are xxx. Please evaluate the quality of your response based on the question text and the response text, combined with your professional knowledge. The evaluation score is 1-5 points.

[0108] Question text: [question text]

[0109] Reply text: [reply text]"

[0110] Among them, "Question text: [question text]" is the question text in the first prompt word, "Reply text: [reply text]" is the reply text in the first prompt word, [reply text] can be reply text A or reply text B, "You are an xxx" is the first information in the first prompt word used to indicate the role corresponding to the first prompt word, xxx can be different roles such as university teacher, computer practitioner, physics professor, etc., "Evaluate the reply quality based on your professional knowledge, and the evaluation score is 1-5 points" is the second information in the first prompt word used to indicate to evaluate the reply quality of the reply text based on the question text.

[0111] After obtaining multiple multi-role evaluation sub-results, the multi-role evaluation result of each response text may be determined to complete the multi-role evaluation.

[0112] In a multidimensional evaluation, different dimensions correspond to different multidimensional evaluation sub-results. When determining the multidimensional evaluation sub-result corresponding to each dimension with the help of a second language model, the second prompt word may be as follows:

[0113] "You are an evaluation expert. Please evaluate the response text A and the response text B based on the following dimensions based on the question text and give the scores for the response text A and the response text B.

[0114] Question text: [question text]

[0115] Reply Text A: [Reply Text A]

[0116] Reply Text B: [Reply Text B]

[0117] Dimension requirement: [dimension requirement]"

[0118] Among them, "Question text: [Question text]" is the question text in the second prompt word, "Reply text A: [Reply text A]" and "Reply text B: [Reply text B]" are multiple reply texts in the second prompt word, "Dimension requirement: [Dimension requirement]" is the third information in the second prompt word used to indicate the dimension, and "You are an evaluation expert. Please evaluate reply text A and reply text B from the following dimensions in combination with the question text and give scores for reply text A and reply text B" is the fourth information in the second prompt word used to indicate the evaluation of the reply quality of multiple reply texts from this dimension in combination with the question text.

[0119] In some possible implementations, the second prompt word may further include at least one reply key point associated with the question text. When extracting the reply key point associated with the question text with the help of the fourth language model, the fourth prompt word may be as follows:

[0120] "Please combine the question text and extract the key points that need to be included in answering the question text.

[0121] Question text: [question text]"

[0122] Among them, "question text: [question text]" is the question text in the fourth prompt word, and "please combine the question text to extract the key points of the response that need to be included in answering the question text" is the sixth information in the fourth prompt word used to indicate the key points of the response to generate the question text.

[0123] In some examples, the response bullet points associated with the question text may be as follows:

[0124] "Does the moon's light have temperature?

[0125] The source of the moon's light

[0126] The moon's light is essentially reflected sunlight

[0127] Sunlight contains components such as infrared rays that can generate heat

[0128] Temperature changes on the lunar surface

[0129] "The feeling of night in the moonlit area"

[0130] In this case, the third information for indicating the dimension in the second prompt word may be the third information for indicating the dimension related to at least one reply key point associated with the question text, and the “dimension requirement: [dimension requirement]” in the second prompt word may be any one of the following:

[0131] “Needs Satisfaction

[0132] Combined with the reply points, examine whether the reply text answers the user's primary and secondary needs in accordance with their intentions and demands.

[0133] Main requirement: refers to the core information that needs to be provided to meet the question text

[0134] Secondary demand: refers to the extended demand other than the constraints or primary demand.

[0135] “True Accuracy

[0136] Combined with the key points of the reply, examine whether the reply text contains errors in objective facts or errors in public consensus.

[0137] “Comprehensive

[0138] Examine whether the reply text can focus on the main needs and provide information with obvious gains in breadth and depth in addition to covering the key points of the reply.

[0139] "Refined

[0140] Check whether there is any repeated expression in the reply text, including extreme repetition, paragraph repetition, sentence repetition, and whether the content is over-expressed and unrefined.

[0141] “Reading convenience

[0142] Check whether the structure of the reply text is reasonable, whether the main content is prominent in structure and position, and whether it is easy to read. For example, if the layout of a large text is chaotic, the main content is placed at the back, etc., it will make it difficult for users to quickly find the main content.

[0143] After obtaining the multi-dimensional evaluation sub-results corresponding to each dimension, the multi-dimensional ranking results of the multiple reply texts can be determined. When the multi-dimensional ranking results of the multiple reply texts are determined with the help of the third language model, the third prompt word can be as follows:

[0144] "You are an evaluation expert. Please combine the question text and the multi-dimensional evaluation sub-results that evaluate the response quality of response text A and response text B from multiple dimensions, sort the response quality of response text A and response text B, and return which of the response quality of response text A and response text B is better, which is worse, or the same.

[0145] Question text: [question text]

[0146] Reply Text A: [Reply Text A]

[0147] Reply Text B: [Reply Text B]

[0148] The multi-dimensional evaluation sub-results corresponding to dimension A: [The multi-dimensional evaluation sub-results corresponding to dimension A]

[0149] The multi-dimensional evaluation sub-results corresponding to dimension B: [The multi-dimensional evaluation sub-results corresponding to dimension B]

[0150] The multi-dimensional evaluation sub-results corresponding to dimension C: [The multi-dimensional evaluation sub-results corresponding to dimension C]”

[0151] Among them, "question text: [question text]" is the question text in the third prompt word, "reply text A: [reply text A]" and "reply text B: [reply text B]" are multiple reply texts in the third prompt word, "multi-dimensional evaluation sub-result corresponding to dimension A: [multi-dimensional evaluation sub-result corresponding to dimension A]", "multi-dimensional evaluation sub-result corresponding to dimension B: [multi-dimensional evaluation sub-result corresponding to dimension B]" and "multi-dimensional evaluation sub-result corresponding to dimension C: [multi-dimensional evaluation sub-result corresponding to dimension C]" are the multi-dimensional evaluation sub-results corresponding to each dimension in the third prompt word, and "as an evaluation expert, please rank the reply qualities of reply text A and reply text B based on the question text and the multi-dimensional evaluation sub-results that evaluate the reply qualities of reply text A and reply text B from multiple dimensions, and return which of the reply qualities of reply text A and reply text B is better, worse, or the same" is the fifth information in the third prompt word used to indicate that multiple reply texts are to be ranked based on the multi-dimensional evaluation sub-results corresponding to each dimension.

[0152] After completing the multi-role evaluation and the multi-dimensional evaluation, determine whether the results obtained by the multi-role evaluation and the multi-dimensional evaluation are consistent. When the results obtained by the multi-role evaluation and the multi-dimensional evaluation are inconsistent, modify at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0153] When modifying at least one of the multiple multi-role evaluation results and the multi-dimensional ranking results by means of the fifth language model, the fifth prompt word may be as follows:

[0154] "You are an evaluation expert. Please arbitrate the results of evaluating the response quality of response text A and response text B based on the question text, multiple multi-role evaluation results, and multi-dimensional ranking results. Based on the inconsistent content, give modified multiple multi-role evaluation results and multi-dimensional ranking results.

[0155] Question text: [question text]

[0156] Reply Text A: [Reply Text A]

[0157] Reply Text B: [Reply Text B]

[0158] Multiple multi-role evaluation results: [Multiple multi-role evaluation results]

[0159] Multi-dimensional sorting results: [Multi-dimensional sorting results]"

[0160] Among them, "question text: [question text]" is the question text in the fifth prompt word, "reply text A: [reply text A]" and "reply text B: [reply text B]" are multiple reply texts in the fifth prompt word, "multiple multi-role evaluation results: [multiple multi-role evaluation results]" are multiple multi-role evaluation results in the fifth prompt word, and "multi-dimensional sorting result: [multi-dimensional sorting result]" is the multi-dimensional sorting result in the fifth prompt word. "You are an evaluation expert. Please arbitrate the results of evaluating the reply quality of reply text A and reply text B based on the question text, multiple multi-role evaluation results and multi-dimensional sorting results, and give modified multiple multi-role evaluation results and multi-dimensional sorting results based on inconsistent content" is the seventh information in the fifth prompt word used to indicate the modification of at least one of the multiple multi-role evaluation results and multi-dimensional sorting results.

[0161] In this way, the results obtained from multi-role evaluation and multi-dimensional evaluation are combined to determine the final quality assessment result.

[0162] Combination of the above Figure 1 and Figure 2 The evaluation method of the model response quality provided in the embodiment of the present application is introduced in detail. The following will introduce the device and equipment provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0163] See also Figure 3 The schematic diagram of the structure of the device for evaluating the model response quality shown in FIG. 30 includes:

[0164] An acquisition module 301 is used to acquire a question text and a plurality of reply texts obtained by replying to the question text using a plurality of target models respectively;

[0165] A first determination module 302 is used to respectively evaluate the response quality of the plurality of response texts through a plurality of different role models, and determine a plurality of multi-role evaluation results respectively corresponding to the plurality of response texts;

[0166] A second determination module 303 is used to sort the reply qualities of the multiple reply texts from multiple different dimensions to determine a multi-dimensional sorting result of the multiple reply texts;

[0167] The third determination module 304 is used to determine the quality evaluation results of the multiple reply texts according to the multiple multi-role evaluation results and the multi-dimensional sorting results.

[0168] In some possible implementations, the first determining module 302 is specifically configured to:

[0169] Performing a reply quality assessment on each reply text through a plurality of different role models, and determining a plurality of multi-role assessment sub-results of each reply text; wherein one multi-role assessment sub-result corresponds to one role;

[0170] The multi-role evaluation result of each reply text is determined according to the multiple multi-role evaluation sub-results of each reply text.

[0171] In some possible implementations, the first determining module 302 is specifically configured to:

[0172] For each reply text in the plurality of reply texts, perform the following steps:

[0173] Generate multiple first prompt words; wherein one of the first prompt words corresponds to one role, and one of the first prompt words includes: the question text, the reply text, first information for indicating the role corresponding to the first prompt word, and second information for indicating that the reply text is evaluated for reply quality in combination with the question text;

[0174] The multiple first prompt words are sent to at least one first language model respectively, and multiple multi-role evaluation sub-results of the reply text returned by the at least one first language model are received.

[0175] In some possible implementations, the second determining module 303 is specifically configured to:

[0176] Performing response quality evaluation on the multiple response texts from multiple different dimensions, respectively, and determining a multi-dimensional evaluation sub-result corresponding to each dimension;

[0177] The multiple reply texts are sorted according to the multi-dimensional evaluation sub-results corresponding to each dimension to determine a multi-dimensional sorting result of the multiple reply texts.

[0178] In some possible implementations, the second determining module 303 is specifically configured to:

[0179] For each of the multiple different dimensions, perform the following steps:

[0180] Generate a second prompt word; wherein the second prompt word includes: the question text, the multiple reply texts, third information for indicating the dimension, and fourth information for indicating that the multiple reply texts are evaluated for reply quality based on the dimension in combination with the question text;

[0181] The second prompt word is sent to a second language model, and a multi-dimensional evaluation sub-result corresponding to the dimension returned by the second language model is received.

[0182] In some possible implementations, the second determining module 303 is specifically configured to:

[0183] Generate a third prompt word; wherein the third prompt word includes: the question text, the multiple reply texts, the multi-dimensional evaluation sub-results corresponding to each dimension, and fifth information for indicating that the multiple reply texts are sorted in combination with the multi-dimensional evaluation sub-results corresponding to each dimension;

[0184] The third prompt word is sent to a third language model, and a multi-dimensional sorting result of the plurality of reply texts returned by the third language model is received.

[0185] In some possible implementations, the multiple different dimensions include at least two of the following:

[0186] The dimension for judging whether the reply text meets the question and answer requirements;

[0187] Determine whether there are any wrong dimensions in the response text;

[0188] The dimension to judge whether the reply text is replied from different angles;

[0189] A dimension to determine whether there is duplicate content in the reply text;

[0190] The dimension for judging whether the text structure of the reply text is reasonable.

[0191] In some possible implementations, the order of the multiple reply texts in the second prompt word is randomly arranged.

[0192] In some possible implementations, the order of the multiple reply texts in the third prompt word is randomly arranged, and the order of the multi-dimensional evaluation sub-results corresponding to each dimension in the third prompt word is randomly arranged.

[0193] In some possible implementations, the second determining module 303 is further configured to:

[0194] determining at least one response point associated with the question text;

[0195] The second prompt word also includes: at least one reply key point associated with the question text;

[0196] The third information in the second prompt word used to indicate the dimension includes: third information used to indicate the dimension related to at least one reply point associated with the question text.

[0197] In some possible implementations, the second determining module 303 is specifically configured to:

[0198] Generate a fourth prompt word; wherein the fourth prompt word includes: the question text and sixth information for indicating the key points of the reply to generate the question text;

[0199] The fourth prompt word is sent to a fourth language model, and at least one reply key point associated with the question text returned by the fourth language model is received.

[0200] In some possible implementations, the third determining module 304 is specifically configured to:

[0201] Determining whether the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results;

[0202] In response to the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results being consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results, determining the multiple multi-role evaluation results and the multi-dimensional sorting results as quality evaluation results of the multiple reply texts;

[0203] In response to the inconsistency between the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results and the reply quality of the multiple reply texts represented by the multi-dimensional sorting results, modify at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results, and determine the modified multiple multi-role evaluation results and the multi-dimensional sorting results as the quality evaluation results of the multiple reply texts.

[0204] In some possible implementations, the multi-role evaluation result includes an evaluation grade of a reply text corresponding to the multi-role evaluation result, the multi-dimensional sorting result is determined based on a multi-dimensional evaluation sub-result corresponding to each dimension, and the multi-dimensional evaluation sub-result corresponding to each dimension includes the evaluation grades of the multiple reply texts under each dimension; the third determination module 304 is specifically used to:

[0205] According to the set verification rules, verify the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results and the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results;

[0206] In response to the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results and the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results satisfying the verification rule, determining that the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results;

[0207] Otherwise, determining that the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are inconsistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results;

[0208] The verification rule includes at least one of the following:

[0209] There is no contradiction between the evaluation levels of the multiple reply texts in the multiple multi-role evaluation results and the order of the multiple reply texts in the multi-dimensional sorting result;

[0210] There is no contradiction between the order of the multiple reply texts in the multi-dimensional sorting result and the evaluation levels of the multiple reply texts in the multi-dimensional evaluation sub-results corresponding to each dimension;

[0211] There is no contradiction between the order of the multiple reply texts in the multi-dimensional sorting result and the order of the multiple reply texts in the multi-dimensional sorting result of the multiple reply texts determined again.

[0212] In some possible implementations, the third determining module 304 is specifically configured to:

[0213] In response to the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results being inconsistent with the reply quality of the multiple reply texts represented by the multi-dimensional sorting result, a fifth prompt word is generated; wherein the fifth prompt word includes: the question text, the multiple reply texts, the multiple multi-role evaluation results, the multi-dimensional sorting result, and seventh information for indicating modification of at least one of the multiple multi-role evaluation results and the multi-dimensional sorting result;

[0214] The fifth prompt word is sent to a fifth language model, and the modified multiple multi-role evaluation results and the multi-dimensional sorting results returned by the fifth language model are received.

[0215] The model response quality evaluation device 30 according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of each module / unit of the model response quality evaluation device 30 are respectively to achieve Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0216] The present application also provides an electronic device. The electronic device is specifically used to implement Figure 3 The function of the device 30 for evaluating the quality of model recovery in the illustrated embodiment.

[0217] Figure 4 A schematic diagram of the structure of an electronic device 400 is provided. Figure 4As shown, the electronic device 400 includes a bus 401, a processor 402, a communication interface 403 and a memory 404. The processor 402, the memory 404 and the communication interface 403 communicate with each other via the bus 401.

[0218] The bus 401 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0219] The processor 402 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0220] The communication interface 403 is used for communicating with the outside. For example, the communication interface 403 can be used for communicating with a terminal.

[0221] The memory 404 may include a volatile memory, such as a random access memory (RAM). The memory 404 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0222] The memory 404 stores executable codes, and the processor 402 executes the executable codes to perform the aforementioned model response quality evaluation method.

[0223] Specifically, in implementing Figure 3 In the case of the embodiment shown, and Figure 3 When each module or unit of the model response quality evaluation device 30 described in the embodiment is implemented by software, the execution Figure 3The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 404. The processor 402 executes the program code corresponding to each unit stored in the memory 404 to perform the aforementioned model response quality evaluation method.

[0224] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned evaluation method for the model response quality of the evaluation device 30 applied to the model response quality.

[0225] The embodiment of the present application further provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0226] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0227] When the computer program product is executed by a computer, the computer executes any of the aforementioned methods for evaluating model response quality. The computer program product may be a software installation package, and when any of the aforementioned methods for evaluating model response quality needs to be used, the computer program product may be downloaded and executed on a computer.

[0228] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0229] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the unit / module does not, in some cases, constitute a limitation on the unit itself.

[0230] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0231] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0232] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0233] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0234] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0235] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0236] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating the quality of model responses, characterized in that: The method comprises: Obtaining a question text and a plurality of reply texts obtained by respectively replying to the question text using a plurality of target models; Performing response quality evaluation on the multiple response texts respectively through multiple different role models, and determining multiple multi-role evaluation results corresponding to the multiple response texts respectively; Sorting the reply qualities of the multiple reply texts from multiple different dimensions to determine a multi-dimensional sorting result of the multiple reply texts; The quality evaluation results of the multiple reply texts are determined according to the multiple multi-role evaluation results and the multi-dimensional sorting results.

2. The method according to claim 1, characterized in that: The step of respectively performing reply quality evaluation on the plurality of reply texts by using a plurality of different role models to determine a plurality of multi-role evaluation results respectively corresponding to the plurality of reply texts includes: Performing a reply quality assessment on each reply text through a plurality of different role models, and determining a plurality of multi-role assessment sub-results of each reply text; wherein one multi-role assessment sub-result corresponds to one role; The multi-role evaluation result of each reply text is determined according to the multiple multi-role evaluation sub-results of each reply text.

3. The method according to claim 2, characterized in that The step of evaluating the quality of each reply text by using a plurality of different role models to determine a plurality of multi-role evaluation sub-results of each reply text includes: For each reply text in the plurality of reply texts, perform the following steps: Generate multiple first prompt words; wherein one of the first prompt words corresponds to one role, and one of the first prompt words includes: the question text, the reply text, first information for indicating the role corresponding to the first prompt word, and second information for indicating that the reply text is evaluated for reply quality in combination with the question text; The multiple first prompt words are sent to at least one first language model respectively, and multiple multi-role evaluation sub-results of the reply text returned by the at least one first language model are received.

4. The method according to claim 1, characterized in that: The step of sorting the reply qualities of the multiple reply texts from multiple different dimensions to determine the multi-dimensional sorting results of the multiple reply texts includes: Performing response quality evaluation on the multiple response texts from multiple different dimensions, respectively, and determining a multi-dimensional evaluation sub-result corresponding to each dimension; The multiple reply texts are sorted according to the multi-dimensional evaluation sub-results corresponding to each dimension to determine a multi-dimensional sorting result of the multiple reply texts.

5. The method according to claim 4, characterized in that The step of respectively evaluating the response quality of the multiple response texts from multiple different dimensions and determining a multi-dimensional evaluation sub-result corresponding to each dimension includes: For each of the multiple different dimensions, perform the following steps: Generate a second prompt word; wherein the second prompt word includes: the question text, the multiple reply texts, third information for indicating the dimension, and fourth information for indicating that the multiple reply texts are evaluated for reply quality based on the dimension in combination with the question text; The second prompt word is sent to a second language model, and a multi-dimensional evaluation sub-result corresponding to the dimension returned by the second language model is received.

6. The method according to claim 4, characterized in that The step of sorting the multiple reply texts according to the multi-dimensional evaluation sub-results corresponding to each dimension to determine the multi-dimensional sorting results of the multiple reply texts includes: Generate a third prompt word; wherein the third prompt word includes: the question text, the multiple reply texts, the multi-dimensional evaluation sub-results corresponding to each dimension, and fifth information for indicating that the multiple reply texts are sorted in combination with the multi-dimensional evaluation sub-results corresponding to each dimension; The third prompt word is sent to a third language model, and a multi-dimensional sorting result of the plurality of reply texts returned by the third language model is received.

7. The method according to claim 1, characterized in that The multiple different dimensions include at least two of the following: The dimension for judging whether the reply text meets the question and answer requirements; Determine whether there are any wrong dimensions in the response text; The dimension to judge whether the reply text is replied from different angles; A dimension to determine whether there is duplicate content in the reply text; The dimension for judging whether the text structure of the reply text is reasonable.

8. The method according to claim 5, characterized in that The order of the multiple reply texts in the second prompt word is randomly arranged.

9. The method according to claim 6, characterized in that The order of the multiple reply texts in the third prompt word is randomly arranged, and the order of the multi-dimensional evaluation sub-results corresponding to each dimension in the third prompt word is randomly arranged.

10. The method according to claim 5, characterized in that The method further comprises: determining at least one response point associated with the question text; The second prompt word also includes: at least one reply key point associated with the question text; The third information in the second prompt word used to indicate the dimension includes: third information used to indicate the dimension related to at least one reply point associated with the question text.

11. The method according to claim 10, characterized in that The determining at least one reply point associated with the question text comprises: Generate a fourth prompt word; wherein the fourth prompt word includes: the question text and sixth information for indicating the key points of the reply to the question text; The fourth prompt word is sent to a fourth language model, and at least one reply key point associated with the question text returned by the fourth language model is received.

12. The method according to any one of claims 1 to 11, characterized in that: Determining the quality evaluation results of the multiple reply texts according to the multiple multi-role evaluation results and the multi-dimensional sorting results includes: Determining whether the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results; In response to the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results being consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results, determining the multiple multi-role evaluation results and the multi-dimensional sorting results as quality evaluation results of the multiple reply texts; In response to the inconsistency between the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results and the reply quality of the multiple reply texts represented by the multi-dimensional sorting results, modify at least one of the multiple multi-role evaluation results and the multi-dimensional sorting results, and determine the modified multiple multi-role evaluation results and the multi-dimensional sorting results as the quality evaluation results of the multiple reply texts.

13. The method according to claim 12, characterized in that The multi-role evaluation result includes the evaluation level of the reply text corresponding to the multi-role evaluation result, and the multi-dimensional sorting result is determined based on the multi-dimensional evaluation sub-result corresponding to each dimension, and the multi-dimensional evaluation sub-result corresponding to each dimension includes the evaluation level of the multiple reply texts under each dimension; The determining whether the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results includes: According to the set verification rules, verify the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results and the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results; In response to the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results and the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results satisfying the verification rule, determining that the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are consistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results; Otherwise, determining that the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results are inconsistent with the reply qualities of the multiple reply texts represented by the multi-dimensional sorting results; The verification rule includes at least one of the following: There is no contradiction between the evaluation levels of the multiple reply texts in the multiple multi-role evaluation results and the order of the multiple reply texts in the multi-dimensional sorting result; There is no contradiction between the order of the multiple reply texts in the multi-dimensional sorting result and the evaluation levels of the multiple reply texts in the multi-dimensional evaluation sub-results corresponding to each dimension; There is no contradiction between the order of the multiple reply texts in the multi-dimensional sorting result and the order of the multiple reply texts in the multi-dimensional sorting result of the multiple reply texts determined again.

14. The method according to claim 12, characterized in that In response to the reply qualities of the multiple reply texts represented by the multiple multi-role evaluation results being inconsistent with the reply qualities of the multiple reply texts represented by the multi-dimensional ranking results, modifying at least one of the multiple multi-role evaluation results and the multi-dimensional ranking results, comprising: In response to the reply quality of the multiple reply texts represented by the multiple multi-role evaluation results being inconsistent with the reply quality of the multiple reply texts represented by the multi-dimensional sorting result, a fifth prompt word is generated; wherein the fifth prompt word includes: the question text, the multiple reply texts, the multiple multi-role evaluation results, the multi-dimensional sorting result, and seventh information for indicating modification of at least one of the multiple multi-role evaluation results and the multi-dimensional sorting result; The fifth prompt word is sent to a fifth language model, and the modified multiple multi-role evaluation results and the multi-dimensional sorting results returned by the fifth language model are received.

15. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The method comprises instructions for instructing an electronic device to execute the method according to any one of claims 1 to 14.

Images