Quality evaluation method, device and equipment of question and answer model and storage medium

Abstract

The invention provides a question and answer model quality evaluation method, device and equipment and a storage medium, and the method comprises the steps: obtaining a plurality of model reply texts corresponding to a target question and an initial model quality score, the plurality of model reply texts being a plurality of reply texts of the target question output by the question and answer model, the initial model quality score is a quality score obtained by performing model quality evaluation on the target question replied by the question and answer model based on the plurality of model reply texts; according to the overlapping ratio of the multiple pieces of model reply text, determining a reverse overlapping ratio which is used for representing the difference degree of the multiple pieces of model reply text; and according to the initial model quality score and the reverse overlapping rate, determining a final model quality score of replying the target question by the question and answer model. According to the technical scheme, quality evaluation of questions replied by the question and answer model can be realized, and the accuracy of quality evaluation is improved.

Description

Technical Field

[0001] The present application relates to the field of question-answering model quality assessment, and in particular to a question-answering model quality assessment method, apparatus, device and storage medium. Background Art

[0002] Intelligent question-answering dialogue system is one of the research hotspots in the field of artificial intelligence in recent years. It can help users solve problems, provide relevant information, and realize the convenience of human-computer interaction.

[0003] Dialogue quality assessment is a key issue in the research of intelligent question-answering dialogue systems, which involves the output effect of the system and the user experience. How to conduct quality assessment on intelligent question-answering dialogue systems has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for evaluating the quality of a question-answering model to perform quality evaluation on an intelligent question-answering dialogue system.

[0005] In a first aspect, a quality assessment method for a question-answering model is provided, comprising:

[0006] Obtain multiple model response texts and initial model quality scores corresponding to the target question, wherein the multiple model response texts are multiple response texts of the target question output by the question-and-answer model, and the initial model quality score is a quality score obtained by performing a model quality assessment on the question-and-answer model's response to the target question based on the multiple model response texts;

[0007] Determining a reverse overlap rate according to the overlap rates between the multiple model reply texts, wherein the reverse overlap rate is used to indicate the degree of difference between the multiple model reply texts;

[0008] A final model quality score of the question-answering model in replying to the target question is determined based on the initial model quality score and the reverse overlap rate.

[0009] In this technical solution, by obtaining multiple model response texts and initial model quality scores corresponding to the target question, and then determining the reverse overlap rate based on the overlap rate between the multiple model response texts, the reverse overlap rate is used to indicate the degree of difference between the multiple model response texts, and finally determining the final model quality score of the question-answering model's response to the target question based on the initial quality score and the reverse overlap rate corresponding to the target question, thereby achieving quality evaluation of the question-answering model's response to the question; by determining the final model quality score of the question-answering model's response to the target question based on the reverse overlap rate indicating the degree of difference between multiple model response texts output by the question-answering model for the same question, and the initial model quality score obtained by performing model quality evaluation on the question-answering model's response to the target question, the diversity of the model generation results can be measured, making the quality evaluation of the question-answering model's response to the question more reasonable and effective, thereby improving the accuracy of the quality evaluation.

[0010] In combination with the first aspect, in a possible implementation method, determining the reverse overlap rate based on the overlap rates between the multiple model reply texts includes: calculating the overlap rates between the multiple model reply texts to obtain at least one overlap rate; determining the reverse overlap rate based on the average of the at least one overlap rate, wherein the sum of the reverse overlap rate and the average of the at least one overlap rate is 1.

[0011] In combination with the first aspect, in a possible implementation method, determining the final model quality score of the question and answer model in replying to the target question based on the initial model quality score and the reverse overlap rate includes: calculating a harmonic index of the initial model quality score and the reverse overlap rate, and determining the harmonic index as the final model quality score of the question and answer model in replying to the target question.

[0012] In combination with the first aspect, in a possible implementation, the obtaining of the initial model quality score corresponding to the target question includes: obtaining multiple reference reply texts, wherein the similarity between the multiple reference reply texts is less than a preset threshold; comparing the target model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the target model reply text, wherein the target model reply text is any one of the multiple model reply texts; and determining the initial model quality score of the question-and-answer model's reply to the target question based on the reply quality scores corresponding to each of the multiple model reply texts. Since the answer to a question is not unique, using multiple reference reply texts with less similarity to compare with the model reply text output by the question-and-answer model to obtain the initial quality score of the question-and-answer model's reply to the question can reduce the evaluation bias caused by the evaluation of the model reply text with a single reference answer.

[0013] In combination with the first aspect, in a possible implementation, the target model reply text is compared with the multiple reference reply texts to obtain a reply quality score of the target model reply text, including: according to a recall-oriented text evaluation method, the target model reply text is compared with a target reference reply text to obtain a text quality score corresponding to the target model reply text, the target reference reply text is any one of the multiple reference reply texts, and the text quality score is a quality score obtained by performing a text quality evaluation on the target model reply text with the target reference reply text as a reference text; according to multiple text quality scores corresponding to the target model reply text, the reply quality score corresponding to the target model reply text is determined, and the multiple text quality scores are quality scores obtained by performing a quality evaluation on the target model reply text with the multiple reference reply texts as reference texts. Evaluating the model reply text output by the question-answering model according to the recall-oriented evaluation method can make the reply quality score explainable.

[0014] In combination with the first aspect, in a possible implementation, the target model reply text is compared with the target reference reply text according to the recall-oriented text evaluation method to obtain a text quality score corresponding to the target model reply text, including: calculating the Rouge-L value between the target model reply text and the target reference reply text to obtain a text quality score corresponding to the target model reply text. Using the Rouge-L value between the model reply text and the reference reply text as a text quality score for the model reply text can make the text quality score more reasonable.

[0015] In combination with the first aspect, in a possible implementation method, determining the reply quality score corresponding to the target model reply text according to multiple text quality scores corresponding to the target model reply text includes: calculating feature values ​​of the multiple text quality scores, the feature values ​​including at least one of a maximum value, a minimum value or an average value; determining the reply quality score corresponding to the target model reply text according to the feature values ​​corresponding to the multiple quality scores.

[0016] In combination with the first aspect, in a possible implementation method, determining an initial model quality score for the question and answer model's reply to the target question based on the reply quality scores corresponding to each of the multiple model reply texts includes: determining the average of the reply quality scores corresponding to each of the multiple model reply texts as the initial model quality score for the question and answer model's reply to the target question.

[0017] In a second aspect, a quality assessment device for a question-answering model is provided, comprising:

[0018] A first scoring module is used to obtain multiple model response texts corresponding to the target question and an initial model quality score, wherein the multiple model response texts are multiple response texts of the target question output by the question-answering model, and the initial model quality score is a quality score obtained by performing a model quality assessment on the response of the question-answering model to the target question based on the multiple model response texts;

[0019] A difference evaluation module, used to determine a reverse overlap rate according to the overlap rates between the multiple model reply texts, wherein the reverse overlap rate is used to indicate the degree of difference between the multiple model reply texts;

[0020] The second scoring module is used to determine the final model quality score of the question-answering model in replying to the target question based on the initial model quality score and the reverse overlap rate.

[0021] In a third aspect, a computer device is provided, comprising a memory and one or more processors, wherein the memory is connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the computer device implements the quality assessment method of the question-answering model of the first aspect.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the quality assessment method of the question-answering model of the first aspect.

[0023] The present application can achieve the following technical effects: it realizes the quality evaluation of the questions answered by the question-answering model; it determines the final model quality score of the question-answering model's reply to the target question by using the inverse overlap rate representing the difference between multiple model reply texts output by the question-answering model for the same question and the initial model quality score of the model quality evaluation of the question-answering model's reply to the target question, which can measure the diversity of the model generation results, making the quality evaluation of the question-answering model's reply to the target question more reasonable and effective, and improving the accuracy of the quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a quality assessment method for a question-answering model provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a flow chart of a method for evaluating the quality of an initial model of a question-answering model provided in an embodiment of the present application;

[0026] Figure 3 It is a structural schematic diagram of a quality assessment device for a question-answering model provided in an embodiment of the present application;

[0027] Figure 4 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0029] The technical solution of this application is applicable to model quality assessment scenarios, especially to question-answering model quality assessment scenarios. Question-answering model quality assessment refers to quality assessment of trained question-answering models to determine the performance of the models. The technical solution of this application can be applied to computer devices, including but not limited to mobile phones, laptops, desktop computers, and the like.

[0030] The technical solution of this application is described in detail below.

[0031] See also Figure 1 , Figure 1 A flow chart of a quality assessment method for a question-answering model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0032] S101, obtaining multiple model response texts and initial model quality scores corresponding to the target question.

[0033] Here, the target question is a question to be answered, and the target question can be any question. For example, the target question can be "My big screen is black, what's going on?".

[0034] The model response text corresponding to the target question is the response text of the target question output by the question-answering model when the target question is input into the question-answering model. The target question can be input into the question-answering model multiple times in succession, and the response texts output by the question-answering model multiple times in succession can be obtained as the multiple model response texts corresponding to the target question. For example, the target question "My big screen is black, what's going on?" can be input into the question-answering model three times in succession. The question-answering model outputs response 1 for the first time: "Check the power connection: Make sure the power cord is firmly connected and the plug is not loose or broken", and the question-answering model outputs response 2 for the second time: "Please restart the big screen: Press the power button on the big screen and hold it for a few seconds, then release the button", and the third time outputs response 3: "Check the signal source: Make sure the correct signal source is selected correctly", then the response 1, response 2, and response 3 output by the answering model three times in succession are the multiple model response texts corresponding to the target question.

[0035] The initial model quality score corresponding to the target question is the quality score obtained by evaluating the model quality of the question-answering model's responses to the target question based on multiple model response texts corresponding to the target question. For the specific implementation method of obtaining the initial model quality score corresponding to the target question, please refer to the subsequent Figure 2 The corresponding technical solutions are not described in detail here.

[0036] S102, determining a reverse overlap rate according to the overlap rates between the multiple model reply texts corresponding to the target question.

[0037] Here, the reverse overlap rate is used to indicate the degree of difference between the multiple model response texts corresponding to the target question. The larger the reverse overlap rate, the higher the degree of difference between the multiple model response texts corresponding to the target question, and the better the diversity of the multiple model response texts corresponding to the target question; the smaller the reverse overlap rate, the lower the degree of difference between the multiple model response texts corresponding to the target question, and the worse the diversity of the multiple model response texts corresponding to the target question.

[0038] In a feasible implementation, the overlap rates between multiple model reply texts corresponding to the target question can be calculated to obtain at least one overlap rate; the reverse overlap rate is determined according to the mean of the at least one overlap rate, and the sum of the reverse overlap rate and the mean of the at least one overlap rate is 1. The calculation formula of the reverse overlap rate can be: P is the reverse overlap rate, p k is the kth overlap rate, K is the number of overlap rates, and the number of overlap rates is m is the number of model response texts corresponding to the target question.

[0039] For example, the multiple model reply texts corresponding to the target question are reply 1, reply 2 and reply 3 introduced in the aforementioned step S102. The overlap rate between reply 1 and reply 2 can be calculated. Assuming that the overlap rate between reply 1 and reply 2 is 0.16, the overlap rate between reply 2 and reply 3 is calculated. Assuming that the overlap rate between reply 2 and reply 3 is 0.226, the overlap rate between reply 1 and reply 3 is calculated. Assuming that the overlap rate between reply 1 and reply 3 is 0.17, the inverse overlap rate P = 1-(0.16+0.226+0.17) / 3 = 0.71.

[0040] Optionally, after calculating the overlap rates between multiple model reply texts corresponding to the target question and obtaining at least one overlap rate, the difference between 1 and each overlap rate in the at least one overlap rate may be calculated to obtain at least one difference, and the mean of the at least one difference may be determined as the reverse overlap rate, that is, the calculation formula for the reverse overlap rate may also be: This application is not limiting.

[0041] S103, determining the final model quality score of the question-answering model in replying to the target question based on the initial model quality score and the reverse overlap rate corresponding to the target question.

[0042] In a feasible implementation, the reconciliation index of the initial model quality score and the reverse overlap rate corresponding to the target question can be calculated, and the reconciliation index of the initial model quality score and the reverse overlap rate corresponding to the target question can be determined as the final model quality score of the question-answering model answering the target question. The reconciliation index calculation formula of the initial model quality score and the reverse overlap rate corresponding to the target question is: R is the reconciliation index, S is the initial model quality score corresponding to the target problem, and b is the reconciliation coefficient of the reverse overlap rate, and b can be preset to 0.3, for example.

[0043] Optionally, the product of the reverse overlap rate and the initial model quality score corresponding to the target question can also be directly determined as the final model quality score of the question-answering model answering the target question. This application does not impose any restrictions.

[0044] exist Figure 1 In the corresponding technical scheme, by obtaining multiple model response texts and initial model quality scores corresponding to the target question, and then determining the reverse overlap rate based on the overlap rate between the multiple model response texts, the reverse overlap rate is used to indicate the degree of difference between the multiple model response texts, and finally determining the final model quality score of the question-answering model's response to the target question based on the initial quality score and the reverse overlap rate corresponding to the target question, thereby achieving quality evaluation of the question-answering model's response to the question; by determining the final model quality score of the question-answering model's response to the target question based on the reverse overlap rate indicating the degree of difference between multiple model response texts output by the question-answering model for the same question, and the initial model quality score obtained by performing model quality evaluation on the question-answering model's response to the target question, the diversity of the model generation results can be measured, making the quality evaluation of the question-answering model's response to the question more reasonable and effective, thereby improving the accuracy of the quality evaluation.

[0045] See also Figure 2 , Figure 2 A flowchart of an initial model quality assessment method for a question-answering model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0046] S201, obtaining multiple model response texts and multiple reference response texts corresponding to the target question.

[0047] Here, regarding the specific implementation method of obtaining multiple model response texts corresponding to the target question, please refer to the relevant description of the aforementioned step S101, which will not be repeated here.

[0048] The multiple reference response texts corresponding to the target question can be understood as multiple standard answers pre-set for the target question, and the multiple reference response texts corresponding to the target question are used as reference texts for the model response text. For example, for the target question "My big screen is black, what's going on?", multiple standard answers can be pre-set, namely: Standard answer A: Hello, please check whether the device is powered on. If it is powered on and the screen is still black, please transfer to manual. Standard answer B: Hello, the black screen of the big screen is likely to be a backlight power problem. Please confirm whether the backlight power indicator is on. Standard answer C: Hello, please confirm whether the switch indicator turns from red to blue and then to red. That is, standard answer A, standard answer B, and standard answer C are the multiple reference response texts corresponding to the target question "My big screen is black, what's going on?"

[0049] The multiple reference reply texts corresponding to the target question can be pre-set by the user and saved in the computer device. The reference reply texts set by the user for the target question can be obtained from the computer device to obtain the multiple reference reply texts corresponding to the target question.

[0050] The similarity between the multiple reference response texts corresponding to the target question is less than the preset threshold to ensure the differences between the multiple reference response texts. For example, if the preset threshold is 0.5, and the multiple reference response texts corresponding to the target question are the aforementioned standard answer A, standard answer B, and standard answer C, then the similarity between standard answer A and standard answer B should be less than 0.5, the similarity between standard answer A and standard answer C should be less than 0.5, and the similarity between standard answer B and standard answer C should be less than 0.5.

[0051] In a feasible implementation, the similarity between reference reply texts can be evaluated based on sentence vectors. The similarity between multiple reference reply texts is less than a preset threshold, which may mean that the similarity between sentence vectors corresponding to the multiple reference reply texts is less than the preset threshold. After obtaining multiple reference reply texts set by the user, the multiple reference reply texts can be converted into sentence vectors respectively to obtain multiple sentence vectors, and then the similarities between the multiple sentence vectors are calculated. If each calculated similarity is less than the preset threshold, it is determined that the similarity between the multiple reference reply texts is less than the preset threshold. Among them, the Euclidean distance, cosine distance, etc. of the two sentence vectors can be calculated to calculate the similarity between the two sentence vectors.

[0052] S202, comparing the target model response text corresponding to the target question with multiple reference response texts corresponding to the target question to obtain a response quality score corresponding to the target model response text.

[0053] Here, the target model reply text is any one of the multiple model reply texts corresponding to the target question. For example, if the multiple model reply texts corresponding to the target question are reply 1, reply 2, and reply 3 in the aforementioned step S101, the target model reply text can be reply 1, reply 2, or reply 3.

[0054] In some possible cases, the model response text corresponding to the target question can be evaluated based on the recall-oriented text evaluation method to obtain the response quality score corresponding to the model response text. The response quality score corresponding to the target model response text can be obtained through the following steps A1-A2:

[0055] A1. According to the recall-oriented text evaluation method, the target model response text corresponding to the target question is compared with the target reference response text to obtain the text quality score corresponding to the target model response text.

[0056] The target reference reply text is any one of the multiple reference reply texts corresponding to the target question. For example, if the multiple reference reply texts corresponding to the target question are the standard answer A, standard answer B, and standard answer C introduced in the aforementioned step S201, the target reference reply text can be the standard answer A, standard answer B, or standard answer C.

[0057] The text quality score corresponding to the target model reply text is a quality score obtained by using the target reference reply text as the reference text and adopting a recall-oriented text evaluation method to perform text evaluation on the target model reply text.

[0058] In a feasible implementation, the Rouge-L value between the target model reply text and the target reference reply text can be calculated to obtain the text quality score corresponding to the target model reply text. The Rouge-L value between the target model reply text and the target reference reply text can be calculated according to the following formulas (1)-(3):

[0059]

[0060]

[0061]

[0062] Where text1 is the target model reply text, text2 is the target reference reply text, LCS(text1, text2) is the length of the largest common substring between the target model reply text and the target reference reply text, L1 is the length of the target reference reply text, L2 is the length of the target model reply text, β is a hyperparameter, R lcs is the recall rate, P lcs is the accuracy, Flcs It is the Rouge-L value.

[0063] By using the Rouge-L value between the model reply text and the reference reply text as a text quality score of the model reply text, the text quality score can be made more reasonable.

[0064] Optionally, other recall-oriented evaluation indicators between the target model reply text and the target reference reply text may also be calculated to obtain a text quality score corresponding to the target model reply text. For example, the Rouge-1 value, Rouge-2 value, or Rouge-N value between the target model reply text and the target reference reply text may also be calculated to obtain a text quality score corresponding to the target model reply text. This application is not limited thereto.

[0065] By taking each of the multiple reference reply texts as the target reference reply text, and taking the target reference reply text as the reference text, and performing text evaluation on the target model reply text to obtain the text quality score corresponding to the model reply text, a plurality of text quality scores corresponding to the model reply text can be obtained. For example, if the multiple reference reply texts corresponding to the target question are the standard answer A, standard answer B, and standard answer C introduced in the aforementioned step S201, then the standard answer A, standard answer B, and standard answer C can be taken as the target reference reply texts, and the text quality score corresponding to the target model reply text when the standard answer A is taken as the target reference reply text, the text quality score corresponding to the target model reply text when the standard answer B is taken as the target reference reply text, and the text quality score corresponding to the target model reply text when the standard answer C is taken as the target reference reply text can be calculated, thereby obtaining three text quality scores corresponding to the target model reply text.

[0066] The method for calculating the quality score of each text is the same. For example, the Rouge-L value between each reference reply text in multiple reference reply texts and the target model reply text can be calculated to obtain multiple Rouge-L values ​​as multiple text quality scores corresponding to the target model reply text. Alternatively, the Rouge-1 value between each reference reply text in multiple reference reply texts and the target model reply text can be calculated to obtain multiple Rouge-1 values ​​as multiple text quality scores corresponding to the target model reply text.

[0067] A2. Determine a response quality score corresponding to the response text of the target model according to multiple text quality scores corresponding to the response text of the target model.

[0068] Here, the characteristic values ​​of multiple text quality scores corresponding to the target model reply text can be calculated, and the characteristic value can be at least one of the maximum value, the minimum value or the average value; according to the characteristic values ​​of multiple text quality scores corresponding to the target model reply text, the reply quality score corresponding to the model reply text is determined.

[0069] For example, the multiple text quality scores corresponding to the target model reply text are the multiple Rouge-L values ​​corresponding to the target model reply text. The maximum value among the multiple Rouge-L values ​​can be determined as the reply quality score corresponding to the target model reply text, that is, the reply quality score corresponding to the target model reply text is s=max(Rouge-Li), i=1,2,…n, n is the number of Rouge-L values, that is, the number of reference reply texts corresponding to the target question. Alternatively, the average value among the multiple Rouge-L values ​​can be determined as the reply quality score corresponding to the target model reply text, that is, the reply quality score corresponding to the target model reply text is Alternatively, at least two eigenvalues ​​of multiple Rouge-L values ​​may be weighted and summed to obtain a response quality score corresponding to the response text of the target model. This application does not impose any restrictions.

[0070] Determining the maximum value of multiple text quality scores corresponding to the model reply text as the reply quality score corresponding to the model reply text can reduce the evaluation bias caused by evaluating the model reply text with a single reference answer; evaluating the model reply text output by the question-answering model according to the recall-oriented evaluation method can make the reply quality score explainable.

[0071] In some other possible situations, the model response text corresponding to the target question can also be evaluated based on the bilingual evaluation understudy (BLEU), BERTScore and other text evaluation methods to obtain the response quality score corresponding to the model response text. This application does not make any restrictions.

[0072] By evaluating multiple model response texts corresponding to the target question according to the same text evaluation method, the response quality scores corresponding to each of the multiple model response texts can be obtained.

[0073] For example, if the multiple model reply texts corresponding to the target question are reply 1, reply 2, and reply 3 in the aforementioned step S101, then the text evaluation method introduced above can be referred to to obtain the reply quality score corresponding to reply 1, the reply quality score corresponding to reply 2, and the reply quality score corresponding to reply 3, respectively.

[0074] S203, determining an initial model quality score for the question-answering model's response to the target question based on the response quality scores corresponding to the multiple model response texts.

[0075] In a feasible implementation, the average of the response quality scores corresponding to the multiple model response texts can be determined as the model quality score of the question-answering model answering the target question, that is, the model quality score of the question-answering model answering the target question is sj is the response quality score corresponding to the j-th model response text corresponding to the target question, and m is the number of response quality scores, that is, the number of model response texts corresponding to the target question.

[0076] Optionally, the response quality scores corresponding to multiple model response texts can be weighted and averaged to obtain the model quality score of the question-answering model answering the target question, that is, the model quality score of the question-answering model answering the target question is aj is the weight coefficient corresponding to the j-th model reply text corresponding to the target question. Among them, the weight coefficient corresponding to each model reply text can be determined based on the frequency of the question-answering model outputting the model reply text. The multiple model reply texts corresponding to the target question are reply 1, reply 2, and reply 3 in the aforementioned step S201. Assuming that the target question is input into the question-answering model 30 times in succession, the question-answering model outputs reply 1 15 times, replies 2 6 times, and replies 3 9 times. Then, the question-answering model outputs reply 1 at a frequency of 0.5, reply 2 at a frequency of 0.2, and reply 3 at a frequency of 0.3. 0.5 can be determined as the weight coefficient corresponding to reply 1, 0.2 as the weight coefficient corresponding to reply 2, and 0.3 as the weight coefficient corresponding to reply 3.

[0077] This application does not specify how to determine the model quality score of the question-answering model's response to the target question based on the response quality scores corresponding to each of the multiple model response texts.

[0078] exist Figure 2The corresponding technical solution obtains multiple model response texts corresponding to the target question, and respectively determines the response quality scores corresponding to the multiple model response texts, and finally determines the model quality score of the question-and-answer model's response to the target question based on the response quality scores corresponding to the multiple model response texts, thereby realizing the quality evaluation of the question-and-answer model's response to the question; determining the quality score of the question-and-answer model's response to the question based on the response quality scores of the multiple model response texts output by the question-and-answer model for the same question can avoid the score fluctuation caused by the randomness of the response generated by the question-and-answer model, and ensure the rationality and reliability of the score; since the answer to the question is not unique, multiple reference response texts with small similarity are used to compare with the model response text output by the question-and-answer model to obtain the quality score of the question-and-answer model's response to the question, which can reduce the evaluation bias caused by the unique reference answer evaluation model response text, making the quality evaluation of the question-and-answer model's response to the question more reasonable and effective, and improving the accuracy of the quality evaluation.

[0079] In some possible cases, for the above Figure 1 and Figure 2 The question-answering model involved in the corresponding technical solution can also be based on the above Figure 1 The final model quality scores of the multiple questions answered by the question-answering model are determined respectively, and the multiple questions can be different; then, the overall quality score of the question-answering model is determined according to the final model quality scores of the multiple questions answered by the question-answering model. The overall quality score of the question-answering model can be the sum of the final model quality scores corresponding to the multiple questions, or the mean of the final model quality scores corresponding to the multiple questions. In this way, the overall quality evaluation of the question-answering model can be achieved.

[0080] The method of the present application is introduced above, and the device of the present application is introduced below.

[0081] See also Figure 3 , Figure 3 Schematic diagram of a quality assessment device for a question-answering model provided in an embodiment of the present application. Figure 3 As shown, the quality assessment device 30 of the question-answering model includes:

[0082] The first scoring module 301 is used to obtain multiple model reply texts corresponding to the target question and an initial model quality score, wherein the multiple model reply texts are multiple reply texts of the target question output by the question-answering model, and the initial model quality score is a quality score obtained by performing a model quality assessment on the question-answering model's reply to the target question based on the multiple model reply texts;

[0083] A difference evaluation module 302, used to determine a reverse overlap rate according to the overlap rates between the plurality of model reply texts, wherein the reverse overlap rate is used to indicate the degree of difference between the plurality of model reply texts;

[0084] The second scoring module 303 is used to determine the final model quality score of the question-answering model in replying to the target question based on the initial model quality score and the reverse overlap rate.

[0085] In one possible design, the difference evaluation module 302 is specifically used to: calculate the overlap rates between the multiple model reply texts to obtain at least one overlap rate; determine the reverse overlap rate based on the mean of the at least one overlap rate, and the sum of the reverse overlap rate and the mean of the at least one overlap rate is 1.

[0086] In one possible design, the second scoring module 303 is specifically used to calculate a reconciliation index between the initial model quality score and the reverse overlap rate, and determine the reconciliation index as the final model quality score of the question-answering model in replying to the target question.

[0087] In one possible design, the first scoring module 301 is specifically used to: obtain multiple reference reply texts, wherein the similarity between the multiple reference reply texts is less than a preset threshold; compare the target model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the target model reply text, wherein the target model reply text is any one of the multiple model reply texts; and determine the initial model quality score of the question-and-answer model in replying to the target question based on the reply quality scores corresponding to each of the multiple model reply texts.

[0088] In one possible design, the above-mentioned first scoring module 301 is specifically used to: compare the target model reply text with the target reference reply text according to a recall-oriented text evaluation method to obtain a text quality score corresponding to the target model reply text, the target reference reply text is any one of the multiple reference reply texts, and the text quality score is a quality score obtained by performing a text quality assessment on the target model reply text with the target reference reply text as a reference text; determine the reply quality score corresponding to the target model reply text according to multiple text quality scores corresponding to the target model reply text, and the multiple text quality scores are quality scores obtained by performing a quality assessment on the target model reply text with the multiple reference reply texts as reference texts respectively.

[0089] In a possible design, the first scoring module 301 is specifically used to calculate the Rouge-L value between the target model reply text and the target reference reply text to obtain a text quality score corresponding to the target model reply text.

[0090] In one possible design, the first scoring module 301 is specifically used to: calculate the characteristic values ​​of the multiple text quality scores, the characteristic values ​​including at least one of the maximum value, the minimum value or the average value; and determine the reply quality score corresponding to the target model reply text based on the characteristic values ​​corresponding to the multiple quality scores.

[0091] In a possible design, the first scoring module 301 is specifically used to determine the average of the response quality scores corresponding to the multiple model response texts as the initial model quality score of the question-answering model's response to the target question.

[0092] It should be noted that Figure 3 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0093] The above-mentioned device obtains multiple model response texts and initial model quality scores corresponding to the target question, and then determines the reverse overlap rate based on the overlap rate between the multiple model response texts. The reverse overlap rate is used to represent the degree of difference between the multiple model response texts. Finally, the final model quality score of the question-answering model's response to the target question is determined based on the initial quality score and the reverse overlap rate corresponding to the target question, thereby realizing the quality evaluation of the question-answering model's response to the question; by determining the final model quality score of the question-answering model's response to the target question based on the reverse overlap rate representing the degree of difference between the multiple model response texts output by the question-answering model for the same question, and the initial model quality score of the model quality evaluation of the question-answering model's response to the target question, the diversity of the model generation results can be measured, so that the quality evaluation of the question-answering model's response to the question is more reasonable and effective, and the accuracy of the quality evaluation is improved.

[0094] See also Figure 4 , Figure 4 4 is a schematic diagram of a computer device provided in an embodiment of the present application, wherein the computer device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example, via a bus.

[0095] The processor 401 is configured to support the computer device 40 to perform the corresponding functions in the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0096] The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

[0097] The processor 401 may call the program code to perform the following operations:

[0098] Obtain multiple model response texts and initial model quality scores corresponding to the target question, wherein the multiple model response texts are multiple response texts of the target question output by the question-and-answer model, and the initial model quality score is a quality score obtained by performing a model quality assessment on the question-and-answer model's response to the target question based on the multiple model response texts;

[0099] Determining a reverse overlap rate according to the overlap rates between the multiple model reply texts, wherein the reverse overlap rate is used to indicate the degree of difference between the multiple model reply texts;

[0100] A final model quality score of the question-answering model in replying to the target question is determined based on the initial model quality score and the reverse overlap rate.

[0101] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0102] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0103] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A quality assessment method for a question-answering model, characterized in that: include: Obtain multiple model response texts and initial model quality scores corresponding to the target question, wherein the multiple model response texts are multiple response texts of the target question output by the question-and-answer model, and the initial model quality score is a quality score obtained by performing a model quality assessment on the question-and-answer model's response to the target question based on the multiple model response texts; Determining a reverse overlap rate according to the overlap rates between the multiple model reply texts, wherein the reverse overlap rate is used to indicate the degree of difference between the multiple model reply texts; A final model quality score of the question-answering model in replying to the target question is determined based on the initial model quality score and the reverse overlap rate.

2. The method according to claim 1, characterized in that Determining the reverse overlap rate according to the overlap rates between the multiple model reply texts includes: Calculating overlapping rates between the multiple model reply texts to obtain at least one overlapping rate; The reverse overlap rate is determined according to the average of the at least one overlap rate, and the sum of the reverse overlap rate and the average of the at least one overlap rate is 1.

3. The method according to claim 1, characterized in that Determining the final model quality score of the question-answering model in replying to the target question according to the initial model quality score and the reverse overlap rate includes: A reconciliation index between the initial model quality score and the reverse overlap rate is calculated, and the reconciliation index is determined as the final model quality score of the question-answering model in replying to the target question.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the initial model quality score corresponding to the target problem includes: Acquire multiple reference reply texts, wherein the similarity between the multiple reference reply texts is less than a preset threshold; Comparing the target model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the target model reply text, wherein the target model reply text is any one of the multiple model reply texts; An initial model quality score of the question-answering model in replying the target question is determined based on the response quality scores corresponding to each of the multiple model response texts.

5. The method according to claim 4, characterized in that The step of comparing the target model reply text with the plurality of reference reply texts to obtain a reply quality score of the target model reply text includes: According to the recall-oriented text evaluation method, the target model reply text is compared with the target reference reply text to obtain a text quality score corresponding to the target model reply text, the target reference reply text is any one of the multiple reference reply texts, and the text quality score is a quality score obtained by performing a text quality evaluation on the target model reply text using the target reference reply text as a reference text; According to multiple text quality scores corresponding to the target model reply text, the reply quality score corresponding to the target model reply text is determined, and the multiple text quality scores are quality scores obtained by performing quality assessment on the target model reply text using the multiple reference reply texts as reference texts respectively.

6. The method according to claim 5, characterized in that The method for evaluating recall-oriented texts is used to compare the target model response text with the target reference response text to obtain a text quality score corresponding to the target model response text, including: The Rouge-L value between the target model reply text and the target reference reply text is calculated to obtain a text quality score corresponding to the target model reply text.

7. The method according to claim 5, characterized in that Determining the response quality score corresponding to the target model response text according to the multiple text quality scores corresponding to the target model response text includes: Calculating feature values ​​of the plurality of text quality scores, the feature values ​​comprising at least one of a maximum value, a minimum value, or an average value; A reply quality score corresponding to the target model reply text is determined according to the feature values ​​corresponding to the multiple quality scores.

8. The method according to claim 4, characterized in that Determining the initial model quality score of the question-answering model in replying the target question according to the response quality scores corresponding to the multiple model response texts includes: The average of the response quality scores corresponding to the multiple model response texts is determined as the initial model quality score of the question-answering model in responding to the target question.

9. A quality assessment device for a question-answering model, characterized in that: include: A first scoring module is used to obtain multiple model response texts corresponding to the target question and an initial model quality score, wherein the multiple model response texts are multiple response texts of the target question output by the question-answering model, and the initial model quality score is a quality score obtained by performing a model quality assessment on the response of the question-answering model to the target question based on the multiple model response texts; A difference evaluation module, used to determine a reverse overlap rate according to the overlap rates between the multiple model reply texts, wherein the reverse overlap rate is used to indicate the degree of difference between the multiple model reply texts; The second scoring module is used to determine the final model quality score of the question-answering model in replying to the target question based on the initial model quality score and the reverse overlap rate.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the computer device implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 8.

Images