Quality evaluation method, device and equipment of question and answer model and storage medium

Abstract

The invention provides a question and answer model quality evaluation method, device and equipment and a storage medium, and the method comprises the steps: obtaining a model reply text corresponding to a target question and a plurality of reference reply texts, the model reply text being a reply text of the target question output by a question and answer model, the similarity among the reference reply texts is smaller than a preset threshold value; comparing the model reply text with the plurality of reference reply texts to obtain a reply quality score corresponding to the model reply text; and according to the reply quality score corresponding to the model reply text, determining a model quality score of replying the target question by the question and answer model. According to the technical scheme, quality evaluation of questions replied by the question and answer model can be realized, and the accuracy of quality evaluation is improved.

Description

Technical Field

[0001] The present application relates to the field of question-answering model quality assessment, and in particular to a question-answering model quality assessment method, apparatus, device and storage medium. Background Art

[0002] Intelligent question-answering dialogue system is one of the research hotspots in the field of artificial intelligence in recent years. It can help users solve problems, provide relevant information, and realize the convenience of human-computer interaction.

[0003] Dialogue quality assessment is a key issue in the research of intelligent question-answering dialogue systems, which involves the output effect of the system and the user experience. How to conduct quality assessment on intelligent question-answering dialogue systems has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for evaluating the quality of a question-answering model to perform quality evaluation on an intelligent question-answering dialogue system.

[0005] In a first aspect, a quality assessment method for a question-answering model is provided, comprising:

[0006] Obtain a model reply text and multiple reference reply texts corresponding to the target question, wherein the model reply text is the reply text of the target question output by the question-answering model, and the similarity between the multiple reference reply texts is less than a preset threshold;

[0007] Comparing the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text;

[0008] The model quality score of the question-answering model in replying the target question is determined according to the response quality score corresponding to the model response text.

[0009] In this technical solution, by obtaining a model reply text and multiple reference reply texts corresponding to a target question, the similarity between the multiple reference reply texts is less than a preset threshold; then the model reply text is compared with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text; finally, based on the reply quality score corresponding to the model reply text, the model quality score of the question-and-answer model's reply to the target question is determined, thereby achieving a quality assessment of the question-and-answer model's reply to the question; since the answer to a question is not unique, multiple reference reply texts with less similarity are used to compare with the model reply text output by the question-and-answer model to obtain the quality score of the question-and-answer model's reply to the question, which can reduce the assessment bias caused by evaluating the model reply text with a single reference answer, making the quality assessment of the question-and-answer model's reply to the question more reasonable and effective, and improving the accuracy of the quality assessment.

[0010] In combination with the first aspect, in a possible implementation, the comparing the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text includes: comparing the model reply text with a target reference reply text according to a recall-oriented text evaluation method to obtain a text quality score corresponding to the model reply text, the target reference reply text being any one of the multiple reference reply texts, the text quality score being a quality score obtained by text evaluation of the model reply text using the target reference reply text as a reference text; determining a reply quality score corresponding to the model reply text according to multiple text quality scores corresponding to the model reply text, the multiple text quality scores being quality scores obtained by text evaluation of the model reply text using the multiple reference reply texts as reference texts respectively. Evaluating the model reply text output by the question-answering model according to the recall-oriented evaluation method can make the reply quality score explainable.

[0011] In combination with the first aspect, in a possible implementation, the model reply text is compared with the target reference reply text according to the recall-oriented text evaluation method to obtain a text quality score corresponding to the model reply text, including: calculating the Rouge-L (recalloriented understudy for gisting evaluation longest common subsequence) value between the model reply text and the target reference reply text to obtain a text quality score corresponding to the model reply text. By using the Rouge-L value between the model reply text and the reference reply text as a text quality score for the model reply text, the text quality score can be made more reasonable.

[0012] In combination with the first aspect, in a possible implementation method, determining the reply quality score corresponding to the model reply text according to multiple text quality scores corresponding to the model reply text includes: calculating characteristic values ​​of the multiple text quality scores, the characteristic values ​​including at least one of a maximum value, a minimum value or an average value; determining the reply quality score corresponding to the model reply text according to the characteristic values ​​corresponding to the multiple quality scores.

[0013] In combination with the first aspect, there are multiple model reply texts; the determining the model quality score of the question-and-answer model's reply to the target question based on the reply quality scores corresponding to the model reply texts includes: determining the model quality score of the question-and-answer model's reply to the target question based on the reply quality scores corresponding to each of the multiple model reply texts. Determining the quality score of the question-and-answer model's reply to the question based on the reply quality scores of each of the multiple model reply texts output by the question-and-answer model for the same question can avoid score fluctuations caused by the randomness of the responses generated by the question-and-answer model, and ensure the rationality and reliability of the score.

[0014] In combination with the first aspect, in a possible implementation method, the model quality score of the question and answer model's reply to the target question is determined based on the reply quality scores corresponding to each of the multiple model reply texts, including: taking the average of the reply quality scores corresponding to each of the multiple model reply texts as the model quality score of the question and answer model's reply to the target question.

[0015] In combination with the first aspect, in a possible implementation method, obtaining the model response text corresponding to the target question includes: inputting the target question into the question-answering model multiple times in succession, obtaining the response text output by the question-answering model, and obtaining the multiple model response texts.

[0016] In a second aspect, a quality assessment device for a question-answering model is provided, comprising:

[0017] A text acquisition module, used to acquire a model reply text and a plurality of reference reply texts corresponding to a target question, wherein the model reply text is a reply text of the target question output by the question-answering model, and the similarity between the plurality of reference reply texts is less than a preset threshold;

[0018] A comparison module, used to compare the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text;

[0019] The scoring module is used to determine the model quality score of the question-answering model's response to the target question based on the response quality score corresponding to the model response text.

[0020] In a third aspect, a computer device is provided, comprising a memory and one or more processors, wherein the memory is connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the computer device implements the quality assessment method of the question-answering model of the first aspect.

[0021] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the quality assessment method of the question-answering model of the first aspect.

[0022] The present application can achieve the following technical effects: it realizes the quality evaluation of the responses of the question-answering model; since the answers to the questions are not unique, multiple reference response texts with small similarity are used to compare with the model response text output by the question-answering model to obtain the quality score of the question-answering model's response, which can reduce the evaluation bias caused by the unique reference answer evaluation model response text, making the quality evaluation of the question-answering model's response more reasonable and effective, and improving the accuracy of the quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a quality assessment method for a question-answering model provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of another quality assessment method for a question-answering model provided in an embodiment of the present application;

[0025] Figure 3 It is a structural schematic diagram of a quality assessment device for a question-answering model provided in an embodiment of the present application;

[0026] Figure 4 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0028] The technical solution of this application is applicable to model quality assessment scenarios, especially to question-answering model quality assessment scenarios. Question-answering model quality assessment refers to quality assessment of trained question-answering models to determine the performance of the models. The technical solution of this application can be applied to computer devices, including but not limited to mobile phones, laptops, desktop computers, and the like.

[0029] The technical solution of this application is described in detail below.

[0030] See also Figure 1 , Figure 1 A flow chart of a quality assessment method for a question-answering model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0031] S101, obtaining a model response text and multiple reference response texts corresponding to a target question.

[0032] Here, the target question is a question to be answered, and the target question can be any question. For example, the target question can be "My big screen is black, what's going on?".

[0033] The model response text corresponding to the target question is the response text of the target question output by the question-answering model after the target question is input into the question-answering model. For example, the target question "My screen is black, what's going on?" is input into the question-answering model, and the question-answering model outputs "Check the power connection: Make sure the power cord is firmly connected and the plug is not loose or broken." "Check the power connection: Make sure the power cord is firmly connected and the plug is not loose or broken" is the model response text.

[0034] The multiple reference response texts corresponding to the target question can be understood as multiple standard answers pre-set for the target question, and the multiple reference response texts corresponding to the target question are used as reference texts for the model response text. For example, for the target question "My big screen is black, what's going on?", multiple standard answers can be pre-set, namely: Standard answer A: Hello, please check whether the device is powered on. If it is powered on and the screen is still black, please transfer to manual. Standard answer B: Hello, the black screen of the big screen is likely to be a backlight power problem. Please confirm whether the backlight power indicator is on. Standard answer C: Hello, please confirm whether the switch indicator turns from red to blue and then to red. Standard answer A, standard answer B, and standard answer C are the multiple reference response texts corresponding to the target question "My big screen is black, what's going on?"

[0035] The multiple reference reply texts corresponding to the target question can be pre-set by the user and saved in the computer device. The reference reply texts set by the user for the target question can be obtained from the computer device to obtain the multiple reference reply texts corresponding to the target question.

[0036] The similarity between the multiple reference response texts corresponding to the target question is less than the preset threshold to ensure the differences between the multiple reference response texts. For example, if the preset threshold is 0.5, and the multiple reference response texts corresponding to the target question are the aforementioned standard answer A, standard answer B, and standard answer C, then the similarity between standard answer A and standard answer B should be less than 0.5, the similarity between standard answer A and standard answer C should be less than 0.5, and the similarity between standard answer B and standard answer C should be less than 0.5.

[0037] In a feasible implementation, the similarity between reference reply texts can be evaluated based on sentence vectors. The similarity between multiple reference reply texts is less than a preset threshold, which may mean that the similarity between sentence vectors corresponding to the multiple reference reply texts is less than the preset threshold. After obtaining multiple reference reply texts set by the user, the multiple reference reply texts can be converted into sentence vectors respectively to obtain multiple sentence vectors, and then the similarities between the multiple sentence vectors are calculated. If each calculated similarity is less than the preset threshold, it is determined that the similarity between the multiple reference reply texts is less than the preset threshold. Among them, the Euclidean distance, cosine distance, etc. of the two sentence vectors can be calculated to calculate the similarity between the two sentence vectors.

[0038] S102, comparing the model response text corresponding to the target question with multiple reference response texts corresponding to the target question to obtain a response quality score corresponding to the model response text.

[0039] In some possible cases, the model response text corresponding to the target question can be evaluated based on the recall-oriented text evaluation method to obtain the response quality score corresponding to the model response text. The response quality score corresponding to the model response text can be obtained through the following steps A1-A2:

[0040] A1. According to the recall-oriented text evaluation method, the model response text corresponding to the target question is compared with the target reference response text to obtain the text quality score corresponding to the model response text.

[0041] The target reference reply text is any one of the multiple reference reply texts corresponding to the target question. For example, if the multiple reference reply texts corresponding to the target question are the standard answer A, standard answer B, and standard answer C introduced in the aforementioned step S101, the target reference reply text can be the standard answer A, standard answer B, or standard answer C.

[0042] The text quality score corresponding to the model response text is the quality score obtained by using the target reference response text as the reference text and adopting a recall-oriented text evaluation method to perform text evaluation on the model response text.

[0043] In a feasible implementation, the Rouge-L value between the model reply text and the target reference reply text can be calculated to obtain the text quality score corresponding to the model reply text. The Rouge-L value between the model reply text and the target reference reply text can be calculated according to the following formulas (1)-(3):

[0044]

[0045]

[0046]

[0047] Where text1 is the model response text, text2 is the target reference response text, LCS(text1, text2) is the length of the largest common substring between the model response text and the target reference response text, L1 is the length of the target reference response text, L2 is the length of the model response text, β is a hyperparameter, R lcs is the recall rate, P lcs is the accuracy, F lcs It is the Rouge-L value.

[0048] By using the Rouge-L value between the model reply text and the reference reply text as a text quality score of the model reply text, the text quality score can be made more reasonable.

[0049] Optionally, other recall-oriented evaluation indicators between the model reply text and the target reference reply text may also be calculated to obtain a text quality score corresponding to the model reply text. For example, the Rouge-1 value, Rouge-2 value, or Rouge-N value between the model reply text and the target reference reply text may also be calculated to obtain a text quality score corresponding to the model reply text. This application is not limited thereto.

[0050] By taking each of the multiple reference reply texts as the target reference reply text, and taking the target reference reply text as the reference text, and performing text evaluation on the model reply text to obtain the text quality score corresponding to the model reply text, a plurality of text quality scores corresponding to the model reply text can be obtained. For example, if the multiple reference reply texts corresponding to the target question are the standard answer A, standard answer B, and standard answer C introduced in the aforementioned step S101, then the standard answer A, standard answer B, and standard answer C can be taken as the target reference reply texts, and the text quality score corresponding to the model reply text when the standard answer A is taken as the target reference reply text, the text quality score corresponding to the model reply text when the standard answer B is taken as the target reference reply text, and the text quality score corresponding to the model reply text when the standard answer C is taken as the target reference reply text can be calculated, thereby obtaining three text quality scores corresponding to the model reply text.

[0051] The method for calculating each text quality score is the same. For example, the Rouge-L value between each reference reply text in multiple reference reply texts and the model reply text can be calculated to obtain multiple Rouge-L values ​​as multiple text quality scores corresponding to the model reply text. Alternatively, the Rouge-1 value between each reference reply text in multiple reference reply texts and the model reply text can be calculated to obtain multiple Rouge-1 values ​​as multiple text quality scores corresponding to the model reply text.

[0052] A2. Determine a response quality score corresponding to the model response text according to multiple text quality scores corresponding to the model response text.

[0053] Here, the characteristic values ​​of multiple text quality scores corresponding to the model reply text can be calculated, and the characteristic value can be at least one of the maximum value, the minimum value or the average value; based on the characteristic values ​​of the multiple text quality scores corresponding to the model reply text, the reply quality score corresponding to the model reply text is determined.

[0054] For example, the multiple text quality scores corresponding to the model reply text are the multiple Rouge-L values ​​corresponding to the model reply text. The maximum value among the multiple Rouge-L values ​​can be determined as the reply quality score corresponding to the model reply text, that is, the reply quality score corresponding to the model reply text is s=max(Rouge-Li), i=1,2,…n, n is the number of Rouge-L values, that is, the number of reference reply texts corresponding to the target question. Alternatively, the average value among the multiple Rouge-L values ​​can be determined as the reply quality score corresponding to the model reply text, that is, the reply quality score corresponding to the model reply text is Alternatively, at least two characteristic values ​​of multiple Rouge-L values ​​may be weighted and summed to obtain a response quality score corresponding to the model response text. This application does not impose any restrictions.

[0055] Determining the maximum value of multiple text quality scores corresponding to the model reply text as the reply quality score corresponding to the model reply text can reduce the evaluation bias caused by evaluating the model reply text with a single reference answer. Evaluating the model reply text output by the question-answering model according to the recall-oriented evaluation method can make the reply quality score explainable.

[0056] In some other possible situations, the model response text corresponding to the target question can also be evaluated based on the bilingual evaluation understudy (BLEU), BERTScore and other text evaluation methods to obtain the response quality score corresponding to the model response text. This application does not make any restrictions.

[0057] S103, determining the model quality score of the question-answering model in replying to the target question according to the reply quality score corresponding to the model reply text.

[0058] In a feasible implementation, the response quality score corresponding to the model response text may be used as the model quality score of the question-answering model's response to the target question.

[0059] exist Figure 1 In the corresponding technical solution, by obtaining the model reply text and multiple reference reply texts corresponding to the target question, the similarity between the multiple reference reply texts is less than a preset threshold; then the model reply text is compared with the multiple reference reply texts to obtain the reply quality score corresponding to the model reply text; finally, according to the reply quality score corresponding to the model reply text, the model quality score of the question and answer model's reply to the target question is determined, thereby realizing the quality evaluation of the question and answer model's reply to the question; since the answer to the question is not unique, multiple reference reply texts with less similarity are used to compare with the model reply text output by the question and answer model to obtain the quality score of the question and answer model's reply to the question, which can reduce the evaluation bias caused by the unique reference answer evaluation model reply text, making the quality evaluation of the question and answer model's reply to the question more reasonable and effective, and improving the accuracy of the quality evaluation.

[0060] See also Figure 2 , Figure 2 A flow chart of another quality assessment method for a question-answering model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0061] S201, obtaining multiple model response texts and multiple reference response texts corresponding to the target question.

[0062] Here, the target question can be input into the question-answering model multiple times in succession, and the response texts output by the question-answering model multiple times in succession can be obtained as the multiple model response texts corresponding to the target question. For example, the target question "My big screen is black, what's going on?" can be input into the question-answering model three times in succession. The question-answering model outputs response 1 for the first time: "Check the power connection: Make sure the power cord is firmly connected and the plug is not loose or broken", and the question-answering model outputs response 2 for the second time: "Please restart the big screen: Press the power button on the big screen and hold it for a few seconds, then release the button", and the third time outputs response 3: "Check the signal source: Make sure the correct signal source is selected correctly", then the response 1, response 2, and response 3 output by the answer-answering model three times in succession are the multiple model response texts corresponding to the target question.

[0063] Regarding the specific implementation method of obtaining multiple reference reply texts corresponding to the target question, please refer to the relevant description of the aforementioned step S101, which will not be repeated here.

[0064] S202, comparing the target model response text corresponding to the target question with multiple reference response texts corresponding to the target question to obtain a response quality score corresponding to the target model response text.

[0065] Here, the target model reply text is any one of the multiple model reply texts corresponding to the target question. For example, if the multiple model reply texts corresponding to the target question are reply 1, reply 2, and reply 3 in the aforementioned step S201, the target model reply text can be reply 1, reply 2, or reply 3.

[0066] Here, the text evaluation method described in the aforementioned step S102 can be referred to to evaluate the target model reply text corresponding to the target question to obtain the reply quality score corresponding to the target model reply text. According to the same text evaluation method, multiple model reply texts corresponding to the target question are evaluated respectively to obtain the reply quality scores corresponding to each of the multiple model reply texts.

[0067] For example, if the multiple model reply texts corresponding to the target question are reply 1, reply 2, and reply 3 in the aforementioned step S201, then you can refer to the text evaluation method introduced in the aforementioned step S102 to obtain the reply quality score corresponding to reply 1, the reply quality score corresponding to reply 2, and the reply quality score corresponding to reply 3, respectively.

[0068] S203, determining the model quality score of the question-answering model in replying the target question according to the response quality scores corresponding to the multiple model response texts.

[0069] In a feasible implementation, the average of the response quality scores corresponding to the multiple model response texts can be determined as the model quality score of the question-answering model answering the target question, that is, the model quality score of the question-answering model answering the target question is sj is the response quality score corresponding to the j-th model response text corresponding to the target question, and m is the number of response quality scores, that is, the number of model response texts corresponding to the target question.

[0070] Optionally, the response quality scores corresponding to multiple model response texts can be weighted and averaged to obtain the model quality score of the question-answering model answering the target question, that is, the model quality score of the question-answering model answering the target question is aj is the weight coefficient corresponding to the jth model reply text corresponding to the target question. Among them, the weight coefficient corresponding to each model reply text can be determined based on the frequency of the question-answering model outputting the model reply text. Taking the multiple model reply texts corresponding to the target question as reply 1, reply 2, and reply 3 in the aforementioned step S201 as an example, assuming that the target question is input into the question-answering model 30 times in succession, the question-answering model outputs reply 1 15 times, replies 2 6 times, and replies 3 9 times, then the question-answering model outputs reply 1 with a frequency of 0.5, replies 2 with a frequency of 0.2, and replies 3 with a frequency of 0.3. 0.5 can be determined as the weight coefficient corresponding to reply 1, 0.2 can be determined as the weight coefficient corresponding to reply 2, and 0.3 can be determined as the weight coefficient corresponding to reply 3.

[0071] This application does not specify how to determine the model quality score of the question-answering model's response to the target question based on the response quality scores corresponding to each of the multiple model response texts.

[0072] exist Figure 2 The corresponding technical solution obtains multiple model response texts corresponding to the target question, and respectively determines the response quality scores corresponding to the multiple model response texts, and finally determines the model quality score of the question-and-answer model's response to the target question based on the response quality scores corresponding to the multiple model response texts, thereby realizing the quality assessment of the question-and-answer model's response to the question; determining the quality score of the question-and-answer model's response to the same question based on the response quality scores of the multiple model response texts output by the question-and-answer model can avoid the score fluctuation caused by the randomness of the responses generated by the question-and-answer model, and ensure the rationality and reliability of the score.

[0073] In some possible cases, for the above Figure 1 and Figure 2 The question-answering model involved in the corresponding technical solution can also be based on the above Figure 1 or Figure 2 The corresponding technical solution determines the model quality scores of the question-answering model for answering multiple questions respectively, and the multiple questions can be different; then, based on the model quality scores of the question-answering model for answering multiple questions, the overall quality score of the question-answering model is determined. Among them, the overall quality score of the question-answering model can be the sum of the model quality scores corresponding to the multiple questions, or the mean of the model quality scores corresponding to the multiple questions. In this way, the overall quality evaluation of the question-answering model can be achieved.

[0074] The method of the present application is introduced above, and the device of the present application is introduced below.

[0075] See also Figure 3 , Figure 3 Schematic diagram of a quality assessment device for a question-answering model provided in an embodiment of the present application. Figure 3 As shown, the quality assessment device 30 of the question-answering model includes:

[0076] A text acquisition module 301 is used to acquire a model reply text and a plurality of reference reply texts corresponding to a target question, wherein the model reply text is a reply text of the target question output by the question-answering model, and the similarity between the plurality of reference reply texts is less than a preset threshold;

[0077] A comparison module 302 is used to compare the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text;

[0078] The scoring module 303 is used to determine the model quality score of the question-answering model's response to the target question based on the response quality score corresponding to the model response text.

[0079] In one possible design, the comparison module 302 is specifically used to: compare the model reply text with the target reference reply text according to a recall-oriented text evaluation method to obtain a text quality score corresponding to the model reply text, the target reference reply text is any one of the multiple reference reply texts, and the text quality score is a quality score obtained by performing a text evaluation on the model reply text with the target reference reply text as a reference text; determine a reply quality score corresponding to the model reply text according to multiple text quality scores corresponding to the model reply text, and the multiple text quality scores are quality scores obtained by performing a text evaluation on the model reply text with the multiple reference reply texts as reference texts respectively.

[0080] In a possible design, the comparison module 302 is specifically used to calculate a Rouge-L value between the model reply text and the target reference reply text to obtain a text quality score corresponding to the model reply text.

[0081] In one possible design, the comparison module 302 is specifically used to: calculate the characteristic values ​​of the multiple text quality scores, where the characteristic values ​​include at least one of a maximum value, a minimum value or an average value; and determine the reply quality score corresponding to the model reply text based on the characteristic values ​​corresponding to the multiple quality scores.

[0082] In one possible design, there are multiple model reply texts; the scoring module 303 is specifically used to determine the model quality score of the question-answering model replying to the target question based on the reply quality scores corresponding to the multiple model reply texts.

[0083] In a possible design, the scoring module 303 is specifically used to determine the average of the response quality scores corresponding to the multiple model response texts as the model quality score of the question-answering model's response to the target question.

[0084] In a possible design, the text acquisition module 301 is specifically used to: input the target question into the question-answering model multiple times in succession, obtain the reply text output by the question-answering model, and obtain the multiple model reply texts.

[0085] It should be noted that Figure 3 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0086] The above-mentioned device obtains a model reply text and multiple reference reply texts corresponding to the target question, and the similarity between the multiple reference reply texts is less than a preset threshold; then the model reply text is compared with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text; finally, according to the reply quality score corresponding to the model reply text, the model quality score of the question and answer model's reply to the target question is determined, thereby realizing the quality evaluation of the question and answer model's reply to the question; since the answer to the question is not unique, multiple reference reply texts with less similarity are used to compare with the model reply text output by the question and answer model to obtain the quality score of the question and answer model's reply to the question, which can reduce the evaluation bias caused by the unique reference answer evaluation model reply text, making the quality evaluation of the question and answer model's reply to the question more reasonable and effective, and improving the accuracy of the quality evaluation.

[0087] See also Figure 4 , Figure 4 4 is a schematic diagram of a computer device provided in an embodiment of the present application, wherein the computer device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example, via a bus.

[0088] The processor 401 is configured to support the computer device 40 to perform the corresponding functions in the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0089] The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

[0090] The processor 401 may call the program code to perform the following operations:

[0091] Obtain a model reply text and multiple reference reply texts corresponding to the target question, wherein the model reply text is the reply text of the target question output by the question-answering model, and the similarity between the multiple reference reply texts is less than a preset threshold;

[0092] Comparing the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text;

[0093] The model quality score of the question-answering model in replying the target question is determined according to the response quality score corresponding to the model response text.

[0094] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0095] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0096] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A quality assessment method for a question-answering model, characterized in that: include: Obtain a model reply text and multiple reference reply texts corresponding to the target question, wherein the model reply text is the reply text of the target question output by the question-answering model, and the similarity between the multiple reference reply texts is less than a preset threshold; Comparing the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text; The model quality score of the question-answering model in replying the target question is determined according to the response quality score corresponding to the model response text.

2. The method according to claim 1, characterized in that The step of comparing the model reply text with the plurality of reference reply texts to obtain a reply quality score corresponding to the model reply text includes: According to the recall-oriented text evaluation method, the model reply text is compared with a target reference reply text to obtain a text quality score corresponding to the model reply text, wherein the target reference reply text is any one of the multiple reference reply texts, and the text quality score is a quality score obtained by performing text evaluation on the model reply text with the target reference reply text as a reference text; According to multiple text quality scores corresponding to the model reply text, the reply quality score corresponding to the model reply text is determined, and the multiple text quality scores are quality scores obtained by performing text evaluation on the model reply text using the multiple reference reply texts as reference texts respectively.

3. The method according to claim 2, characterized in that The recall-oriented text evaluation method compares the model response text with the target reference response text to obtain a text quality score corresponding to the model response text, including: The Rouge-L value between the model reply text and the target reference reply text is calculated to obtain a text quality score corresponding to the model reply text.

4. The method according to claim 2, characterized in that: The step of determining the response quality score corresponding to the model response text according to the plurality of text quality scores corresponding to the model response text comprises: Calculating feature values ​​of the plurality of text quality scores, the feature values ​​comprising at least one of a maximum value, a minimum value, or an average value; A reply quality score corresponding to the model reply text is determined according to the feature values ​​corresponding to the multiple quality scores.

5. The method according to any one of claims 1 to 4, characterized in that: The model reply text has multiple items; Determining the model quality score of the question-answering model in replying the target question according to the reply quality score corresponding to the model reply text includes: The model quality score of the question-answering model in replying the target question is determined according to the response quality scores corresponding to the multiple model response texts.

6. The method according to claim 5, characterized in that Determining the model quality score of the question-answering model in replying the target question according to the reply quality scores corresponding to the plurality of model reply texts includes: The average of the response quality scores corresponding to the multiple model response texts is determined as the model quality score of the question-answering model in responding to the target question.

7. The method according to claim 5, characterized in that The step of obtaining the model response text corresponding to the target question includes: The target question is input into the question-answering model multiple times in succession, and the reply text output by the question-answering model is obtained to obtain the multiple model reply texts.

8. A quality assessment device for a question-answering model, characterized in that: include: A text acquisition module, used to acquire a model reply text and a plurality of reference reply texts corresponding to a target question, wherein the model reply text is a reply text of the target question output by the question-answering model, and the similarity between the plurality of reference reply texts is less than a preset threshold; A comparison module, used to compare the model reply text with the multiple reference reply texts to obtain a reply quality score corresponding to the model reply text; The scoring module is used to determine the model quality score of the question-answering model's response to the target question based on the response quality score corresponding to the model response text.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the computer device implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images