Question and answer model evaluation method, computer equipment, storage medium and program product

Abstract

The embodiment of the invention provides an evaluation method of a question and answer model, computer equipment, a storage medium and a program product. The method can be applied to the field of artificial intelligence, and comprises the following steps: respectively carrying out word segmentation processing on a reference text and a reply text to obtain a first reference segmented word and a first reply segmented word; the reply text is a text replied by the question-answer model for the question, and the reference text is a reference answer of the question; performing synonym expansion on the first reference segmented word and the first reply segmented word to obtain a second reference segmented word and a second reply segmented word; determining an integrity index of the reply text based on similar segmented words in the first reference segmented word and the second reply segmented word; determining an effectiveness index of the reply text based on similar segmented words in the first reply segmented word and the second reference segmented word; and determining a reply evaluation score of the question and answer model based on the integrity index and the validity index of the reply text. The method is used for achieving the effect of improving the efficiency of evaluating the question and answer model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an evaluation method for a question-answering model, a computer device, a storage medium, and a program product. Background Art

[0002] The question-answering model is used to perform question-answering tasks. In order to ensure the accuracy of the questions answered by the question-answering model, it is necessary to evaluate the answer effect of the question-answering model.

[0003] In the related art, a referee model is trained and used to evaluate the answering effect of the question-answering model. The process of training the referee model and the process of evaluating the referee model require processing a large amount of data and a large amount of calculation, resulting in low efficiency in evaluating the answering effect of the question-answering model. Summary of the invention

[0004] The embodiments of the present application provide a question-answering model evaluation method, a computer device, a storage medium, and a program product to achieve the effect of improving the efficiency of evaluating the question-answering model.

[0005] In a first aspect, an embodiment of the present application provides an evaluation method for a question-answering model, including: performing segmentation processing on a reference text and a reply text respectively to obtain a first reference segmentation and a first reply segmentation; the reply text is the text responded to a question by the question-answering model, and the reference text is the reference answer to the question; performing synonym expansion on the first reference segmentation and the first reply segmentation respectively to obtain a second reference segmentation and a second reply segmentation; determining a completeness index of the reply text based on similar segmentations in the first reference segmentation and the second reply segmentation; determining a validity index of the reply text based on similar segmentations in the first reply segmentation and the second reference segmentation; and determining a reply evaluation score of the question-answering model based on the completeness index and the validity index of the reply text.

[0006] In a second aspect, an embodiment of the present application provides an evaluation device for a question-answering model, comprising:

[0007] The word segmentation module is used to perform word segmentation processing on the reference text and the reply text respectively to obtain the first reference word segmentation and the first reply word segmentation; the reply text is the text replied by the question-answering model to the question, and the reference text is the reference answer to the question;

[0008] An expansion module, used for performing synonym expansion on the first reference participle and the first reply participle respectively, to obtain a second reference participle and a second reply participle;

[0009] A first indicator determination module, used to determine the completeness indicator of the reply text based on the first reference segmentation and the similar segmentation in the second reply segmentation;

[0010] A second indicator determination module, used to determine the validity indicator of the reply text based on similar segmentations in the first reply segmentation and the second reference segmentation;

[0011] The evaluation module is used to determine the response evaluation score of the question-answering model based on the completeness index and effectiveness index of the response text.

[0012] In a third aspect, an embodiment of the present application provides a computer device, comprising: a memory, a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the first aspect above and / or various possible implementations of the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0015] The evaluation method, computer device, storage medium and program product of the question-answering model provided in the embodiment of the present application respectively perform segmentation processing on the reference text and the reply text, and then perform synonym expansion on the obtained segmentations to obtain a second reference segmentation and a second reply segmentation respectively, and determine the completeness index of the reply text according to similar segmentations in the first reference segmentation and the second reply segmentation, and determine the validity index of the reply text according to similar segmentations in the first reply segmentation and the second reference segmentation, and increase the segmentations with the same meaning in the reference text and the reply text through synonym expansion, that is, the number of determined similar segmentations is increased through synonym expansion, thereby improving the accuracy of the completeness index and the validity index; determine the reply evaluation score of the question-answering model according to the completeness index and the validity index of the reply text, that is, comprehensively evaluate the reply effect of the question-answering model from the completeness and validity of the reply of the question-answering model, thereby improving the comprehensiveness of the evaluation of the reply effect of the question-answering model, and this process does not require the introduction of an additional neural network model, and involves a small amount of data and calculation, and can quickly determine the reply evaluation score of the question-answering model, thereby improving the efficiency of evaluating the question-answering model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] Figure 1A schematic diagram of a scenario for the evaluation method of the question-answering model provided in this application;

[0018] Figure 2 Schematic diagram of the evaluation method of the question-answering model provided in this application Figure 1 ;

[0019] Figure 3 Schematic diagram of the evaluation method of the question-answering model provided in this application Figure 2 ;

[0020] Figure 4 A schematic diagram of the structure of the evaluation device of the question-answering model provided in this application;

[0021] Figure 5 A schematic diagram of the structure of the computer device provided for this application.

[0022] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0024] It should be noted that the evaluation method, computer equipment, storage medium and program product of the question-answering model provided in this application can be used in the field of artificial intelligence, and can also be used in any field other than artificial intelligence. The application field of the evaluation method, computer equipment, storage medium and program product of the question-answering model in this application is not limited.

[0025] The evaluation method of the question-answering model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers; the media information display method can be executed by the terminal 102 or the server 104, or it can be executed by the terminal 102 and the server 104 in collaboration.

[0026] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc.

[0027] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0028] Figure 2 Schematic diagram of the evaluation method of the question-answering model provided in this application Figure 1 , we take the evaluation method of the question-answering model applied to computer equipment as an example. The computer equipment can be Figure 1 The terminal or server in Figure 2 As shown, the method includes:

[0029] S201. Perform word segmentation processing on the reference text and the reply text respectively to obtain a first reference word segmentation and a first reply word segmentation; the reply text is the text that the question-answering model responds to the question, and the reference text is the reference answer to the question.

[0030] Among them, the question-answering model can be a large model that performs question-answering tasks. In practical applications, the large model is trained with data in advance so that it has the ability to understand knowledge in a specific field, and then the large model can perform question-answering tasks in a specific field. A specific field may be, for example, the business field of a financial institution.

[0031] A question can be text that a user inputs into a question-answering model and that needs to be processed by the question-answering model; for example, the question is: "What is the processing procedure for a certain financial business?"

[0032] The reference text is the reference answer to the pre-set question; the response text is the text that the question-answering model responds to the question.

[0033] Specifically, the computer device obtains the problem processed by the large model, obtains the reply text output by the large model to the problem, and obtains the reference text corresponding to the problem; uses a word segmenter to segment the reply text to obtain a first reply word segmentation, and segments the reference text to obtain a first reference word segmentation.

[0034] S202 , perform synonym expansion on the first reference participle and the first reply participle respectively to obtain a second reference participle and a second reply participle.

[0035] Among them, synonym expansion refers to obtaining synonyms.

[0036] Specifically, the computer device obtains a reference synonym of the first reference participle, and determines a second reference participle based on the first reference participle and the reference synonym; obtains a reply synonym of the first reply participle, and determines a second reply participle based on the first reply participle and the reply synonym; that is, the second reference participle includes the first reference participle and the reply synonym, and the second reply participle includes the first reply participle and the reply synonym.

[0037] The computer device may obtain a reference synonym of the first reference participle from a synonym library, and obtain a reply synonym of the first reply participle.

[0038] S203: Determine a completeness index of the reply text based on the first reference segmentation and similar segmentations in the second reply segmentation.

[0039] Among them, the similar participles in the first reference participle and the second reply participle include the participles in the second reply participle that are similar to the first reference participle; the completeness index is used to indicate the completeness of the reply text; for example, if the words in the reference text that are similar to those in the reply text occupy a larger proportion of the reference text, then the completeness index of the reply text is higher.

[0040] Specifically, the computer device compares the first reference participle and the second reply participle, selects a participle similar to the first reference participle from the second reply participle, and determines the integrity index based on the selected participle similar to the first reference participle and the first reference participle.

[0041] In an optional manner, based on the same segmentation in the first reference segmentation and the second reply segmentation, the completeness index of the reply text is determined; specifically, the computer device compares the first reference segmentation and the second reply segmentation, selects the segmentation that is the same as the first reference segmentation in the second reply segmentation, and determines the completeness index based on the selected segmentation that is the same as the first reference segmentation and the first reference segmentation. It can be understood that when the similarity of two segmentations reaches 100%, the two segmentations are the same, and therefore, similar segmentations include the same segmentations.

[0042] S204: Determine the validity index of the reply text based on the similar participles in the first reply participle and the second reference participle.

[0043] Among them, similar participles in the first reply participle and the second reference participle include participles in the second reference participle that are similar to the first reply participle; the effectiveness index is used to indicate the effectiveness of the reply text; for example, if the words in the reply text that are similar to those in the reference text occupy a larger proportion of the reply text, the effectiveness index of the reply text is higher.

[0044] The computer device compares the second reference participle and the first reply participle, selects a participle similar to the first reply participle from the second reference participle, and determines the validity index based on the selected participle similar to the first reply participle and the first reply participle.

[0045] In an optional manner, the validity index is determined based on the same segmentation in the first reply segmentation and the second reference segmentation; specifically, the computer device compares the second reference segmentation with the first reply segmentation, selects the segmentation that is the same as the first reply segmentation in the second reference segmentation, and determines the validity index based on the selected segmentation that is the same as the first reply segmentation and the first reply segmentation. It can be understood that when the similarity of two segmentations reaches 100%, the two segmentations are the same, and therefore, similar segmentations include the same segmentations.

[0046] S205. Determine the response evaluation score of the question-answering model based on the completeness index and effectiveness index of the response text.

[0047] Among them, the reply evaluation score is used to reflect the reply effect of the question-answering model.

[0048] Specifically, the computer device fuses the completeness index and the effectiveness index of the reply text to obtain a target score for the reply text, and fuses the target scores corresponding to the reply texts corresponding to multiple questions to obtain a reply evaluation score.

[0049] In an optional method, the weights of the integrity index and the effectiveness index are obtained respectively, and the weighted sum of the integrity index and the effectiveness index is performed according to their weights to obtain the target score of the reply text, and the target scores of the reply texts corresponding to multiple questions are averaged to obtain the reply evaluation score of the question-answering model.

[0050] In an optional method, the reciprocal of the completeness index is calculated, the reciprocal of the effectiveness index is calculated, the sum of the reciprocals of the completeness index and the effectiveness index is calculated, the reciprocal of the sum is calculated, the target score of the reply text is obtained, the target scores of the reply texts corresponding to multiple questions are averaged, and the reply evaluation score of the question and answer model is obtained.

[0051] The evaluation method of the question-answering model performs word segmentation processing on the reference text and the reply text respectively, and then performs synonym expansion on the obtained word segmentations to obtain a second reference word segmentation and a second reply word segmentation respectively. According to the similar word segmentations in the first reference word segmentation and the second reply word segmentation, the completeness index of the reply text is determined, and according to the similar word segmentations in the first reply word segmentation and the second reference word segmentation, the validity index of the reply text is determined. Through synonym expansion, the word segmentations with the same meaning in the reference text and the reply text are increased, that is, the number of determined similar word segmentations is increased through synonym expansion, thereby improving the accuracy of the completeness index and the validity index; according to the completeness index and the validity index of the reply text, the reply evaluation score of the question-answering model is determined, that is, the reply effect of the question-answering model is comprehensively evaluated from the completeness and validity of the reply of the question-answering model, thereby improving the comprehensiveness of the evaluation of the reply effect of the question-answering model, and this process does not require the introduction of an additional neural network model, and the amount of data and calculation involved is small, and the reply evaluation score of the question-answering model can be quickly determined, thereby improving the efficiency of evaluating the question-answering model.

[0052] In some embodiments, based on the first reference participle and similar participles in the second reply participle, determining the completeness index of the reply text includes: determining a first candidate participle similar to the first reference participle in the second reply participle; and determining the completeness index of the reply text based on the first candidate participle and the first reference participle.

[0053] Specifically, the computer device traverses the first reference participle to obtain the first reference participle currently being processed, determines whether there is a participle similar to the first reference participle currently being processed in the second reply participle, and if so, uses the participle as the first candidate participle and continues to traverse the first reference participle; if not, continues to traverse the first reference participle until the traversal of the first reference participle is completed; counts the number of first candidate participles obtained in the second reply participle, and determines the completeness index of the reply text based on the number of first candidate participles and the number of first reference participles.

[0054] Among them, determining whether there is a segmentation similar to the currently processed first reference segmentation in the second reply segmentation can be done by comparing the second reply segmentation with segmentations similar to the first reference segmentation, or by comparing the distance between the word vector of the second reply segmentation and the word vector of the first reference segmentation, and determining whether the second reply segmentation and the first reference segmentation are similar based on the distance.

[0055] In an optional manner, in order to reduce the amount of calculation and improve efficiency, the same segmentation as the first reference segmentation may be selected in the second reply segmentation.

[0056] It should be noted that the second reply participle includes the first reply participle and the corresponding synonymous reply participle. If the first reply participle is similar (or identical) to the first reference participle, the first reply participle is used as the first candidate participle. If the first reply participle is not similar (or identical) to the first reference participle, the synonymous reply participle corresponding to the first reply participle is compared with the first reference participle to see whether they are similar (or identical). If they are similar (or identical), the synonymous reply participle is used as the first candidate participle. Therefore, the first candidate participle will not include the first reply participle and the corresponding synonymous reply participle at the same time.

[0057] In the above embodiment, the integrity index and validity index of the reply text are determined by using the second reference segmentation and similar segmentations in the second reply segmentation, so that the integrity index and validity index of the reply text can be quickly obtained, thereby improving the evaluation efficiency.

[0058] In some embodiments, the first candidate participle includes a first participle and a second participle; in the second response participle, determining a first candidate participle similar to the first reference participle includes: extracting a first key participle from the first reference participle; in the second response participle, obtaining a first participle similar to the first key participle; in the second response participle, obtaining a second participle similar to the first reference participle.

[0059] The first key participle is a participle expressing key semantics in the first reference participle.

[0060] Specifically, the computer device obtains the part of speech of the first reference participle, and takes the noun and verb in the first reference participle as the first key participle; it should be noted that the noun and verb are the key information in the first reference participle; for each first key participle, a first participle similar to the targeted first key participle is determined in the second response participle, and for each second reference participle, a second participle similar to the targeted second reference participle is determined in the second response participle.

[0061] In an optional manner, in the second reply participle, the first participle that is the same as the first key participle is obtained; in the second reply participle, the second participle that is the same as the second reference participle is obtained; the amount of calculation can be reduced and the efficiency can be improved.

[0062] It should be noted that the second reply participle includes the first reply participle and the corresponding synonymous reply participle. If the first reply participle is similar (or identical) to the first key participle, the first reply participle will be used as the first participle. If the first reply participle is not similar (or identical) to the first key participle, the synonymous reply participle corresponding to the first reply participle is compared to see whether it is similar (or identical) to the first key participle. If so, the synonymous reply participle will be used as the first participle. Therefore, the first participle will not include the first reply participle and the corresponding synonymous reply participle at the same time; similarly, the second participle will not include the first reply participle and the corresponding synonymous reply participle at the same time.

[0063] In the above embodiment, the first key participle is extracted from the first reference participle, so that the response effect of the question-answering model on the key information is subsequently determined based on the first key participle, thereby improving the comprehensiveness of the evaluation of the question-answering model.

[0064] In some embodiments, the second reply participle includes the first reply participle and the synonymous reply participle; in the second reply participle, obtaining the first participle similar to the first key participle includes: obtaining the first similar participle similar to the first key participle in the first reply participle, and obtaining the second similar participle similar to the first key participle in the synonymous reply participle if the first similar participle is not obtained; the first participle includes the first similar participle and the second similar participle;

[0065] In the second reply participle, obtaining a second participle similar to the first reference participle includes: obtaining a third similar participle similar to the first reference participle in the first reply participle, and obtaining a fourth similar participle similar to the first reference participle in the synonymous reply participle if the third similar participle is not obtained; the second participle includes the third similar participle and the fourth similar participle.

[0066] Specifically, for the current first key participle, for each first reply participle, determine whether the first reply participle is similar to the first key participle. If they are similar, the first reply participle is used as the first similar participle, and the next first key participle is processed; if they are not similar, that is, the first similar participle is not obtained, determine whether the synonymous reply participle corresponding to the first reply participle is similar to the first key participle. If they are similar, the synonymous reply participle corresponding to the first reply participle is used as the second similar participle. If they are not similar, the next first key participle is processed. In this way, the first participle (including the first similar participle and the second similar participle) will not include the first reply participle and the synonymous reply participle corresponding to the first reply participle at the same time, avoiding the situation where reply participles with the same meaning are counted repeatedly.

[0067] For the current first reference participle, for each first reply participle, determine whether the first reply participle is similar to the first reference participle. If so, take the first reply participle as the third similar participle and process the next first reference participle. If not, that is, the third similar participle is not obtained, determine whether the synonymous reply participle corresponding to the first reply participle is similar to the first reference participle. If so, take the synonymous reply participle corresponding to the first reply participle as the fourth similar participle. If not, process the next first reference participle. In this way, the second participle (including the third similar participle and the fourth similar participle) will not include the first reply participle and the synonymous reply participle corresponding to the first reply participle at the same time, avoiding the situation where reply participles with the same meaning are counted repeatedly.

[0068] In the above embodiment, by replying with synonymous participles, participles with the same meaning in the reference text and the reply text are added, and the similarity between the participles can be determined from the level of meaning, thereby improving the accuracy of evaluating the question-answering model.

[0069] In some embodiments, the completeness index includes a key completeness index and a reply completeness index; determining the completeness index of the reply text based on the first candidate participle and the first reference participle, including: determining the key completeness index based on the number of first participles and the number of first key participles; determining the reply completeness index based on the number of second participles and the number of first reference participles.

[0070] Specifically, the computer device obtains a first quantity of the first participle, obtains a second quantity of the first key participle, and uses the ratio between the first quantity and the second quantity as a key integrity indicator; the computer device obtains a third quantity of the second participle, obtains a fourth quantity of the first reference participle, and uses the ratio between the third quantity and the fourth quantity as a response integrity indicator.

[0071] Exemplarily, the reference text includes sentence r1 and sentence r2;

[0072] The first reference participles corresponding to sentence r1 include: r1x1 / r1x2 / r1x3;

[0073] The first reference participle corresponding to sentence r2 includes: r2x1 / r2x2;

[0074] The reply text includes sentence a1 and sentence a2; the first reply participle corresponding to sentence a1 includes: a1y1 / r1x2 / a1y3 / a1y4;

[0075] The first response participles corresponding to sentence a2 include: a2y1 / r2x2 / a2y3.

[0076] Perform synonym expansion on the first reference participle and the first response participle, and we get:

[0077] The second reference participle corresponding to sentence r1: r1x1(x1t1, a1y1) / r1x2 / r1x3; that is, x1t1 and a1y1 are synonyms of r1x1;

[0078] The second reference participle corresponding to sentence r2 is: r2x1(a2y1) / r2x2, that is, a2y1 is a synonym of r2x1;

[0079] The second reply participles corresponding to sentence a1 include: a1y1(x1t1, r1x1) / r1x2 / a1y3 / a1y4; that is, x1t1 and r1x1 are synonyms of a1y1;

[0080] The second reply participles corresponding to sentence a2 include: a2y1(r2x1) / r2x2 / a2y3; that is, r2x1 is a synonym of a2y1.

[0081] Extract the first key participle in the first reference participle to obtain: the first key participle corresponding to sentence r1: r1x3; the first key participle corresponding to sentence r2: r2x2;

[0082] In the second reply participle, the first participle similar to the first key participle is obtained, and the result is: r2x2.

[0083] Determine the key integrity index: record the number of first participles as m1, and the number of first key participles as n1; the key integrity index is equal to m1 / n1. In this example, m1=1, n1=2, so the key integrity index is 1 / 2.

[0084] Determine the reply completeness index: record the number of second participles as m2, and the number of first reference participles as n2; the reply completeness index is equal to m2 / n2; in the second reply participle, obtain the second participle similar to the first reference participle, the process is as follows:

[0085] For the first reference participle r1x1 in sentence r1, there is r1x1 in the second response participle (the synonymous response participle corresponding to the first response participle a1y1 in sentence a1), which is the same as it; for the first reference participle r1x2 in sentence r1, there is r1x2 in the second response participle (the first response participle r1x2 in sentence a1), and for the first reference participle r1x3 in sentence r1, there is no participle similar to r1x3 in the second response participle; for the first reference participle r2x1 in sentence r2, there is r2x1 in the second response participle (the synonymous response participle corresponding to the first response participle a2y1 in sentence a2), and for the first reference participle r2x2 in sentence r2, there is r2x2 in the second response participle (the first response participle r2x2 in sentence a2); it can be seen that m2=4, n2=5, so the response completeness index is 4 / 5.

[0086] In the above embodiment, the first key participle is extracted from the first reference participle, and the key completeness index is determined based on the hit situation of the first key participle. The reply completeness index is determined based on the hit situation of the first reference participle, so that the completeness index includes the hit situation of the key participle, which improves the influence of the key information replied by the question and answer model on the reply effect and makes the evaluation more comprehensive.

[0087] In some embodiments, based on the first reply participle and similar participles in the second reference participle, determining the validity index of the reply text includes: determining a second candidate participle similar to the first reply participle in the second reference participle; and determining the validity index of the reply text based on the second candidate participle and the first reply participle.

[0088] Specifically, the computer device traverses the first reply participle to obtain the currently processed first reply participle, determines whether there is a participle similar to the currently processed first reply participle in the second reference participle, and if so, uses the participle as the second candidate participle and continues to traverse the first reply participle; if not, continues to traverse the first reply participle until the traversal of the first reply participle is completed, counts the number of second candidate participles obtained in the second reference participle, and determines the effectiveness index of the reply text based on the number of second candidate participles and the number of first reply participles.

[0089] Among them, determining whether there is a segmentation similar to the currently processed first reply segmentation in the second reference segmentation can be done by comparing the distance between the word vector of the second reference segmentation and the word vector of the first reply segmentation, and determining whether the second reference segmentation and the first reply segmentation are similar based on the distance.

[0090] In an optional manner, in order to reduce the amount of calculation and improve efficiency, it can be determined whether there is a segmentation in the second reference segmentation that is the same as the first reply segmentation currently being processed.

[0091] In the above embodiment, the effectiveness index of the reply text is determined by using the second reference segmentation and similar segmentations in the first reply segmentation, so that the effectiveness index of the reply text can be quickly obtained, thereby improving the evaluation efficiency.

[0092] In some embodiments, the second candidate participle includes a third participle and a fourth participle; in the second reference participle, determining a second candidate participle similar to the first response participle includes: extracting a second key participle from the first response participle; in the second reference participle, obtaining a third participle similar to the second key participle; in the second reference participle, obtaining a fourth participle similar to the first response participle.

[0093] Among them, the second key participle is the participle that expresses the key semantics in the first reply participle.

[0094] Specifically, the computer device obtains the part of speech of the first reply participle, and uses the noun and verb in the first reply participle as the second key participle. For each second key participle, a third participle similar to the targeted second key participle is determined in the second reference participle, and for each second reply participle, a fourth participle similar to the targeted second key participle is determined in the second reference participle.

[0095] In an optional manner, in the second reference participle, a third participle identical to the second key participle is obtained; in the second reference participle, a fourth participle identical to the second reply participle is obtained; this can reduce the amount of calculation and improve efficiency.

[0096] It should be noted that the second reference participle includes the first reference participle and the corresponding synonymous reference participle. If the first reference participle is similar (or identical) to the second key participle, the first reference participle is used as the third participle. If the first reference participle is not similar (or identical) to the second key participle, the synonymous reference participle corresponding to the first reference participle is compared with the second key participle to see whether they are similar (or identical). If they are similar (or identical), the synonymous reference participle is used as the third participle. Therefore, the third participle will not include the first reference participle and the corresponding synonymous reference participle at the same time; similarly, the fourth participle will not include the first reference participle and the corresponding synonymous reference participle at the same time.

[0097] In some embodiments, the second reference participle includes the first reference participle and a synonymous reference participle; in the second reference participle, obtaining a third participle similar to the second key participle includes: obtaining a fifth similar participle similar to the second key participle in the first reference participle, and if the fifth similar participle is not obtained, obtaining a sixth similar participle similar to the first key participle in the synonymous reference participle; the third participle includes the fifth similar participle and the sixth similar participle.

[0098] In the second reference participle, obtaining a fourth participle similar to the first response participle includes: obtaining a seventh similar participle similar to the first response participle in the first reference participle, and if the seventh similar participle is not obtained, obtaining an eighth similar participle similar to the first reference participle in the synonymous reference participle; the fourth participle includes the seventh similar participle and the eighth similar participle.

[0099] In the above embodiment, the second key participle is extracted from the first reply participle, so that the reply effect of the question-answering model on the key information is subsequently determined based on the second key participle, thereby improving the comprehensiveness of the evaluation of the question-answering model.

[0100] In some embodiments, the effectiveness index includes a key effectiveness index and a reply effectiveness index; based on the second candidate participle and the first reply participle, the effectiveness index of the reply text is determined, including: based on the number of third participles and the number of second key participles, the key effectiveness index is determined; based on the number of fourth participles and the number of first reply participles, the reply effectiveness index is determined.

[0101] Specifically, the computer device obtains the fifth number of the third participle, obtains the sixth number of the second key participle, and uses the ratio between the fifth number and the sixth number as the key effectiveness indicator; the computer device obtains the seventh number of the fourth participle, obtains the eighth number of the first reply participle, and uses the ratio between the seventh number and the eighth number as the reply effectiveness indicator.

[0102] Exemplarily, the reference text includes sentence r1 and sentence r2;

[0103] The first reference participles corresponding to sentence r1 include: r1x1 / r1x2 / r1x3;

[0104] The first reference participle corresponding to sentence r2 includes: r2x1 / r2x2;

[0105] The reply text includes sentence a1 and sentence a2; the first reply participle corresponding to sentence a1 includes: a1y1 / r1x2 / a1y3 / a1y4;

[0106] The first response participles corresponding to sentence a2 include: a2y1 / r2x2 / a2y3.

[0107] Perform synonym expansion on the first reference participle and the first response participle, and we get:

[0108] The second reference participle corresponding to sentence r1: r1x1(x1t1, a1y1) / r1x2 / r1x3; that is, x1t1 and a1y1 are synonyms of r1x1;

[0109] The second reference participle corresponding to sentence r2: r2x1(a2y1) / r2x2; that is, a2y1 is a synonym of r2x1;

[0110] The second reply participles corresponding to sentence a1 include: a1y1(x1t1, r1x1) / r1x2 / a1y3 / a1y4; that is, x1t1 and r1x1 are synonyms of a1y1;

[0111] The second reply participles corresponding to sentence a2 include: a2y1(r2x1) / r2x2 / a2y3; that is, r2x1 is a synonym of a2y1.

[0112] Extract the second key participle in the first reply participle to obtain the second key participle corresponding to sentence a1: a1y4; the second key participles corresponding to sentence a2: r2x2 and a2y3;

[0113] Obtain the third participle similar to the second key participle in the second reference participle, and obtain: r2x2;

[0114] Determine the key effectiveness index: record the number of third participles as m3, and the number of second key participles as n3; the key effectiveness index is equal to m3 / n3. In this example, m3=1, n3=3, so the key completeness index is 1 / 3.

[0115] Determine the reply validity index: record the number of fourth participles as m4, and the number of first reply participles as n4; the reply validity index is equal to m4 / n4; in the second reference participle, obtain the fourth participle similar to the first reply participle, the process is as follows:

[0116] For the first response participle a1y1 of sentence a1, there is a1y1 identical to it in the second reference participle; for the first response participle r1x2 of sentence a1, there is r1x2 identical to it in the second reference participle; for the first response participle a1y3 of sentence a1, there is no participle similar to r1x2 in the second reference participle; for the first response participle a1y4 of sentence a1, there is no participle similar to a1y4 in the second reference participle; for the first response participle a2y1 of sentence a2, there is a2y1 identical to it in the second reference participle; for the first response participle r2x2 of sentence a2, there is r2x2 identical to it in the second reference participle; for the first response participle a2y3 of sentence a2, there is no participle similar to a2y3 in the second reference participle; thus, m4=4, n4=7, so the response effectiveness index is 4 / 7.

[0117] In the above embodiment, the second key participle is extracted from the first reply participle, and the key effectiveness index is determined based on the hit situation of the second key participle, and the reply effectiveness index is determined based on the hit situation of the first reply participle, so that the hit situation of the key participle is included in the effectiveness index, which improves the influence of the key information of the question-answering model reply on the reply effect and makes the evaluation more comprehensive.

[0118] In some embodiments, based on the completeness index and effectiveness index of the reply text, determining the reply evaluation score of the question-answering model includes: determining a first harmonic mean based on the completeness index and effectiveness index of the reply text; determining the reply evaluation score of the question-answering model based on the first harmonic mean.

[0119] Specifically, the computer device determines the reciprocal of the integrity index, determines the reciprocal of the effectiveness index, calculates the sum of the reciprocals of the integrity index and the effectiveness index, calculates the reciprocal of the sum, obtains the first harmonic mean corresponding to the reply text, averages the first harmonic means of the reply texts corresponding to multiple questions, and obtains the answer evaluation score of the question and answer model.

[0120] For example, as shown in formula (1).

[0121] (1)

[0122] in, It's a problem Corresponding reply text The first harmonic mean of Is the reply text The integrity index, Is the reply text effectiveness indicators.

[0123] In some embodiments, the integrity indicator includes a key integrity indicator and a response integrity indicator, and the effectiveness indicator includes a key effectiveness indicator and a response effectiveness indicator; determining a first harmonic mean based on the integrity indicator and effectiveness indicator of the response text includes: determining the first harmonic mean based on the key integrity indicator, the response integrity indicator, the key effectiveness indicator and the response effectiveness indicator of the response text.

[0124] For example, as shown in formula (2).

[0125] (2)

[0126] in, It's a problem Corresponding reply text The first harmonic mean of Is the reply text Key integrity indicators, Is the reply text The response completeness indicator, Is the reply text The key effectiveness indicators are Is the reply text Response effectiveness indicator.

[0127] In the above embodiment, through the first harmonic mean, when the completeness index and the effectiveness index of the reply text perform well, the value of the first harmonic mean of the reply text is relatively high, and then the reply evaluation score of the question-answering model is relatively high, so that the reply evaluation score of the question-answering model is obtained by balancing the completeness index and the effectiveness index, thereby improving the reliability of the evaluation score.

[0128] In some embodiments, the evaluation method of the question-answering model also includes: when there is a word order identifier in the reference text, obtaining a word group in the second reference word segmentation according to the word order identifier; the word group includes a preceding word segmentation and a following word segmentation; in the second reply word segmentation, determining a second candidate word segmentation similar to the second reference word segmentation; when the second candidate word segmentation includes the word group, determining the word order difference value of the word group according to the order of the preceding word segmentation and the following word segmentation in the second candidate word segmentation; determining the word order index of the reply text according to the word order difference value of the word group; determining the reply evaluation score of the question-answering model based on the completeness index and effectiveness index of the reply text, including: determining a second harmonic mean based on the completeness index, effectiveness index and word order index of the reply text; determining the reply evaluation score of the question-answering model based on the second harmonic mean.

[0129] Among them, there are word order markers in the reference text, which means that there is a specific order between some participles in the reference text. If the order changes, the meaning of the reference text will change. Therefore, it is necessary to evaluate whether the order of these participles in the reply text is the same as that in the reference text, that is, to add word order indicators in the reply evaluation.

[0130] It should be noted that the pre-order participle and the post-order participle are usually in the same sentence in the reference text, that is, the pre-order participle and the post-order participle usually do not appear across sentences; that is to say, the pre-order participle and the post-order participle included in the word group are in the same sentence in the reference text. Therefore, the second candidate participle includes the word group, which means that the pre-order participle and the post-order participle exist in the same sentence of the second candidate participle.

[0131] The word order identifier includes a group number and a sequence number. The group number can be used to determine the preceding and succeeding participles belonging to the same participle group.

[0132] Exemplarily, the word order of r1x1 in the second reference participle is identified as o1, and the word order of r1x2 is identified as o2; wherein, "o" is the group number. Since r1x1 and r1x2 have the same group number, r1x1 and r1x2 belong to the same participle group. Since the "1" in o1 indicates arrangement in front, and the "2" in o2 indicates arrangement in the back, r1x1 is the preceding participle and r1x2 is the succeeding participle.

[0133] The word order difference value indicates whether the word order is consistent. When the word order of the word group in the second reference participle is the same as the word order of the word group in the second candidate participle, the word order difference value is 0; when the word order of the word group in the second reference participle is different from the word order of the word group in the second candidate participle, the word order difference value is 1.

[0134] Specifically, when the order of word segments in the reference text affects the semantics, a word order identifier is added to the word segment in advance; when the computer device determines that there is a word order identifier in the reference text, a word segment group is obtained in the second reference word segment; the number of word segment groups can be multiple; for each word segment group, when the second candidate word segment includes the preceding word segment and the following word segment in the word segment group, the order of the preceding word segment in the second candidate word segment is determined, and the order of the following word segment in the second candidate word segment is obtained.

[0135] If the order of the preceding participle in the second candidate participle is before the order of the following participle in the second candidate participle, the word order of the participle group in the second reference participle is the same as the word order of the participle group in the second candidate participle, and the word order difference value is 1; if the order of the preceding participle in the second candidate participle is after the order of the following participle in the second candidate participle, the word order of the participle group in the second reference participle is different from the word order of the participle group in the second candidate participle, and the word order difference value is 0.

[0136] The computer device calculates a first sum of the word order difference values ​​of each word group, determines a second sum of the first sum and 1, and uses the reciprocal of the second sum as a word order indicator.

[0137] For example, as shown in formula (3).

[0138] (3)

[0139] in, Is the reply text The word order indicator of It is a participle The word order difference value.

[0140] Exemplarily, the reference text includes sentence r1, sentence r2 and sentence r3; wherein there are word order identifiers in sentence r2 and sentence r3, such as: r2x1(o1) / r2x2(o2); r3x1(p1) / r3x2(p2) / r3x3. According to the word order identifier, the word group d1: (r2x1, r2x2) and the word group d2: (r3x1, r3x2) are obtained in the second reference word group, the preceding word group d1 is r2x1, and the following word group is r2x2, and the preceding word group d2 is r3x1, and the following word group is r3x2.

[0141] The second candidate word segmentations include: a2y1(r2x1) / r2x2 / a2y3; r3x2 / r3x3 / r3x1.

[0142] For word group d1 and word group d2; the order of the preceding word of word group d1 in the second candidate word is before the order of the following word of word group d1 in the second candidate word, so the word order difference value of word group d1 is 0, and the order of the preceding word of word group d2 in the second candidate word is after the order of the following word of word group d2 in the second candidate word, so the word order difference value of word group d2 is 1; according to the above formula (3), it can be determined that the word order index of the reply text is 1 / 2.

[0143] After determining the word order index of the reply text, the computer device determines the reciprocal of the completeness index, the reciprocal of the validity index and the reciprocal of the word order index, calculates the sum of the reciprocal of the completeness index, the reciprocal of the validity index and the reciprocal of the word order index, and then uses the reciprocal of the sum as the second harmonic mean corresponding to the reply text, averages the second harmonic means of the reply texts corresponding to multiple questions, and obtains the reply evaluation score of the question and answer model.

[0144] In the above embodiment, for word groups that have a word order in the reference text, it is necessary to determine whether the word order of the word group in the reference text is consistent with the word order of the word group in the reply text, and then determine the word order index of the reply text. The word order index is added to the evaluation of the question-answering model, thereby improving the comprehensiveness of the evaluation of the question-answering model.

[0145] In some embodiments, the completeness index includes a key completeness index and a reply completeness index, and the effectiveness index includes a key effectiveness index and a reply effectiveness index; determining the second harmonic mean based on the completeness index, effectiveness index and word order index of the reply text includes: determining the second harmonic mean based on the key completeness index, the reply completeness index, the key effectiveness index, the reply effectiveness index and the word order index.

[0146] Specifically, the computer device determines the reciprocal of the key integrity indicator, the reciprocal of the reply integrity indicator, the reciprocal of the key validity indicator, the reciprocal of the reply validity indicator and the reciprocal of the word order indicator, calculates the sum of the reciprocal of the key integrity indicator, the reciprocal of the reply integrity indicator, the reciprocal of the key validity indicator, the reciprocal of the reply validity indicator and the reciprocal of the word order indicator, and takes the reciprocal of the sum as the second harmonic mean.

[0147] For example, as shown in formula (4).

[0148] (4)

[0149] in, It's a problem Corresponding reply text The second harmonic mean of Is the reply text Key integrity indicators, Is the reply text The response completeness indicator, Is the reply text The key effectiveness indicators are Is the reply text Response effectiveness indicator.

[0150] In the above embodiment, the second harmonic mean is determined based on the key completeness indicator, the response completeness indicator, the key validity indicator, the response validity indicator and the word order indicator, and the response evaluation score of the question-answering model can be subsequently determined based on the second harmonic mean, so that the response evaluation score of the question-answering model is obtained by balancing the key completeness indicator, the response completeness indicator, the key validity indicator, the response validity indicator and the word order indicator, thereby improving the reliability of the evaluation score.

[0151] Figure 3 Schematic diagram of the evaluation method of the question-answering model provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the evaluation method of the question-answering model is described in detail, and the method includes:

[0152] S301, perform word segmentation processing on the reference text and the reply text respectively to obtain a first reference word segmentation and a first reply word segmentation; the reply text is the text replied by the question-answering model to the question, and the reference text is the reference answer to the question;

[0153] S302, performing synonym expansion on the first reference participle and the first reply participle respectively to obtain a second reference participle and a second reply participle; the second reply participle includes the first reply participle and the synonymous reply participle;

[0154] S303, extracting a first key participle from the first reference participle; obtaining a first similar participle similar to the first key participle from the first reply participle, and obtaining a second similar participle similar to the first key participle from the synonymous reply participle if the first similar participle is not obtained; the first participle includes the first similar participle and the second similar participle;

[0155] S304, obtaining a third similar participle similar to the first reference participle in the first reply participle, and obtaining a fourth similar participle similar to the first reference participle in the synonymous reply participle if the third similar participle is not obtained; the second participle includes the third similar participle and the fourth similar participle;

[0156] S305: Determine a key completeness index based on the number of first participles and the number of first key participles; determine a response completeness index based on the number of second participles and the number of first reference participles;

[0157] S306, extracting a second key participle from the first reply participle; obtaining a third participle similar to the second key participle from the second reference participle; obtaining a fourth participle similar to the first reply participle from the second reference participle;

[0158] S307, determining a key effectiveness index according to the number of the third participles and the number of the second key participles; determining a reply effectiveness index according to the number of the fourth participles and the number of the first reply participles;

[0159] S308, when there is a word order mark in the reference text, obtain a word group in the second reference word according to the word order mark; the word group includes a pre-order word and a post-order word; when the second candidate word includes the word group, determine the word order difference value of the word group according to the order of the pre-order word and the post-order word in the second candidate word; determine the word order index of the reply text according to the word order difference value of the word group;

[0160] S309. Determine a second harmonic mean based on the key completeness index, the response completeness index, the key effectiveness index, the response effectiveness index and the word order index; and determine the response evaluation score of the question-answering model based on the second harmonic mean.

[0161] The evaluation method of the question-answering model performs word segmentation processing on the reference text and the reply text respectively, and then performs synonym expansion on the obtained word segmentations to obtain a second reference word segmentation and a second reply word segmentation respectively. According to the similar word segmentations in the first reference word segmentation and the second reply word segmentation, the completeness index of the reply text is determined, and according to the similar word segmentations in the first reply word segmentation and the second reference word segmentation, the validity index of the reply text is determined. Through synonym expansion, the word segmentations with the same meaning in the reference text and the reply text are increased, that is, the number of determined similar word segmentations is increased through synonym expansion, thereby improving the accuracy of the completeness index and the validity index; according to the completeness index and the validity index of the reply text, the reply evaluation score of the question-answering model is determined, that is, the reply effect of the question-answering model is comprehensively evaluated from the completeness and validity of the reply of the question-answering model, thereby improving the comprehensiveness of the evaluation of the reply effect of the question-answering model, and this process does not require the introduction of an additional neural network model, and the amount of data and calculation involved is small, and the reply evaluation score of the question-answering model can be quickly determined, thereby improving the efficiency of evaluating the question-answering model.

[0162] Figure 4 A schematic diagram of the structure of the evaluation device of the question-answering model provided in this application, such as Figure 4 As shown, the evaluation device 40 of the question-answering model provided in this embodiment includes:

[0163] The word segmentation module 401 is used to perform word segmentation processing on the reference text and the reply text respectively to obtain a first reference word segmentation and a first reply word segmentation; the reply text is the text replied by the question-answering model to the question, and the reference text is the reference answer to the question;

[0164] An expansion module 402 is used to perform synonym expansion on the first reference participle and the first reply participle respectively to obtain a second reference participle and a second reply participle;

[0165] A first indicator determination module 403, configured to determine a completeness indicator of the reply text based on the first reference segmentation and similar segmentations in the second reply segmentation;

[0166] A second indicator determination module 404, configured to determine the effectiveness indicator of the reply text based on similar segmentations in the first reply segmentation and the second reference segmentation;

[0167] Evaluation module 405 is used to determine the response evaluation score of the question-answering model based on the completeness index and effectiveness index of the response text.

[0168] In a possible implementation, the first indicator determination module 403 is further used to determine a first candidate participle similar to the first reference participle in the second reply participle; and determine the completeness indicator of the reply text based on the first candidate participle and the first reference participle.

[0169] In one possible implementation, the first candidate participle includes a first participle and a second participle; the first determination module 403 is also used to extract a first key participle from the first reference participle; in the second reply participle, obtain a first participle similar to the first key participle; in the second reply participle, obtain a second participle similar to the first reference participle.

[0170] In one possible implementation, the second reply participle includes the first reply participle and a synonymous reply participle; the first determination module 403 is further used to obtain a first similar participle similar to the first key participle in the first reply participle, and if the first similar participle is not obtained, obtain a second similar participle similar to the first key participle in the synonymous reply participle; the first participle includes a first similar participle and a second similar participle; obtain a third similar participle similar to the first reference participle in the first reply participle, and if the third similar participle is not obtained, obtain a fourth similar participle similar to the first reference participle in the synonymous reply participle; the second participle includes a third similar participle and a fourth similar participle.

[0171] In one possible implementation, the integrity index includes a key integrity index and a response integrity index; the index determination module 403 is further used to determine the key integrity index based on the number of first participles and the number of first key participles; and to determine the response integrity index based on the number of second participles and the number of first reference participles.

[0172] In a possible implementation, the second indicator determination module 404 is further used to determine a second candidate participle similar to the first reply participle in the second reference participle; and determine the validity indicator of the reply text based on the second candidate participle and the first reply participle.

[0173] In one possible implementation, the second candidate participle includes a third participle and a fourth participle; the second determination module 404 is also used to extract a second key participle from the first reply participle; in the second reference participle, obtain a third participle similar to the second key participle; in the second reference participle, obtain a fourth participle similar to the first reply participle.

[0174] In one possible implementation, the effectiveness index includes a key effectiveness index and a reply effectiveness index; the second indicator determination module 404 is further used to determine the key effectiveness index based on the number of third participles and the number of second key participles; and to determine the reply effectiveness index based on the number of fourth participles and the number of first reply participles.

[0175] In a possible implementation, the evaluation module 405 is further used to determine a first harmonic mean based on the completeness index and the effectiveness index of the reply text; and determine the reply evaluation score of the question-answering model based on the first harmonic mean.

[0176] In a possible implementation, the evaluation device of the question-answering model further includes a third indicator determination module, which is used to obtain a word group in the second reference word segmentation according to the word order identifier when there is a word order identifier in the reference text; the word group includes a pre-order word segmentation and a post-order word segmentation; in the second reply word segmentation, determine a second candidate word segmentation similar to the second reference word segmentation; when the second candidate word segmentation includes the word group, determine the word order difference value of the word group according to the order of the pre-order word segmentation and the post-order word segmentation in the second candidate word segmentation; determine the word order indicator of the reply text according to the word order difference value of the word group;

[0177] The evaluation module 405 is also used to determine the second harmonic mean based on the completeness index, validity index and word order index of the reply text; and determine the reply evaluation score of the question-answering model based on the second harmonic mean.

[0178] In one possible implementation, the integrity index includes a key integrity index and a response integrity index, and the effectiveness index includes a key effectiveness index and a response effectiveness index; the evaluation module 405 is also used to determine the second harmonic mean based on the key integrity index, the response integrity index, the key effectiveness index, the response effectiveness index and the word order index.

[0179] The evaluation device of the question-answering model provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0180] Figure 5 This is a schematic diagram of the structure of the computer device provided in this application. Figure 5 As shown, the computer device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0181] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0182] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0183] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0184] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0185] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0186] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0187] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0188] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0189] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0190] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0193] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0194] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0195] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for evaluating a question-answering model, characterized in that: include: Perform word segmentation processing on the reference text and the reply text respectively to obtain a first reference word segmentation and a first reply word segmentation; The reply text is the text that the question-answering model replies to the question, and the reference text is the reference answer to the question; Performing synonym expansion on the first reference participle and the first reply participle respectively to obtain a second reference participle and a second reply participle; Determining a completeness index of the reply text based on similar participles in the first reference participle and the second reply participle; Determining a validity index of the reply text based on similar participles in the first reply participle and the second reference participle; Based on the completeness index and the effectiveness index of the reply text, the reply evaluation score of the question-answering model is determined.

2. The method according to claim 1, characterized in that The determining the completeness index of the reply text based on the similar participles in the first reference participle and the second reply participle includes: Determining, from the second reply participles, a first candidate participle that is similar to the first reference participle; Determine a completeness index of the reply text based on the first candidate segmentation word and the first reference segmentation word.

3. The method according to claim 2, characterized in that The first candidate participle includes a first participle and a second participle; Determining, in the second reply participle, a first candidate participle similar to the first reference participle, comprises: Extracting a first key participle from the first reference participles; In the second reply participle, obtaining the first participle that is similar to the first key participle; In the second reply participle, the second participle similar to the first reference participle is obtained.

4. The method according to claim 3, characterized in that The second response participle includes the first response participle and a synonymous response participle; The acquiring, in the second reply participle, the first participle that is similar to the first key participle comprises: Acquire a first similar participle similar to the first key participle in the first reply participle, and acquire a second similar participle similar to the first key participle in the synonymous reply participle if the first similar participle is not acquired; the first participle includes the first similar participle and the second similar participle; The acquiring, in the second reply participle, the second participle that is similar to the first reference participle comprises: A third similar participle similar to the first reference participle is obtained in the first response participle, and if the third similar participle is not obtained, a fourth similar participle similar to the first reference participle is obtained in the synonymous response participle; the second participle includes the third similar participle and the fourth similar participle.

5. The method according to claim 3, characterized in that: The integrity indicators include key integrity indicators and response integrity indicators; The determining, based on the first candidate segmentation word and the first reference segmentation word, a completeness index of the reply text includes: Determining the key integrity indicator according to the number of the first participles and the number of the first key participles; The reply completeness indicator is determined according to the number of the second participles and the number of the first reference participles.

6. The method according to claim 1, characterized in that The determining the validity index of the reply text based on similar participles in the first reply participle and the second reference participle includes: Determine, from the second reference participles, a second candidate participle that is similar to the first response participle; Determine a validity index of the reply text based on the second candidate segmentation word and the first reply segmentation word.

7. The method according to claim 6, characterized in that The second candidate participle includes a third participle and a fourth participle; The determining, in the second reference participles, a second candidate participle similar to the first reply participle, comprises: Extracting a second key participle from the first reply participle; In the second reference participle, obtaining the third participle similar to the second key participle; The fourth participle similar to the first reply participle is obtained from the second reference participle.

8. The method according to claim 7, characterized in that The effectiveness indicators include key effectiveness indicators and response effectiveness indicators; The determining, based on the second candidate segmentation word and the first reply segmentation word, a validity index of the reply text includes: Determining the key effectiveness indicator according to the number of the third participles and the number of the second key participles; The reply effectiveness index is determined according to the number of the fourth participles and the number of the first reply participles.

9. The method according to any one of claims 1 to 8, characterized in that The step of determining the answer evaluation score of the question-answer model based on the completeness index and the effectiveness index of the reply text includes: Determining a first harmonic mean according to the completeness indicator and the effectiveness indicator of the reply text; The answer evaluation score of the question-answering model is determined based on the first harmonic mean.

10. The method according to any one of claims 1 to 8, characterized in that The method further comprises: In the case where the reference text has a word order identifier, obtaining a word segment group in the second reference word segment according to the word order identifier; the word segment group includes a preceding word segment and a succeeding word segment; Determining, in the second reply participle, a second candidate participle that is similar to the second reference participle; In the case where the second candidate participle includes the participle group, determining the word order difference value of the participle group according to the order of the preceding participle and the succeeding participle in the second candidate participle; Determining a word order index of the reply text according to the word order difference value of the word group; The step of determining the answer evaluation score of the question-answer model based on the completeness index and the effectiveness index of the reply text includes: Determining a second harmonic mean according to the completeness index, the validity index and the word order index of the reply text; The answer evaluation score of the question-answering model is determined based on the second harmonic mean.

11. The method according to claim 10, characterized in that The integrity indicators include key integrity indicators and response integrity indicators, and the effectiveness indicators include key effectiveness indicators and response effectiveness indicators; The determining of the second harmonic mean according to the completeness index, the validity index and the word order index of the reply text comprises: A second harmonic mean is determined based on the key completeness indicator, the response completeness indicator, the key effectiveness indicator, the response effectiveness indicator and the word order indicator.

12. An evaluation device for a question-answering model, characterized in that: include: A word segmentation module is used to perform word segmentation processing on the reference text and the reply text respectively to obtain a first reference word segmentation and a first reply word segmentation; The reply text is the text that the question-answering model replies to the question, and the reference text is the reference answer to the question; An expansion module, configured to perform synonym expansion on the first reference participle and the first reply participle respectively, to obtain a second reference participle and a second reply participle; A first indicator determination module, configured to determine a completeness indicator of the reply text based on similar segmentations in the first reference segmentation and the second reply segmentation; A second indicator determination module, configured to determine a validity indicator of the reply text based on similar segmentations in the first reply segmentation and the second reference segmentation; An evaluation module is used to determine the response evaluation score of the question-answering model based on the completeness index and the effectiveness index of the response text.

13. A computer device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product, characterized in that The method comprises computer-executable instructions, which implement the method according to any one of claims 1 to 11 when the computer-executable instructions are executed by a processor.

Images