Model evaluation method, device, apparatus and storage medium
By automatically identifying negative example statements and updating test statements during model evaluation, the problem of poor model evaluation accuracy in existing technologies is solved, and automatic correction of evaluation results and efficiency improvement are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 北京中关村科金技术有限公司
- Filing Date
- 2022-05-31
- Publication Date
- 2026-06-02
AI Technical Summary
Existing model evaluation methods are not very accurate, mainly because the testers are subjective during the testing process, which leads to a large deviation between the evaluation results and the actual results.
By acquiring the test statements and original intent, an automated process is used to identify negative example statements. The test statements are then updated based on the training statements and negative example statements, and a second evaluation is performed to improve accuracy.
It enables automatic correction of evaluation results, improves the accuracy of model evaluation, saves labor costs, and increases evaluation efficiency.
Smart Images
Figure CN115859065B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model evaluation method, apparatus, device, and storage medium. Background Technology
[0002] Intent recognition, also known as intent detection, categorizes sentences into different classes, falling under the category of multi-class classification. In intent recognition tasks, machine learning is typically used to build a classification model. Then, testers simulate real-world user statements (i.e., test data) to test the model, and the model's deployment is determined based on the test results. However, due to the subjectivity of testers, their test statements may significantly deviate from actual performance, resulting in inaccurate evaluation results. Summary of the Invention
[0003] The main objective of this invention is to provide a model evaluation method, apparatus, device, and storage medium, aiming to solve the problem of poor accuracy in existing model evaluation methods.
[0004] To achieve the above objectives, the present invention provides a model evaluation method, the model evaluation method comprising:
[0005] Obtain the test statement and its corresponding original intent, and input the test statement into the model under test to obtain the first predicted intent output by the model under test, so as to obtain the first evaluation result based on the first predicted intent;
[0006] If the first evaluation result does not meet the preset conditions, a negative example statement is determined from the test statement based on the original intent and the first predicted intent.
[0007] Obtain the training statements of the model under test, and determine the target statements based on the training statements and the negative example statements;
[0008] Update the test statement according to the target statement;
[0009] The updated test statement is input into the model under test to obtain the second prediction intent output by the model under test, and a second evaluation result is obtained based on the second prediction intent.
[0010] Optionally, the step of determining negative example statements in the test statements based on the original intent and the first predicted intent includes:
[0011] By comparing the original intent with the first predicted intent, negative example predicted intents that are inconsistent with the original intent are identified.
[0012] Based on the negative example prediction intent, the corresponding negative example statement is determined in the test statement.
[0013] Optionally, the target statement includes a target exclusion statement;
[0014] The step of determining the target statement based on the training statement and the negative example statement includes:
[0015] Obtain the text similarity between the negative example statement and the training statement, and determine the maximum similarity based on the text similarity;
[0016] If the maximum similarity is less than a first preset threshold, the negative example statement is identified as the target removal statement.
[0017] Optionally, the training statement includes a first statement and a second statement, the first statement and the second statement having corresponding statement intents, and the target statement includes a target elimination statement;
[0018] The step of determining the target statement based on the training statement and the negative example statement includes:
[0019] Obtain the first text similarity between the negative example statement and the first statement, and the second text similarity between the negative example statement and the first statement, and obtain the similarity difference between the first text similarity and the second text similarity;
[0020] If the similarity difference is less than a second preset threshold and the intent of the first statement and the second statement are inconsistent, the negative example statement will be identified as the target removal statement.
[0021] Optionally, the step of updating the test statement according to the target statement includes:
[0022] Remove the target removal statement from the test statement.
[0023] Optionally, the training statement includes a third statement, the third statement having a corresponding statement intent, and the target statement includes a misjudged statement;
[0024] The step of obtaining the training statements of the model under test and determining the target statements based on the training statements and the negative example statements further includes:
[0025] Obtain the text similarity between the negative example statement and the third statement;
[0026] If the text similarity is less than a third preset threshold and the original intent of the negative example statement is inconsistent with the intent of the third statement, the negative example statement is determined to be a misjudged statement.
[0027] The step of updating the test statement according to the target statement includes:
[0028] Adjust the misjudged statement to a positive example statement, and retain the positive example statement.
[0029] Furthermore, to achieve the above objectives, the present invention also provides a model evaluation device, the model evaluation device comprising:
[0030] The first evaluation module obtains the test statement and its corresponding original intent, and inputs the test statement into the model under test to obtain the first predicted intent output by the model under test, so as to obtain the first evaluation result based on the first predicted intent;
[0031] The first determining module determines a negative example statement from the test statement based on the original intent and the first predicted intent when the first evaluation result does not meet the preset conditions.
[0032] The second determining module acquires the training statements of the model to be tested, and determines the target statement based on the training statements and the negative example statements;
[0033] The statement update module updates the test statement according to the target statement;
[0034] The second evaluation module inputs the updated test statement into the model under test to obtain the second prediction intent output by the model under test, so as to obtain the second evaluation result based on the second prediction intent.
[0035] In addition, to achieve the above objectives, the present invention also provides a model evaluation device, the model evaluation device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model evaluation method as described above.
[0036] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the model evaluation method as described above.
[0037] This invention provides a model evaluation method, apparatus, device, and storage medium. It involves acquiring a test statement and its corresponding original intent, inputting the test statement into a model under test to obtain a first predicted intent output by the model, and then obtaining a first evaluation result based on the first predicted intent. If the first evaluation result does not meet preset conditions, a negative example statement is determined from the test statement based on the original intent and the first predicted intent. The training statement of the model under test is acquired, and a target statement is determined based on the training statement and the negative example statement. The test statement is updated based on the target statement. The updated test statement is input into the model under test to obtain a second predicted intent output by the model, and then a second evaluation result is obtained based on the second predicted intent. Through this method, after the initial model evaluation, this embodiment utilizes an automated process to adjust and update the test data, and then uses the updated test data for a second evaluation to obtain a second evaluation result. This achieves automatic correction of the evaluation result, which is beneficial for improving the accuracy of model evaluation, while saving the manpower costs required for evaluation and improving evaluation efficiency. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention;
[0039] Figure 2 This is a flowchart illustrating the first embodiment of the model evaluation method of the present invention;
[0040] Figure 3 This is a schematic diagram of the functional modules of the first embodiment of the model evaluation device of the present invention.
[0041] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0042] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0043] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.
[0044] The model evaluation device in this embodiment of the invention can be a server, or a terminal device such as a PC (Personal Computer), tablet computer, portable computer, or mobile terminal.
[0045] like Figure 1As shown, the model evaluation device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0046] Those skilled in the art will understand that Figure 1 The model evaluation equipment structure shown does not constitute a limitation on the model evaluation equipment. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0047] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and computer programs.
[0048] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and / or database and to communicate with the backend server and / or database; the user interface 1003 is mainly used to connect to the client and to communicate with the client; and the processor 1001 can be used to call the computer program stored in the memory 1005 and execute various embodiments of the model evaluation method of the present invention.
[0049] Based on the above hardware structure, various embodiments of the model evaluation method of the present invention are proposed.
[0050] This invention provides a model evaluation method.
[0051] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the model evaluation method of the present invention.
[0052] In this embodiment, the model evaluation method includes:
[0053] Step S10: Obtain the test statement and its corresponding original intent, and input the test statement into the model under test to obtain the first predicted intent output by the model under test, so as to obtain the first evaluation result based on the predicted intent;
[0054] Intent recognition, also known as intent detection, categorizes sentences into different classes, falling under the category of multi-class classification. In intent recognition tasks, machine learning is typically used to build a classification model. Testers then simulate real-world user statements (i.e., test data) to test the model, and the model's deployment is determined based on the test results. However, testers' subjective biases can lead to significant discrepancies between the evaluation results and actual performance, resulting in poor accuracy. Current methods address this by manually correcting the test data using specialized testers, but this is costly, inefficient, and susceptible to human subjectivity, further compromising accuracy. This embodiment proposes a model evaluation method that, after the initial model evaluation, uses an automated process to adjust and update the test data. A second evaluation is then conducted using the updated data, yielding a second evaluation result. This automatic correction of the evaluation results improves accuracy, saves manpower, and increases efficiency.
[0055] The model evaluation method in this embodiment is implemented by a model evaluation device, which can be a server, or a terminal device such as a PC (Personal Computer), tablet computer, portable computer, or mobile terminal. The model to be tested is an intent recognition model, which is a classification model constructed through machine learning.
[0056] In this embodiment, the test statements and their corresponding original intents are first obtained. These test statements are written by testers based on the original intents; that is, each test statement has a corresponding original intent. For example, if the original intent is "navigation," a test statement written would be "go to XX building." Of course, test statements can also be automatically generated through relevant models, and this embodiment does not impose any restrictions.
[0057] After obtaining the test statement, it can be input into the model under test. The model analyzes the test statement and outputs a prediction result. The prediction result includes a first predictive intent, which can be considered the model's classification of the test statement. It should be noted that the first predictive intent corresponds one-to-one with each test statement; that is, each test statement has a corresponding first predictive intent. The prediction result may also include the prediction time, i.e., how long the model takes to classify the test statement.
[0058] After obtaining the first predicted intent, the first predicted intent of each test statement can be compared with the original intent to determine whether the model under test has correctly classified the test statement. For example, if a test statement C0 is written based on the original intent T0, and the first predicted intent output by the model under test for this test statement is T1, if the original intent T0 and the first predicted intent T1 are consistent, then the model under test can be considered to have correctly classified the test statement C0; if the original intent T0 and the first predicted intent T1 are inconsistent, then the model under test can be considered to have incorrectly classified the test statement C0. Then, the first evaluation result of the model under test can be obtained based on the first predicted intent. This first evaluation result can include the classification accuracy of the test statements, which is calculated by counting the number of accurately classified test statements and then using the ratio of this number to the total number of test statements as the classification accuracy. Of course, this first evaluation result can also include other evaluation metrics, such as average prediction time (total prediction time / total number of test statements), etc.
[0059] Step S20: If the first evaluation result does not meet the preset conditions, determine the negative example statement in the test statement according to the original intent and the first predicted intent;
[0060] In this embodiment, after obtaining the first evaluation result, it can be determined whether the first evaluation result meets a preset condition, which can be set according to the evaluation index. Specifically, when the first evaluation result includes the classification accuracy of the test statement, the preset condition is that the classification accuracy is greater than the accuracy threshold. If the classification accuracy is less than or equal to the accuracy threshold, the first evaluation result is considered not to meet the preset condition, and further analysis of the test situation is needed to investigate whether the inaccurate evaluation result may be due to problems with the test data. If the classification accuracy is greater than the accuracy threshold, the first evaluation result is considered to meet the preset condition, and the test process can be terminated.
[0061] In this embodiment, if the first evaluation result does not meet the preset conditions, a bad case statement will be determined in the test statement based on the original intent and the first predicted intent.
[0062] Specifically, the original intent and the first predicted intent can be compared to determine negative example predicted intents that are inconsistent with the original intent; and based on the negative example predicted intents, the corresponding negative example statements can be determined in the test statements.
[0063] In this embodiment, negative example statements can be defined by comparing the original intent of the test statement with the first predicted intent. If the first predicted intent of the original intent is inconsistent with the first predicted intent, then the first predicted intent can be considered a negative example predicted intent, and the test statement corresponding to the negative example predicted intent is a negative example statement (i.e., a test statement that is incorrectly classified). For example, if a test statement C0 is written based on the original intent T0, and the first predicted intent output by the model under test for this test statement is T1, then if the original intent T0 and the first predicted intent T1 are inconsistent, then the first predicted intent T1 can be considered a negative example predicted intent, and the corresponding test statement C0 is a negative example statement.
[0064] Step S30: Obtain the training statement of the model to be tested, and determine the target statement based on the training statement and the negative example statement;
[0065] In this embodiment, the training statements of the model under test will also be obtained. These training statements are those used to train the model in the early stages. The training statements can be compared with negative example statements, and the target statement can be determined from the negative example statements based on the comparison results. The target statement can be considered as a statement that has a negative impact on the model evaluation.
[0066] In some implementations, the target statement includes a target exclusion statement, and step S30 includes:
[0067] Step a31: Obtain the text similarity between the negative example statement and the training statement, and determine the maximum similarity based on the text similarity;
[0068] The target statement can include the target elimination statement. Based on the negative example statement and the training statement, the text similarity between the negative example statement and each training statement can be calculated, and the maximum similarity can be determined based on the text similarity; this text similarity can be represented by methods such as cosine similarity. For example, the training statements include the first statement C1 and the second statement C2. For the negative example statement C0, the text similarity X01 between the negative example statement C0 and the first statement C1, and the text similarity X02 between the negative example statement C0 and the second statement C2 can be calculated; among these text similarities, X02 has the largest value, so X02 can be determined as the maximum similarity.
[0069] Step a32: If the maximum similarity is less than a first preset threshold, the negative example statement is identified as the target removal statement.
[0070] After determining the maximum similarity, this maximum similarity is compared with a first preset threshold. If the maximum similarity is less than the first preset threshold, then the negative example statement C0 is considered dissimilar to all training statements, meaning that the negative example statement C0 is a non-business statement (unrelated to business). This non-business statement will have an adverse impact on model evaluation, and in this case, the negative example statement can be identified as a target for removal. This target statement can then be removed from the negative example statements in subsequent iterations.
[0071] By using the above method, non-business statements can be filtered out from negative examples and removed as target statements, thus facilitating subsequent adjustments and updates to negative examples (or test statements).
[0072] In some implementations, the training statements include a first statement and a second statement, the first statement and the second statement having corresponding statement intents, and the target statements include target elimination statements. Step S30 includes:
[0073] Step a33: Obtain the first text similarity between the negative example statement and the first statement, and the second text similarity between the negative example statement and the first statement, and obtain the similarity difference between the first text similarity and the second text similarity;
[0074] The target statement can include the target elimination statement. Based on the negative example statement and the training statement, the text similarity between the negative example statement and each training statement can be calculated (characterized by methods such as cosine similarity). Then, the similarity difference between the two text similarities is determined by pairwise comparison. Specifically, the training statements include the first statement C1 and the second statement C2, which have corresponding statement intentions. For the negative example statement C0, the text similarity X01 between the negative example statement C0 and the first statement C1, and the text similarity X02 between the negative example statement C0 and the second statement C2 can be calculated. Then, the similarity difference X0102 between the text similarity X01 and the text similarity X02 is calculated.
[0075] Step a34: If the similarity difference is less than the second preset threshold and the intent of the first statement and the second statement are inconsistent, the negative example statement is identified as the target removal statement.
[0076] If the calculated similarity difference X0102 is less than the second preset threshold, and the first statement C1 and the second statement C2 correspond to different statement intentions (inconsistent intention types), then the negative example statement C0 can be considered to have a similar matching degree with multiple statement intention types. That is, the negative example statement C0 is a multi-intention statement, which will have an adverse effect on the model evaluation. In this case, the negative example statement can be identified as a target elimination statement. This target elimination statement can then be removed from the negative example statements.
[0077] By using the above method, multi-intent statements can be filtered out from negative examples and used as target statements for elimination, which facilitates subsequent adjustments and updates to negative examples (or test statements).
[0078] In some implementations, the training statement includes a third statement, which has a corresponding statement intent, and the target statement includes a misjudged statement. Step S30 includes:
[0079] Step a35: Obtain the text similarity between the negative example statement and the third statement;
[0080] For negative example statements, misjudgments may occur. For instance, a test statement written according to intent A might have the original intent A, but due to writing errors or other reasons, it might actually correspond to intent B. In this case, the test statement is considered a misjudged statement. To address this, misjudged statements can be identified from the negative example statements and adjusted accordingly. Specifically, the training statements include a third statement C3, which has a corresponding statement intent. For negative example statement C0, the text similarity X03 between negative example statement C0 and the third statement C3 can be calculated using methods such as text similarity.
[0081] Step a36: If the text similarity is less than a third preset threshold and the original intent of the negative example statement is inconsistent with the intent of the third statement, then the negative example statement is determined to be a misjudged statement.
[0082] If the calculated text similarity X03 is less than the third preset threshold, and the original intent of the negative example statement is inconsistent with the intent of the third statement C3 (corresponding to different intent types), then the negative example statement C0 can be considered a misjudgment. In this case, the negative example statement can be identified as a misjudged statement. This misjudged statement can then be adjusted from a negative example statement to a positive example statement and retained in the test statements. Furthermore, the original intent of the misjudged statement can also be adjusted.
[0083] By using the above methods, misjudged statements can be filtered out from negative example statements, which facilitates subsequent adjustments and updates to negative example statements (or test statements).
[0084] Step S40: Update the test statement according to the target statement;
[0085] In this embodiment, after the target statement is determined, the test statement can be updated based on the target statement to obtain the updated test statement. Specifically, the target removal statement in the target statement can be deleted from the test statement because it will have an adverse effect on the model evaluation; while the misjudged statement in the target statement can be changed from a negative example statement to a positive example statement and retained in the test statement. Of course, the original intent of the misjudged statement can also be adjusted.
[0086] Step S50: Input the updated test statement into the model under test to obtain the second prediction intent output by the model under test, so as to obtain the second evaluation result based on the second prediction intent.
[0087] In this embodiment, after obtaining the updated test statement, the updated test statement can be input into the model under test, analyzed by the model under test, and the prediction result output by the model under test can be obtained. The prediction result includes the second prediction intent. Then, the second evaluation result of the model under test can be obtained based on the second prediction intent, including indicators such as the classification accuracy of the test statement.
[0088] In this embodiment, by acquiring a test statement and its corresponding original intent, and inputting the test statement into the model under test, a first predicted intent output by the model under test is obtained, and a first evaluation result is obtained based on the first predicted intent. If the first evaluation result does not meet preset conditions, a negative example statement is determined from the test statement based on the original intent and the first predicted intent. The training statement of the model under test is acquired, and a target statement is determined based on the training statement and the negative example statement. The test statement is updated based on the target statement. The updated test statement is input into the model under test, and a second predicted intent output by the model under test is obtained, and a second evaluation result is obtained based on the second predicted intent. Through this method, after the initial model evaluation, this embodiment uses an automated process to adjust and update the test data, and then uses the updated test data for a second evaluation, thereby obtaining a second evaluation result. This achieves automatic correction of the evaluation result, which is beneficial for improving the accuracy of model evaluation, while saving the manpower costs required for evaluation and improving evaluation efficiency.
[0089] The present invention also provides a model evaluation device.
[0090] Reference Figure 3 , Figure 3 This is a schematic diagram of the functional modules of the first embodiment of the model evaluation device of the present invention.
[0091] like Figure 3 As shown, the model evaluation device includes:
[0092] The first evaluation module 10 acquires the test statement and its corresponding original intent, and inputs the test statement into the model under test to obtain the first predicted intent output by the model under test, so as to obtain the first evaluation result based on the first predicted intent;
[0093] The first determining module 20 determines a negative example statement from the test statement based on the original intent and the first predicted intent when the first evaluation result does not meet the preset conditions.
[0094] The second determining module 30 acquires the training statements of the model to be tested, and determines the target statement based on the training statements and the negative example statements;
[0095] Statement update module 40 updates the test statement according to the target statement;
[0096] The second evaluation module 50 inputs the updated test statement into the model under test to obtain the second prediction intent output by the model under test, so as to obtain the second evaluation result based on the second prediction intent.
[0097] Furthermore, the first determining module 20 is specifically used for:
[0098] By comparing the original intent and the first predicted intent, negative example predicted intents that are inconsistent with the original intent are determined; based on the negative example predicted intents, the corresponding negative example statements are determined in the test statements.
[0099] Furthermore, the target statement includes a target elimination statement, and the second determining module 30 includes:
[0100] The first acquisition unit is used to acquire the text similarity between the negative example statement and the training statement, and to determine the maximum similarity based on the text similarity.
[0101] The first determining unit is used to determine the negative example statement as the target removal statement when the maximum similarity is less than a first preset threshold.
[0102] Furthermore, the training statements include a first statement and a second statement, the first statement and the second statement having corresponding statement intentions, and the second determining module 30 includes:
[0103] The second acquisition unit is used to acquire the first text similarity between the negative example statement and the first statement, and the second text similarity between the negative example statement and the first statement, and to acquire the similarity difference between the first text similarity and the second text similarity.
[0104] The second determining unit is used to determine the negative example statement as the target removal statement when the similarity difference is less than a second preset threshold and the statement intents of the first statement and the second statement are inconsistent.
[0105] Furthermore, the statement update module 40 is specifically used to remove the target removal statement from the test statement.
[0106] Furthermore, the training statement includes a third statement, the third statement having a corresponding statement intent, the target statement includes a misjudged statement, and the second determining module 30 further includes:
[0107] The third acquisition unit is used to acquire the text similarity between the negative example statement and the third statement;
[0108] The third determining unit is used to determine that the negative example statement is a misjudged statement when the text similarity is less than a third preset threshold and the original intent of the negative example statement is inconsistent with the statement intent of the third statement.
[0109] The statement update module 40 is specifically used to adjust the misjudged statement into a positive example statement and retain the positive example statement.
[0110] The functions of each module in the above-mentioned model evaluation device correspond to the steps in the above-mentioned model evaluation method embodiment, and their functions and implementation processes will not be described in detail here.
[0111] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the model evaluation method as described in any of the above embodiments.
[0112] The specific embodiments of the computer-readable storage medium of the present invention are basically the same as the embodiments of the above model evaluation method, and will not be described in detail here.
[0113] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the model evaluation method as described in any of the preceding claims.
[0114] The specific embodiments of the computer program product of the present invention are basically the same as the embodiments of the above model evaluation method, and will not be described in detail here.
[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0116] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0118] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A model evaluation method, characterized in that, The model evaluation method includes: Obtain the test statement and its corresponding original intent, and input the test statement into the model under test to obtain the first predicted intent output by the model under test, so as to obtain the first evaluation result based on the first predicted intent; If the first evaluation result does not meet the preset conditions, a negative example statement is determined from the test statement based on the original intent and the first predicted intent. Obtain the training statements of the model under test, and determine the target statements based on the training statements and the negative example statements; Update the test statement according to the target statement; The updated test statement is input into the model under test to obtain the second prediction intent output by the model under test, and a second evaluation result is obtained based on the second prediction intent; the target statement includes a target elimination statement. The step of determining the target statement based on the training statement and the negative example statement includes: Obtain the text similarity between the negative example statement and the training statement, and determine the maximum similarity based on the text similarity. If the maximum similarity is less than a first preset threshold, the negative example statement is identified as the target removal statement.
2. The model evaluation method as described in claim 1, characterized in that the step of determining negative example statements in the test statements based on the original intent and the first prediction intent includes: By comparing the original intent with the first predicted intent, negative example predicted intents that are inconsistent with the original intent are identified. Based on the negative example prediction intent, the corresponding negative example statement is determined in the test statement.
3. The model evaluation method as described in claim 1, characterized in that the training statement includes a first statement and a second statement, the first statement and the second statement have corresponding statement intentions, and the target statement includes a target elimination statement; The step of determining the target statement based on the training statement and the negative example statement includes: Obtain the first text similarity between the negative example statement and the first statement, and the second text similarity between the negative example statement and the first statement, and obtain the similarity difference between the first text similarity and the second text similarity; If the similarity difference is less than a second preset threshold and the intent of the first statement and the second statement are inconsistent, the negative example statement will be identified as the target removal statement.
4. The model evaluation method as described in claim 3, characterized in that the step of updating the test statement according to the target statement includes: Remove the target removal statement from the test statement.
5. The model evaluation method as described in claim 1, characterized in that, The training statement includes a third statement, which has a corresponding statement intent, and the target statement includes a misjudged statement. The step of obtaining the training statements of the model under test and determining the target statements based on the training statements and the negative example statements further includes: Obtain the text similarity between the negative example statement and the third statement; If the text similarity is less than a third preset threshold and the original intent of the negative example statement is inconsistent with the intent of the third statement, the negative example statement is determined to be a misjudged statement. The step of updating the test statement according to the target statement includes: Adjust the misjudged statement to a positive example statement, and retain the positive example statement.
6. A model evaluation device, characterized in that, The model evaluation device includes: The first evaluation module obtains the test statement and its corresponding original intent, and inputs the test statement into the model under test to obtain the first predicted intent output by the model under test, so as to obtain the first evaluation result based on the first predicted intent; The first determining module determines a negative example statement from the test statement based on the original intent and the first predicted intent when the first evaluation result does not meet the preset conditions. The second determining module acquires the training statements of the model to be tested, and determines the target statement based on the training statements and the negative example statements; The statement update module updates the test statement according to the target statement; The second evaluation module inputs the updated test statement into the model under test to obtain the second prediction intent output by the model under test, and obtains the second evaluation result based on the second prediction intent; the target statement includes a target elimination statement; The step of determining the target statement based on the training statement and the negative example statement includes: Obtain the text similarity between the negative example statement and the training statement, and determine the maximum similarity based on the text similarity. If the maximum similarity is less than a first preset threshold, the negative example statement is identified as the target removal statement.
7. A model evaluation device, characterized in that, The model evaluation device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the model evaluation method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the model evaluation method as described in any one of claims 1 to 5.