Evaluation method, device and network equipment for natural language understanding system

Abstract

Embodiments of the present invention provide a method, apparatus, and network device for evaluating a natural language understanding system, relating to the field of parking navigation technology. The method includes: obtaining an evaluation data set; processing the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result; comparing the prediction result with a preset result to obtain a target result, wherein the target result includes at least one of the following: recall rate, overall coverage, and error results in the prediction result; when the target result includes error results, determining a first number of target error results from the error results, and obtaining a second number of expanded error results corresponding to each target error result; evaluating the NLU system using the target results, the first number, and the second number to obtain an evaluation result. The above scheme comprehensively evaluates the NLU system using multiple indicators such as the target result, the first number, and the second number, making the evaluation result more comprehensive and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of system evaluation, and in particular to an evaluation method, device and network equipment for a natural language understanding system. Background Art

[0002] The NLU system is a complex system, with key modules focusing on deep learning tasks such as intent recognition, named entity recognition, and knowledge bases. Furthermore, the rule engine and blacklist and whitelist functions are also crucial components. During its implementation, evaluation and testing remain particularly important, complex, and demanding.

[0003] The current main method, borrowed from the research field (academia), focuses on evaluating a system's deep learning model. Specifically, before model training, the dataset is divided into training, validation, and test sets according to a certain ratio. The training set is used for model training, the validation set is used to verify the model's effectiveness, and the test set is used to test and evaluate the model's performance. However, this evaluation method is incomplete for NLU systems and has low accuracy. Summary of the Invention

[0004] The present invention provides a method, apparatus, and network device for evaluating a natural language understanding system, so as to, to some extent, solve the problem that existing evaluation methods for NLU systems are incomplete and have low accuracy.

[0005] In a first aspect of the present invention, a method for evaluating a natural language understanding system is provided, the method comprising:

[0006] Get the evaluation dataset;

[0007] Processing the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result;

[0008] Comparing the predicted result with the preset result to obtain a target result, wherein the target result includes at least one of the following: precision recall rate, overall coverage rate, and error results in the predicted result;

[0009] In a case where the target result includes the error result, determining a first number of target error results from the error results, and obtaining a second number of expanded error results corresponding to each target error result;

[0010] The NLU system is evaluated using the target result, the first quantity, and the second quantity to obtain an evaluation result.

[0011] Optionally, the NLU system includes a prediction model, and the prediction result includes: a first prediction result;

[0012] The evaluation data set is processed by a natural language understanding (NLU) system to obtain a prediction result, including:

[0013] Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result;

[0014] The step of comparing the predicted result with the preset result to obtain the target result includes:

[0015] The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

[0016] Optionally, the prediction result includes: a second prediction result;

[0017] The evaluation data set is processed by a natural language understanding (NLU) system to obtain a prediction result, including:

[0018] Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result;

[0019] The step of comparing the predicted result with the preset result to obtain the target result includes:

[0020] The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

[0021] Optionally, when the target result includes the error result, determining a first number of target error results from the error results, and obtaining a second number of expanded error results corresponding to each target error result, includes:

[0022] In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results;

[0023] The target keyword in each of the target error results is replaced with similar words to obtain a second number of expanded error results corresponding to each of the target error results.

[0024] Optionally, methods for determining target error results that are of high priority include:

[0025] If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

[0026] In a second aspect of the present invention, there is provided an evaluation device for a natural language understanding system, the device comprising:

[0027] A first acquisition module is used to acquire an evaluation data set;

[0028] A first processing module is used to process the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result;

[0029] A first comparison module is configured to compare the prediction result with a preset result to obtain a target result, wherein the target result includes at least one of the following: a precision recall rate, an overall coverage rate, and an error result in the prediction result;

[0030] a second processing module, configured to, when the target result includes the error result, determine a first number of target error results from the error results, and obtain a second number of expanded error results corresponding to each target error result;

[0031] A first evaluation module is used to evaluate the NLU system based on the target result, the first quantity, and the second quantity to obtain an evaluation result.

[0032] Optionally, the NLU system includes a prediction model, and the prediction result includes: a first prediction result;

[0033] The first processing module is specifically configured to:

[0034] Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result;

[0035] The first comparison module is specifically configured to:

[0036] The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

[0037] Optionally, the prediction result includes: a second prediction result;

[0038] The first processing module is specifically configured to:

[0039] Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result;

[0040] The first comparison module is specifically configured to:

[0041] The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

[0042] Optionally, the second processing module is specifically configured to:

[0043] In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results;

[0044] The target keyword in each of the target error results is replaced with similar words to obtain a second number of expanded error results corresponding to each of the target error results.

[0045] Optionally, methods for determining target error results that are of high priority include:

[0046] If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

[0047] In a third aspect of the present invention, a network device is provided, including a memory, a transceiver, and a processor:

[0048] A memory for storing a computer program; a transceiver for transmitting and receiving data under the control of the processor; and a processor for reading the computer program in the memory and executing the steps in the evaluation method of the natural language understanding system as described above.

[0049] In a fourth aspect of the present invention, a processor-readable storage medium is provided, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the evaluation method of the natural language understanding system as described above.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] In the above embodiment of the present invention, the evaluation data set is processed by the NLU system to obtain a prediction result, and the prediction result is compared with the preset result to obtain a target result of at least one of the recall rate, overall coverage rate, and error results in the prediction result. When the target result includes the error result, the first number of target error results is determined from the error result, and the second number of expanded error results corresponding to each target error result is obtained. The NLU system is comprehensively evaluated through multiple indicators such as the above target results, the first number, and the second number, so that the evaluation result is more comprehensive and accurate.

[0052] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments.

[0054] Figure 1 A flowchart of a method for evaluating a natural language understanding system according to an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a specific process of the evaluation method of the natural language understanding system provided by an embodiment of the present invention;

[0056] Figure 3 A block diagram of an evaluation device for a natural language understanding system provided by an embodiment of the present invention;

[0057] Figure 4 A block diagram of a network device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0060] Currently, existing evaluation methods for natural language understanding (NLU) systems primarily use two metrics: precision and recall. Alternatively, the harmonic mean score (HMS) metric, a combination of the two, can be used. Precision and recall are excellent indicators for assessing a model's generalization capabilities. However, these evaluation methods still face many challenges in business implementation:

[0061] First, there's a mismatch between evaluation subjects: Academic research focuses on the model itself, with the task itself focusing on its structure and performance. However, industrial implementation delivers a system, the result of an organic integration of multiple technologies. Evaluating only the model is inaccurate and imprecise.

[0062] Second, the issue of business coverage assessment: During the implementation process in the industrial sector, there are generally requirements for business coverage, but traditional methods only care about the readiness rate and not the coverage rate.

[0063] Third, there's the issue of priority: During industrial implementation, case priorities within a dataset vary. Some unimportant cases can be omitted, while others must be supported. Traditional approaches assume that all cases have the same priority. This problem is particularly acute during model iteration. Traditional evaluation methods primarily focus on model capability metrics (categorical accuracy and generalization), but these metrics cannot directly guide model iteration. In practice, a model may have excellent metrics but fail some high-priority cases.

[0064] Fourth, the problem of severity determination: If a bad case is found during testing and evaluation, the severity of the problem needs to be determined, but traditional methods cannot provide a corresponding judgment method.

[0065] Therefore, the embodiments of the present invention provide a method, apparatus, and network device for evaluating a natural language understanding system, which can comprehensively evaluate the NLU system through multiple indicators, making the evaluation results more comprehensive and accurate.

[0066] Specifically, such as Figure 1 As shown, an embodiment of the present invention provides a method for evaluating a natural language understanding system, which may specifically include the following steps:

[0067] Step 101: Obtain an evaluation dataset.

[0068] Specifically, we first collect a set of evaluation data for evaluating the NLU system, namely the evaluation dataset. This evaluation dataset can be divided into a training set, a validation set, and a test set according to a certain ratio as needed. The training set is used to train the prediction model in the NLU system, the validation set is used to verify the effectiveness of the prediction model, and finally the test set is used to predict and evaluate the performance of the model.

[0069] The evaluation dataset can be divided into a training set, a validation set, and a test set in a ratio of 8:1:1 as needed, and the division ratio is not limited here.

[0070] Step 102: Process the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result.

[0071] Specifically, after obtaining the evaluation dataset, the evaluation dataset is input into the NLU system, and after being processed by the NLU system, a prediction result is output. For example, if the evaluation dataset is a navigation dataset, the output prediction result is a navigation identifier.

[0072] Step 103 : Compare the predicted result with the preset result to obtain a target result, wherein the target result includes at least one of the following: precision recall rate, overall coverage rate, and error results in the predicted result.

[0073] Specifically, the prediction results output by the NLU system are compared with the pre-set preset results for consistency, and the recall rate can be obtained through the consistency comparison; or, the prediction results output by the NLU system are compared with the pre-set preset results for consistency, and the overall coverage rate can be obtained through the consistency comparison; or, the prediction results output by the NLU system are compared with the pre-set preset results for consistency, and the prediction results can be obtained that are different from the corresponding preset results, that is, wrong results, and the number of wrong results is at least one.

[0074] Among them, the recall rate is used to determine the accuracy and generalization ability of the NLU system in a certain category (such as navigation).

[0075] The overall coverage rate reflects the coverage of the business requirements by the NLU system as a whole. That is, the evaluation dataset can be used as a standard to reflect the business scope. This indicator is used to evaluate the overall coverage of the NLU system for business requirements.

[0076] The error results in the prediction results reflect the priority of the coverage. In the industry, this indicator can evaluate the coverage of all high-priority error results by the NLU system.

[0077] It should be noted that the target result is not limited to at least one of the precision rate, the overall coverage rate, and the error results in the prediction results, and the above indicators can be increased or decreased as needed.

[0078] Step 104 : When the target result includes the error result, determine a first number of target error results from the error results, and obtain a second number of expanded error results corresponding to each target error result.

[0079] Specifically, if the target results include error results, the target error results need to be determined from the error results, thereby obtaining a first number of target error results. Furthermore, based on the target error results, the expandable expanded error results corresponding to each target error result are obtained, thereby obtaining a second number of expanded error results corresponding to each target error result. Furthermore, the total number of all expanded error results corresponding to all target error results can be obtained.

[0080] Step 105: Evaluate the NLU system based on the target result, the first quantity, and the second quantity to obtain an evaluation result.

[0081] Specifically, the NLU system is evaluated in multiple aspects through multiple indicators such as the above-mentioned target results, the first quantity, the second quantity, etc., to obtain a more comprehensive and accurate evaluation result.

[0082] In the above embodiment of the present invention, the evaluation data set is processed by the NLU system to obtain a prediction result, and the prediction result is compared with the preset result to obtain a target result of at least one of the recall rate, overall coverage rate, and error results in the prediction result. When the target result includes the error result, the first number of target error results is determined from the error result, and the second number of expanded error results corresponding to each target error result is obtained. The NLU system is comprehensively evaluated through multiple indicators such as the above target results, the first number, and the second number, so that the evaluation result is more comprehensive and accurate.

[0083] As an optional embodiment, the NLU system includes a prediction model, and the prediction result includes: a first prediction result;

[0084] Step 102 processes the evaluation dataset through a natural language understanding (NLU) system to obtain a prediction result, which may specifically include:

[0085] Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result;

[0086] The step 103 compares the predicted result with the preset result to obtain the target result, which may specifically include:

[0087] The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

[0088] Specifically, after obtaining the evaluation dataset, a test dataset is generated based on the partitioning of the evaluation dataset. The test dataset is then fed into the prediction model within the NLU system, where it is processed and a first prediction result is output. This first prediction result is then compared with a pre-set result for consistency. This consistency comparison yields two evaluation metrics: precision and recall.

[0089] As an optional embodiment, the prediction result includes: a second prediction result;

[0090] Step 102 processes the evaluation dataset through a natural language understanding (NLU) system to obtain a prediction result, which may specifically include:

[0091] Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result;

[0092] The step 103 compares the predicted result with the preset result to obtain the target result, which may specifically include:

[0093] The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

[0094] Specifically, after obtaining the evaluation data set, the evaluation data set is not divided, and the entire evaluation data set is input into the complete NLU system. After processing by the NLU system, a second prediction result is output; then the second prediction result output by the NLU system is compared with the pre-set preset result for consistency. After the consistency comparison, a result that is different from the corresponding preset result in the prediction result, that is, an error result, can be obtained; or, the second prediction result output by the NLU system is compared with the pre-set preset result for consistency. After the consistency comparison, the overall coverage rate can be obtained.

[0095] For example: if the second prediction result is "I want to go to the first destination" and the preset result is "I want to go to the second destination", if the first destination and the second destination are not the same place, the second prediction result is different from the preset result, and the second prediction result is an incorrect result.

[0096] As an optional embodiment, in step 104, when the target result includes the error result, determining a first number of target error results from the error results and obtaining a second number of expanded error results corresponding to each target error result may specifically include:

[0097] In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results;

[0098] The target keyword in each of the target error results is replaced with similar words to obtain a second number of expanded error results corresponding to each of the target error results.

[0099] Specifically, if the target results include error results, it is necessary to determine the target error results of high priority from the error results, thereby obtaining a first number of target error results. Furthermore, the target keyword in each target error result is replaced with a similar character, thereby obtaining an expanded error result corresponding to each target error result, i.e., the expanded error result after the target keyword is replaced with a similar character. This results in a second number of expanded error results corresponding to each target error result. Furthermore, the total number of all expanded error results corresponding to all target error results can be obtained. The total number of expanded error results can be used to assess the severity of the NLU system defects.

[0100] For example: if the target error result is "I want to go to the first destination", and the target keyword of the target error result is the first destination, then the first destination can be replaced with similar words, that is, the first destination can be replaced with other destinations, such as the second destination, the third destination, the fourth destination, etc., that is, the expanded error results corresponding to the target error result include: "I want to go to the second destination", "I want to go to the third destination", "I want to go to the fourth destination".

[0101] As an optional embodiment, a method of determining target error results belonging to a high priority may include:

[0102] If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

[0103] Specifically, the similarity between each of the at least one error result and its corresponding preset result is obtained. If the similarity between the first error result and its corresponding preset result is greater than the preset similarity, the first error result is determined to be a high-priority target error result. Conversely, if the similarity between the first error result and its corresponding preset result is less than or equal to the preset similarity, the first error result is determined not to be a high-priority target error result. The preset similarity is a preset similarity limit used to determine whether the similarity between the error result and its corresponding preset result is a high-priority target error result.

[0104] The first error result only refers to any one error result among at least one error result, and does not specifically refer to any one error result.

[0105] like Figure 2 As shown, the above solution is described in detail below through a specific embodiment:

[0106] Step 201: Obtain an evaluation dataset.

[0107] Step 202: Split the evaluation data set into a test data set.

[0108] Step 203: Perform result prediction processing using the test data set based on the prediction model to obtain a first prediction result.

[0109] Step 204: Compare the first prediction result with the preset result to obtain the precision and recall rate.

[0110] Step 205: Perform result prediction processing based on the NLU system using the evaluation data set to obtain a second prediction result.

[0111] Step 206: Compare the second prediction result with the preset result to obtain the overall coverage.

[0112] Step 207: Compare the second prediction result with the preset result to obtain an error result.

[0113] Step 207: Determine a first number of target error results with high priority from the error results.

[0114] Step 208: Replace the target keyword in each target error result with a similar word to obtain a second number of expanded error results corresponding to each target error result.

[0115] Step 209: Evaluate the NLU system based on the recall rate, overall coverage, first quantity, and second quantity to obtain an evaluation result.

[0116] To summarize, in an embodiment of the present invention, a first prediction result is obtained by subjecting a test data set in an evaluation data set to a prediction model, and the first prediction result is obtained. The first prediction result is compared with the preset result to obtain the recall rate, and the evaluation data set is subjected to a result prediction process by an NLU system to obtain a second prediction result. The second prediction result is compared with the preset result to obtain the error results and the overall coverage rate in the prediction result. A first number of target error results with a high priority is determined from the error results, and the target keywords in each target error result are replaced with similar words to obtain a second number of expanded error results corresponding to each target error result. The total number of all expanded error results corresponding to all target error results can be further obtained. The NLU system is evaluated as a whole through multiple evaluation indicators such as the recall rate, the overall coverage rate, the first number, and the second number, so that the evaluation result is more comprehensive and accurate. Moreover, the severity of the defects of the NLU system can be evaluated through the total number of expanded error results.

[0117] like Figure 3 As shown, an evaluation device 300 for a natural language understanding system provided by an embodiment of the present invention includes:

[0118] A first acquisition module 301 is used to acquire an evaluation data set;

[0119] A first processing module 302 is configured to process the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result;

[0120] A first comparison module 303 is configured to compare the prediction result with a preset result to obtain a target result, wherein the target result includes at least one of the following: a precision recall rate, an overall coverage rate, and an error result in the prediction result;

[0121] A second processing module 304 is configured to, when the target result includes the error result, determine a first number of target error results from the error results, and obtain a second number of expanded error results corresponding to each target error result;

[0122] The first evaluation module 305 is configured to evaluate the NLU system based on the target result, the first quantity, and the second quantity to obtain an evaluation result.

[0123] Optionally, the NLU system includes a prediction model, and the prediction result includes: a first prediction result;

[0124] The first processing module 302 is specifically configured to:

[0125] Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result;

[0126] The first comparison module 303 is specifically configured to:

[0127] The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

[0128] Optionally, the prediction result includes: a second prediction result;

[0129] The first processing module 302 is specifically configured to:

[0130] Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result;

[0131] The first comparison module 303 is specifically configured to:

[0132] The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

[0133] Optionally, the second processing module 304 is specifically configured to:

[0134] In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results;

[0135] Replacing the target keyword in each target error result with a similar word to obtain an expanded error result corresponding to each target error result;

[0136] Get the second number of extended error results corresponding to all target error results.

[0137] Optionally, methods for determining target error results that are of high priority include:

[0138] If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

[0139] It should be noted that the embodiment of the evaluation device of the natural language understanding system is a device corresponding to the evaluation method of the above-mentioned natural language understanding system. All implementation methods of the above-mentioned method embodiment are applicable to the embodiment of the device and can achieve the same technical effects as thereof, which will not be repeated here.

[0140] To summarize, in an embodiment of the present invention, a first prediction result is obtained by subjecting a test data set in an evaluation data set to a prediction model, and the first prediction result is obtained. The first prediction result is compared with the preset result to obtain the recall rate, and the evaluation data set is subjected to a result prediction process by an NLU system to obtain a second prediction result. The second prediction result is compared with the preset result to obtain the error results and the overall coverage rate in the prediction result. A first number of target error results with a high priority is determined from the error results, and the target keywords in each target error result are replaced with similar words to obtain a second number of expanded error results corresponding to each target error result. The total number of all expanded error results corresponding to all target error results can be further obtained. The NLU system is evaluated as a whole through multiple evaluation indicators such as the recall rate, the overall coverage rate, the first number, and the second number, so that the evaluation result is more comprehensive and accurate. Moreover, the severity of the defects of the NLU system can be evaluated through the total number of expanded error results.

[0141] like Figure 4 As shown, an embodiment of the present invention further provides a network device, including a memory 420, a transceiver 410, and a processor 400:

[0142] Memory 420, for storing computer programs;

[0143] a transceiver 410, configured to transmit and receive data under the control of the processor;

[0144] The processor 400 is configured to read the computer program in the memory and perform the following operations:

[0145] Get the evaluation dataset;

[0146] Processing the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result;

[0147] Comparing the predicted result with the preset result to obtain a target result, wherein the target result includes at least one of the following: precision recall rate, overall coverage rate, and error results in the predicted result;

[0148] In a case where the target result includes the error result, determining a first number of target error results from the error results, and obtaining a second number of expanded error results corresponding to each target error result;

[0149] The NLU system is evaluated using the target result, the first quantity, and the second quantity to obtain an evaluation result.

[0150] Optionally, the NLU system includes a prediction model, and the prediction result includes: a first prediction result;

[0151] When the processor 400 processes the evaluation data set through the NLU system to obtain a prediction result, it is specifically configured to:

[0152] Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result;

[0153] When the processor 400 compares the predicted result with the preset result to obtain the target result, it is specifically configured to:

[0154] The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

[0155] Optionally, the prediction result includes: a second prediction result;

[0156] When the processor 400 processes the evaluation data set through the NLU system to obtain a prediction result, it is specifically configured to:

[0157] Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result;

[0158] When the processor 400 compares the predicted result with the preset result to obtain the target result, it is specifically configured to:

[0159] The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

[0160] Optionally, when the target result includes the error result, the processor 400 determines a first number of target error results from the error results and obtains a second number of extended error results corresponding to each target error result, specifically for:

[0161] In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results;

[0162] Replacing the target keyword in each target error result with a similar word to obtain an expanded error result corresponding to each target error result;

[0163] Get the second number of extended error results corresponding to all target error results.

[0164] Optionally, methods for determining target error results that are of high priority include:

[0165] If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

[0166] Among them, Figure 4 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 400 and memory represented by memory 420. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. The transceiver 410 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, such as a wireless channel, a wired channel, an optical cable, and the like. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 may store data used by the processor 400 when performing operations.

[0167] The processor 400 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0168] The processor calls the computer program stored in the memory to execute the evaluation method of any natural language understanding system provided in the embodiments of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0169] It should be noted here that the above-mentioned network device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned evaluation method embodiment of the natural language understanding system, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0170] An embodiment of the present invention further provides a processor-readable storage medium storing a computer program, wherein the computer program is used to enable the processor to execute the above-mentioned evaluation method of the natural language understanding system.

[0171] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASH), solid-state drives (SSDs)), etc.

[0172] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) that contain computer-usable program code.

[0173] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor-readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These processor-executable instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for evaluating a natural language understanding system, characterized in that: include: Get the evaluation dataset; Processing the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result; Comparing the predicted result with the preset result to obtain a target result, wherein the target result includes at least one of the following: precision recall rate, overall coverage rate, and error results in the predicted result; In a case where the target result includes the error result, determining a first number of target error results from the error results, and obtaining a second number of expanded error results corresponding to each target error result; Evaluate the NLU system using the target result, the first quantity, and the second quantity to obtain an evaluation result; In the case where the target result includes the error result, determining a first number of target error results from the error results, and obtaining a second number of expanded error results corresponding to each target error result, includes: In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results; The target keyword in each of the target error results is replaced with similar words to obtain a second number of expanded error results corresponding to each of the target error results.

2. The method according to claim 1, characterized in that The NLU system includes a prediction model, and the prediction results include: a first prediction result; The evaluation data set is processed by a natural language understanding (NLU) system to obtain a prediction result, including: Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result; The step of comparing the predicted result with the preset result to obtain the target result includes: The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

3. The method according to claim 1, characterized in that The prediction results include: a second prediction result; The evaluation data set is processed by a natural language understanding (NLU) system to obtain a prediction result, including: Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result; The step of comparing the predicted result with the preset result to obtain the target result includes: The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

4. The method according to claim 1, wherein Methods for determining target error results that are high priority include: If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

5. An evaluation device for a natural language understanding system, characterized in that: include: A first acquisition module is used to acquire an evaluation data set; A first processing module is used to process the evaluation data set through a natural language understanding (NLU) system to obtain a prediction result; A first comparison module is configured to compare the prediction result with a preset result to obtain a target result, wherein the target result includes at least one of the following: a precision recall rate, an overall coverage rate, and an error result in the prediction result; a second processing module, configured to, when the target result includes the error result, determine a first number of target error results from the error results, and obtain a second number of expanded error results corresponding to each target error result; a first evaluation module, configured to evaluate the NLU system using the target result, the first quantity, and the second quantity to obtain an evaluation result; The second processing module is specifically configured to: In a case where the target result includes the error result, determining a first number of target error results belonging to a high priority from the error results; The target keyword in each of the target error results is replaced with similar words to obtain a second number of expanded error results corresponding to each of the target error results.

6. The device according to claim 5, characterized in that The NLU system includes a prediction model, and the prediction results include: a first prediction result; The first processing module is specifically configured to: Performing result prediction processing on the test data set in the evaluation data set through the prediction model to obtain a first prediction result; The first comparison module is specifically configured to: The first prediction result is compared with the preset result to obtain the precision rate, which includes: precision rate and recall rate.

7. The device according to claim 5, characterized in that The prediction results include: a second prediction result; The first processing module is specifically configured to: Performing result prediction processing on the evaluation data set through the NLU system to obtain a second prediction result; The first comparison module is specifically configured to: The second prediction result is compared with the preset result to obtain at least one of an error result in the prediction result and the overall coverage rate.

8. The device according to claim 5, characterized in that Methods for determining target error results that are high priority include: If the similarity between a first error result in the error results and the corresponding preset result is greater than a preset similarity, it is determined that the first error result belongs to a target error result with a high priority.

9. A network device, characterized in that: Including memory, transceiver, processor: Memory for storing computer programs; a transceiver, configured to transmit and receive data under the control of the processor; A processor, configured to read the computer program in the memory and execute the steps in the evaluation method for a natural language understanding system according to any one of claims 1 to 4.

10. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the evaluation method of the natural language understanding system according to any one of claims 1 to 4.

Images