Method and device for evaluating intention understanding result

Abstract

The invention provides an intention understanding result evaluation method and device, and relates to the artificial intelligence fields of deep learning, large language models, natural language understanding, voice assistants and the like. The method comprises the steps that target evaluation information is acquired, the target evaluation information comprises a target problem and a target result, and the target result is an intention understanding result generated by an intention understanding model for the target problem; target reference information corresponding to the target evaluation information is determined from candidate reference information in a knowledge base; according to the target reference information, an evaluation result corresponding to the target evaluation information is generated, and the evaluation result is used for explaining whether the target result correctly understands the intention of the target problem or not. By applying the scheme of the invention, the accuracy of the evaluation result can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, large language models, natural language understanding and voice assistants, and especially to methods and apparatus for evaluating intent understanding results. Background Technology

[0002] Currently, voice assistants are no longer simple voice command response tools, but intelligent agents with complex intent understanding and execution capabilities, driven by large language models (LLMs). A large language model refers to a deep learning model trained on a large amount of text data, capable of generating natural language text or understanding the meaning of language text. Summary of the Invention

[0003] This disclosure provides methods and apparatus for evaluating the results of intent understanding.

[0004] An evaluation method for intent understanding results includes:

[0005] Obtain target evaluation information, which includes a target question and a target result, wherein the target result is an intent understanding result generated by the intent understanding model for the target question;

[0006] The target reference information corresponding to the target evaluation information is determined from the candidate reference information in the knowledge base;

[0007] Based on the target reference information, an evaluation result corresponding to the target evaluation information is generated. The evaluation result is used to explain whether the target result correctly understands the intent of the target problem.

[0008] An apparatus for evaluating intent understanding results includes: a first acquisition module, a first screening module, and a first evaluation module;

[0009] The first acquisition module is used to acquire target evaluation information, which includes a target question and a target result, wherein the target result is an intent understanding result generated by the intent understanding model for the target question;

[0010] The first filtering module is used to determine the target reference information corresponding to the target evaluation information from the candidate reference information in the knowledge base;

[0011] The first evaluation module is used to generate an evaluation result corresponding to the target evaluation information based on the target reference information. The evaluation result is used to explain whether the target result correctly understands the intent of the target problem.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of a first embodiment of the method for evaluating the intent understanding results described in this disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the process of determining the target reference information corresponding to the target evaluation information based on the knowledge base as described in this disclosure;

[0022] Figure 3 This is a flowchart of a second embodiment of the method for evaluating the intent understanding results described in this disclosure;

[0023] Figure 4 This is a schematic diagram of the composition structure of the first embodiment 400 of the intent understanding result evaluation device described in this disclosure;

[0024] Figure 5 This is a schematic diagram of the composition structure of the second embodiment 500 of the intent understanding result evaluation device described in this disclosure;

[0025] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0028] Figure 1 This is a flowchart of a first embodiment of the method for evaluating the intent understanding results described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0029] In step 101, target evaluation information is obtained, which includes the target question and the target result. The target result is the intent understanding result generated by the intent understanding model for the target question.

[0030] In step 102, the target reference information corresponding to the target evaluation information is determined from the candidate reference information in the knowledge base.

[0031] In step 103, an evaluation result corresponding to the target evaluation information is generated based on the target reference information. The evaluation result is used to explain whether the target result correctly understands the intent of the target problem.

[0032] In voice assistants, intent understanding models can be used to interpret user input and obtain intent understanding results. These models can be large language models. Currently, the accuracy of intent understanding results primarily relies on manual reporting. For example, if a user asks a question and the voice assistant provides a clearly incorrect response, it indicates an error in the intent understanding model's result, which can be reported. However, in most cases, users may not even notice the error, or even if they do, they may not report it, making it impossible to effectively evaluate the correctness of the intent understanding results.

[0033] By adopting the scheme described in the above method embodiment, for the target evaluation information composed of the target question and the target result (i.e. the intent understanding result corresponding to the target question), the target reference information corresponding to the target evaluation information can be determined first. Then, the corresponding evaluation result can be generated based on the target reference information. It can be determined whether the target result correctly understands the intent of the target question based on the target reference information, thereby realizing the effective evaluation of the intent understanding result and improving the accuracy of the evaluation result with the help of the target reference information.

[0034] The problems and intent understanding results described in the embodiments of this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. Furthermore, the entity implementing the solution described in this disclosure can obtain the problems through various public, legal, and compliant means, such as obtaining them from the user with their authorization. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0035] For example, if a user inputs the question "Does *** hot pot restaurant have a restroom?", and the intent understanding model generates the intent understanding result "Check if the POI (Point of Interest) of *** hot pot restaurant has the 'restroom' tag", then "Does *** hot pot restaurant have a restroom?" can be taken as the target question, and "Check if the POI of *** hot pot restaurant has the 'restroom' tag" can be taken as the target result. The target question and the target result can form the target evaluation information.

[0036] For target evaluation information, corresponding target reference information can be determined from candidate reference information in the knowledge base. In some embodiments of this disclosure, the candidate reference information in the knowledge base can be initially screened to obtain intermediate reference information, and then the intermediate reference information can be screened a second time to obtain the target reference information.

[0037] The knowledge base can be pre-generated and may include multiple candidate references. The source of these candidate references is not limited; for example, they may be manually constructed or collected online. The target references can be selected from the candidate references in the knowledge base through a two-stage filtering process, with different filtering methods used for each stage, offering great flexibility and convenience.

[0038] In some embodiments of this disclosure, the method for initial screening of candidate reference information in the knowledge base may include: obtaining the target similarity between each candidate reference information in the knowledge base and the target evaluation information, and screening intermediate reference information from the knowledge base based on the target similarity.

[0039] Since similarity calculation is usually relatively simple, and similarity calculation can be performed in parallel among candidate reference information, the processing efficiency of the initial screening can be improved. This allows for the rapid selection of the required intermediate reference information from the knowledge base, thus laying a good foundation for subsequent secondary screening.

[0040] In some embodiments of this disclosure, the target evaluation information may further include: target context information, which may be the question-and-answer content of at least one round of question-and-answer with the voice assistant before the target question (usually referring to at least one round of question-and-answer closest to the target question), the voice assistant may use an intent understanding model to perform intent understanding on the input question, and the target result may be the intent understanding result generated by combining the target context information. Correspondingly, the candidate reference information may include: reference question, reference result, and reference context information, the reference result may be the intent understanding result corresponding to the reference question generated by combining the reference context information, and the reference context information may be the question-and-answer content of at least one round of question-and-answer with the voice assistant before the reference question.

[0041] For example, the user engaged in multiple rounds of question-and-answer sessions with the voice assistant, including:

[0042] User: Navigate to Beijing Capital International Airport;

[0043] Voice assistant: Okay, route planning successful, estimated time 45 minutes;

[0044] User: Are there any gas stations along the way? ......

[0046] If the target question is "Are there any gas stations along the way?", then the content of the previous round of questions and answers can be used as the target information above.

[0047] By utilizing contextual information about the target, the intent understanding model can better understand the intent of the target question. For example, without contextual information, the intent understanding model cannot determine which gas station the user is asking about. However, by combining contextual information, it can be determined that the user is asking about a gas station on the planned route from the user's current location to the Capital Airport.

[0048] The content included in the candidate reference information corresponds one-to-one with the content included in the target evaluation information. For example, the target evaluation information includes the target question, while the candidate reference information includes the reference question; the target evaluation information includes the context information of the target, while the candidate reference information includes the context information of the reference; the target evaluation information includes the target result, while the candidate reference information includes the reference result, to facilitate the calculation of target similarity, etc.

[0049] Accordingly, in some embodiments of this disclosure, the target similarity may include at least one of the following: a first similarity, a second similarity, and a third similarity, wherein the first similarity may be the similarity between candidate reference information determined based on the target problem and the reference problem and the target evaluation information; the second similarity may be the similarity between candidate reference information determined based on the target problem, the target context information, the reference problem, and the reference context information and the target evaluation information; and the third similarity may be the similarity between candidate reference information determined based on the target result and the reference result and the target evaluation information.

[0050] In addition, in some embodiments of this disclosure, the target similarity may refer to vector similarity.

[0051] For example, taking any candidate reference information 'a' in the knowledge base, the reference question in candidate reference information 'a' can be converted into its corresponding vector representation (i.e., the reference question is vectorized and converted into numbers in the 0 and 1 space), and the target question in the target evaluation information can be converted into its corresponding vector representation. Then, the similarity between the two vector representations can be calculated to obtain the first similarity between candidate reference information 'a' and target evaluation information. In addition, the reference question in candidate reference information 'a' can be concatenated with the reference context information and converted into its corresponding vector representation, and the target question in the target evaluation information can be concatenated with the target context information and converted into its corresponding vector representation. Then, the similarity between the two vector representations can be calculated to obtain the second similarity between candidate reference information 'a' and target evaluation information. Furthermore, the reference result in candidate reference information 'a' can be converted into its corresponding vector representation, and the target result in the target evaluation information can be converted into its corresponding vector representation. Then, the similarity between the two vector representations can be calculated to obtain the third similarity between candidate reference information 'a' and target evaluation information.

[0052] In practical applications, one of the first, second, and third similarity scores can be used to filter intermediate reference information from the knowledge base. Alternatively, two or all three scores can be combined. Using only one score reduces computation and thus increases the speed of filtering intermediate reference information. Using two or all three scores improves the accuracy and reliability of the filtering results. The specific method used depends on the actual needs, making it very flexible and convenient.

[0053] In some embodiments of this disclosure, when filtering intermediate reference information from the knowledge base based on target similarity, in response to determining that the number of target similarities is 1, the candidate reference information in the knowledge base can be sorted in descending order of target similarity values. The candidate reference information in the top M positions after sorting can be determined as intermediate reference information, where M is a positive integer greater than 1. In response to determining that the number of target similarities is greater than 1, for each candidate reference information in the knowledge base, a comprehensive similarity can be determined by combining each target similarity. The candidate reference information can be sorted in descending order of comprehensive similarity values, and the candidate reference information in the top M positions after sorting can be determined as intermediate reference information. The specific value of M can be determined according to actual needs, such as 10.

[0054] For example, assuming the target similarity is the first similarity, after obtaining the first similarity between each candidate reference information in the knowledge base and the target evaluation information, the candidate reference information in the knowledge base can be sorted in descending order of the first similarity value, and the candidate reference information in the top 10 after sorting can be determined as intermediate reference information.

[0055] For example, assuming the target similarity includes first similarity, second similarity, and third similarity, after obtaining the first, second, and third similarities between each candidate reference information and the target evaluation information in the knowledge base, for each candidate reference information, the first, second, and third similarities can be multiplied by the corresponding weights, and the three products can be added together. The sum is determined as the comprehensive similarity of the candidate reference information. The specific values ​​of each weight can be determined according to actual needs. Furthermore, the candidate reference information can be sorted in descending order of comprehensive similarity value, and the top 10 candidate reference information after sorting can be determined as intermediate reference information.

[0056] Combining multiple similarity metrics to filter intermediate reference information can more comprehensively and accurately measure the semantic relevance between candidate reference information and target evaluation information. In addition, taking the top 10 candidate reference information after ranking as intermediate reference information can improve the quality and representativeness of intermediate reference information and control the subsequent computational complexity.

[0057] The solution described in this disclosure considers "I want to go to ** Gate" and "I want to go to ** Building" to be more similar than "I want to go to ** Gate" and "Is ** Gate fun?". However, calculating similarity based on the similarity between words can easily lead to errors. Therefore, the solution described in this disclosure proposes to use the similarity between vectors to calculate similarity. That is, the first similarity, the second similarity, and the third similarity can all be vector similarities, thereby effectively capturing semantic-level connections and improving the accuracy and robustness of the calculated similarity.

[0058] After filtering out the intermediate reference information, a second filtering can be performed on the intermediate reference information to obtain the required target reference information. In some embodiments of this disclosure, a first prompt can be generated first based on the intermediate reference information and the target evaluation information. Then, the first prompt can be input into the filtering model to obtain N target reference information items filtered out from the intermediate reference information by the filtering model. N is a positive integer and N is less than M. The specific value of N can be determined according to actual needs, such as 3.

[0059] For example, the target questions, target results, and target context information in the target evaluation information, as well as the reference questions, reference results, and reference context information in each intermediate reference information, can be structured to obtain the first prompt word. In addition, the first prompt word can also include relevant instructions, which instruct the screening model to select the N intermediate reference information that best matches the target evaluation information from each intermediate reference information based on dimensions such as semantic relevance and contextual consistency.

[0060] The filtering model can be a large language model or a pre-trained model. By using the filtering model to perform secondary filtering on intermediate reference information, the strong dependence on vector similarity can be reduced, possible erroneous filtering results can be eliminated, and the semantic understanding capabilities of the large language model can be used to refine the most relevant few intermediate reference information from the coarse filtering results, reducing the workload of subsequent processing.

[0061] Accordingly, Figure 2 This is a schematic diagram illustrating the process of determining the target reference information corresponding to the target evaluation information based on the knowledge base, as described in this disclosure. For example... Figure 2As shown, the knowledge base may include multiple candidate reference information. Assuming M is 10 and N is 3, the candidate reference information in the knowledge base can be initially screened to obtain 10 intermediate reference information. For example, the target similarity between each candidate reference information in the knowledge base and the target evaluation information can be obtained. Based on the target similarity, 10 intermediate reference information can be screened from the knowledge base, namely intermediate reference information 1 to intermediate reference information 10. The target similarity can be vector similarity. Then, the intermediate reference information can be screened a second time to obtain the target reference information. For example, a first prompt word can be generated based on each intermediate reference information and the target evaluation information. The first prompt word can be input into the screening model to obtain 3 intermediate reference information screened from the 10 intermediate reference information, assuming they are intermediate reference information 1, intermediate reference information 2 and intermediate reference information 5. These 3 intermediate reference information can then be determined as the target reference information.

[0062] Based on the target reference information, an evaluation result can be generated to determine whether the target outcome correctly understands the intent of the target problem. In some embodiments of this disclosure, a second prompt word can be generated based on the target evaluation information and the target reference information, and the second prompt word can be input into the evaluation model to obtain the output evaluation result.

[0063] The evaluation model can be a large language model, or it can be pre-trained. The generation of the second prompt word aims to guide the evaluation model to accurately judge the target result in the target evaluation information based on the target reference information. For example, the target question, target result, and target context information in the target evaluation information, as well as the reference questions, reference results, and reference context information in each target reference information, can be structured to obtain the second prompt word. Furthermore, the second prompt word can also include relevant instructions, directing the evaluation model to act as an "evaluation expert," such as "Please compare with the reference case, determine whether the target result correctly understands the intent of the target question, and output a conclusion of 'correct' or 'incorrect'," etc. Through the second prompt word, it is emphasized that the comparison is based on domain knowledge and correct logic in the target reference information, rather than simply relying on the model's own knowledge to generate the evaluation result, thereby improving the accuracy and reliability of the obtained evaluation result.

[0064] In some embodiments of this disclosure, the evaluation result may include: a first result and a second result, wherein the first result may indicate that the target result correctly understands the intent of the target problem, and the second result may indicate that the target result does not correctly understand the intent of the target problem, such as the first result being 1 and the second result being 0.

[0065] Suppose the user's input question is "Does *** hot pot restaurant have a restroom?" If the target result is "Check if the POI of *** hot pot restaurant has the 'restroom' tag", then the evaluation result can be determined as the first result. If the target result is "Check if there is a place named 'restroom' near *** hot pot restaurant", then the evaluation result can be determined as the second result.

[0066] In addition, in some embodiments of this disclosure, in response to determining that the evaluation result is the second result, a manual verification result for the second result can also be obtained; in response to determining that the manual verification result is an evaluation error, the target evaluation information can be added to the knowledge base as candidate reference information; in response to determining that the manual verification result is an evaluation correct, a manual correction result for the target result can be obtained, and the corrected target evaluation information can be added to the knowledge base as candidate reference information; and the intent understanding model can be optimized based on the corrected target evaluation information.

[0067] In other words, if the evaluation result is the first result, manual verification is unnecessary to reduce the workload of manual processing. Conversely, if the evaluation result is the second result, manual verification can be introduced. If the manual verification determines that the evaluation is incorrect, meaning the target result is actually correct, the target evaluation information can be added to the knowledge base as candidate reference information, thereby achieving continuous optimization and expansion of the knowledge base. If the manual verification determines that the evaluation is correct, meaning the target result is indeed incorrect, the incorrect target result can be corrected to the correct target result through manual correction. For example, "Check if there is a place called 'toilet' near *** hot pot restaurant" can be corrected to "Check if the POI of *** hot pot restaurant has the 'toilet' tag." The corrected target evaluation information can then be added to the knowledge base as candidate reference information. In addition, the corrected target evaluation information can be used to optimize the intent understanding model, enabling the intent understanding model to learn specifically from cases of misunderstanding, thereby strengthening the weak links of the intent understanding model and optimizing its performance.

[0068] In some embodiments of this disclosure, in response to determining that the evaluation result is the second result, the error type corresponding to the target evaluation information can also be determined and recorded. In response to determining that the report generation conditions are met, the number of occurrences of different error types within the most recent predetermined time period can be counted based on the recorded content, and a visual report can be generated and displayed based on the statistical results.

[0069] When the evaluation model generates a second evaluation result, it can automatically analyze and record the error types corresponding to the target evaluation information. These error types may include domain identification errors, entity extraction errors, and referencing resolution errors. By continuously recording these error type information, the system can automatically count the occurrence frequency of different error types within the most recent predetermined time period when report generation conditions are met. An intuitive visual analysis report can be generated based on the statistical results. Meeting the report generation conditions can refer to a predetermined period of time; the specific values ​​of the period and the most recent predetermined time period can be determined according to actual needs. In this way, the maintainers of the intent understanding model can quickly identify model defects based on the visual analysis report and perform targeted optimizations, thereby continuously improving the performance of the intent understanding model.

[0070] Based on the foregoing introduction, Figure 3 This is a flowchart of a second embodiment of the method for evaluating the intent understanding results described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0071] In step 301, target evaluation information is obtained.

[0072] The target assessment information may include the target problem, the target result, and the information preceding the target.

[0073] In step 302, the first similarity between the target evaluation information and each candidate reference information in the knowledge base is obtained.

[0074] Candidate reference information may include: reference questions, reference results, and references to previous text.

[0075] The first similarity can refer to the similarity between candidate reference information determined based on the target problem and the reference problem, and the target evaluation information. The first similarity can be vector similarity.

[0076] In step 303, the candidate reference information in the knowledge base is sorted in descending order of the first similarity value, and the candidate reference information in the top 10 after sorting is determined as the intermediate reference information.

[0077] Assuming the target problem is "lower navigation volume", then the reference problem in intermediate reference information 1 might be "turn the navigation volume down", the reference problem in intermediate reference information 2 might be "turn the navigation volume down a bit", the reference problem in intermediate reference information 3 might be "reduce the navigation volume to 40%", ..., the reference problem in intermediate reference information 10 might be "turn off the navigation sound", etc.

[0078] In step 304, a first prompt word is generated based on the intermediate reference information and the target evaluation information, and the first prompt word is input into the filtering model to obtain 3 target reference information selected by the filtering model from 10 intermediate reference information.

[0079] In step 305, a second prompt word is generated based on the target evaluation information and the target reference information, and the second prompt word is input into the evaluation model to obtain the output evaluation result.

[0080] The evaluation results may include a first result and a second result. The first result may indicate that the target result correctly understands the intent of the target problem, while the second result may indicate that the target result does not correctly understand the intent of the target problem.

[0081] In step 306, it is determined whether the evaluation result is the first result or the second result. If it is the first result, the process ends; if it is the second result, step 307 is executed.

[0082] If the result is the first one, the process can end, and there are no restrictions on how to proceed afterward.

[0083] In step 307, the manual verification result for the second result is obtained.

[0084] In step 308, it is determined whether the manual verification result is an evaluation error or an evaluation correct. If it is an evaluation error, step 309 is executed; if it is an evaluation correct, step 310 is executed.

[0085] In step 309, the target evaluation information is added to the knowledge base as candidate reference information, and then the process ends.

[0086] In step 310, the manual correction results for the target result are obtained, and the corrected target evaluation information is added to the knowledge base as candidate reference information. The intent understanding model is then optimized based on the corrected target evaluation information, and the process ends.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0088] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0089] Figure 4 This is a schematic diagram of the structural composition of the first embodiment 400 of the intent understanding result evaluation device described in this disclosure. Figure 4 The system includes: a first acquisition module 401, a first screening module 402, and a first evaluation module 403.

[0090] The first acquisition module 401 is used to acquire target evaluation information, which includes the target question and the target result. The target result is the intent understanding result generated by the intent understanding model for the target question.

[0091] The first filtering module 402 is used to determine the target reference information corresponding to the target evaluation information from the candidate reference information in the knowledge base.

[0092] The first evaluation module 403 is used to generate an evaluation result corresponding to the target evaluation information based on the target reference information. The evaluation result is used to explain whether the target result correctly understands the intent of the target problem.

[0093] Based on the target evaluation information acquired by the first acquisition module 401, the first filtering module 402 can first perform an initial filtering of the candidate reference information in the knowledge base to obtain the filtered intermediate reference information. Then, it can perform a second filtering of the intermediate reference information to obtain the target reference information corresponding to the target evaluation information. The knowledge base can be pre-generated and may include multiple candidate reference information entries.

[0094] When performing initial screening of candidate reference information in the knowledge base, the first screening module 402 can obtain the target similarity between each candidate reference information in the knowledge base and the target evaluation information, and can screen out intermediate reference information from the knowledge base based on the target similarity.

[0095] The target evaluation information may further include: target context information, which may be the question and answer content of at least one round of question and answer with the voice assistant before the target question (usually referring to at least one round of question and answer closest to the target question). The voice assistant may use an intent understanding model to understand the intent of the input question. The target result may be the intent understanding result generated by combining the target context information. Correspondingly, the candidate reference information may include: reference question, reference result, and reference context information. The reference result may be the intent understanding result corresponding to the reference question generated by combining the reference context information. The reference context information may be the question and answer content of at least one round of question and answer with the voice assistant before the reference question.

[0096] Accordingly, the target similarity may include at least one of the following: a first similarity, a second similarity, and a third similarity, wherein the first similarity may be the similarity between candidate reference information determined based on the target problem and the reference problem and the target evaluation information; the second similarity may be the similarity between candidate reference information determined based on the target problem, the target context information, the reference problem, and the reference context information and the target evaluation information; and the third similarity may be the similarity between candidate reference information determined based on the target result and the reference result and the target evaluation information.

[0097] When the first filtering module 402 filters intermediate reference information from the knowledge base based on target similarity, in response to determining that the number of target similarities is 1, it can sort the candidate reference information in the knowledge base in descending order of target similarity values, and determine the candidate reference information in the top M positions after sorting as intermediate reference information, where M is a positive integer greater than 1. In response to determining that the number of target similarities is greater than 1, for each candidate reference information in the knowledge base, it can determine a comprehensive similarity by combining each target similarity, and sort the candidate reference information in descending order of comprehensive similarity values, and determine the candidate reference information in the top M positions after sorting as intermediate reference information. The target similarity can be vector similarity.

[0098] After filtering out the intermediate reference information, the first filtering module 402 can further filter the intermediate reference information to obtain the required target reference information. For example, the first filtering module 402 can generate a first prompt word based on the intermediate reference information and the target evaluation information, and input the first prompt word into the filtering model to obtain N target reference information selected by the filtering model from M intermediate reference information, where N is a positive integer and N is less than M.

[0099] Furthermore, the first evaluation module 403 can generate an evaluation result based on the target reference information to determine whether the target result correctly understands the intent of the target problem.

[0100] For example, the first evaluation module 403 can generate a second prompt word based on the target evaluation information and the target reference information, and can input the second prompt word into the evaluation model to obtain the output evaluation result. The evaluation result may include a first result and a second result. The first result may indicate that the target result correctly understands the intent of the target problem, and the second result may indicate that the target result does not correctly understand the intent of the target problem.

[0101] Figure 5 This is a schematic diagram of the structural composition of a second embodiment 500 of the intent understanding result evaluation device described in this disclosure. Figure 5 The system includes: a second acquisition module 501, a second screening module 502, a second evaluation module 503, and a post-processing module 504.

[0102] Among them, the second acquisition module 501, the second screening module 502, and the second evaluation module 503 are respectively connected to... Figure 4 The first acquisition module 401, the first filtering module 402, and the first evaluation module 403 shown are the same, compared to Figure 4 The embodiment shown, Figure 5 The illustrated embodiment further includes a post-processing module 504.

[0103] In response to determining that the evaluation result is the second result, the post-processing module 504 can obtain the manual verification result of the second result. In response to determining that the manual verification result is an evaluation error, the target evaluation information can be added to the knowledge base as candidate reference information. In response to determining that the manual verification result is an evaluation correct, the manual correction result of the target result can be obtained, and the corrected target evaluation information can be added to the knowledge base as candidate reference information. The intent understanding model can also be optimized based on the corrected target evaluation information.

[0104] In addition, in response to determining that the evaluation result is the second result, the post-processing module 504 can also determine and record the error type corresponding to the target evaluation information. In response to determining that the report generation conditions are met, it can count the number of occurrences of different error types within the most recent predetermined time period based on the recorded content, and generate a visual report based on the statistical results for display.

[0105] The specific workflow of each of the above device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0106] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly deep learning, large language models, natural language understanding, and voice assistants. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0107] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Figure 6A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0117] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0118] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for evaluating the outcome of intent understanding, comprising: Obtain target evaluation information, which includes a target question and a target result, wherein the target result is an intent understanding result generated by the intent understanding model for the target question; The target reference information corresponding to the target evaluation information is determined from the candidate reference information in the knowledge base; Based on the target reference information, an evaluation result corresponding to the target evaluation information is generated. The evaluation result is used to explain whether the target result correctly understands the intent of the target problem.

2. The method according to claim 1, wherein, The step of determining the target reference information corresponding to the target evaluation information from the candidate reference information in the knowledge base includes: The candidate reference information in the knowledge base is initially screened to obtain intermediate reference information, and the intermediate reference information is then screened a second time to obtain the target reference information.

3. The method according to claim 2, wherein, The initial screening of candidate reference information in the knowledge base includes: The target similarity between each candidate reference information in the knowledge base and the target evaluation information is obtained respectively, and the intermediate reference information is selected from the knowledge base based on the target similarity.

4. The method according to claim 3, wherein, The target evaluation information also includes: target context information, which is the question and answer content of at least one round of question and answer with the voice assistant before the target question, the voice assistant uses the intent understanding model to understand the intent of the input question, and the target result is the intent understanding result generated by combining the target context information; The candidate reference information includes: reference question, reference result, and reference context information. The reference result is the intent understanding result corresponding to the reference question generated by combining the reference context information. The reference context information is the question and answer content of at least one round of question and answer with the voice assistant before the reference question.

5. The method according to claim 4, wherein, The target similarity includes at least one of the following: first similarity, second similarity, and third similarity; The first similarity is the similarity between the candidate reference information determined based on the target question and the reference question and the target evaluation information; the second similarity is the similarity between the candidate reference information determined based on the target question, the target context information, the reference question, and the reference context information and the target evaluation information; and the third similarity is the similarity between the candidate reference information determined based on the target result and the reference result and the target evaluation information.

6. The method according to claim 5, wherein, The step of filtering the intermediate reference information from the knowledge base based on the target similarity includes: In response to determining that the number of target similarities is 1, the candidate reference information in the knowledge base is sorted in descending order of the target similarity values, and the candidate reference information in the first M positions after sorting is determined as the intermediate reference information, where M is a positive integer greater than 1; In response to the determination that the number of target similarities is greater than 1, for each candidate reference information in the knowledge base, a comprehensive similarity is determined by combining each target similarity, and the candidate reference information is sorted in descending order of the comprehensive similarity value. The candidate reference information in the top M positions after sorting is determined as the intermediate reference information.

7. The method according to claim 3, wherein, The target similarity includes vector similarity.

8. The method according to claim 2, wherein, The secondary filtering of the intermediate reference information to obtain the target reference information includes: A first prompt word is generated based on the intermediate reference information and the target evaluation information; The first prompt word is input into the filtering model to obtain N target reference information items filtered out from the intermediate reference information by the filtering model, where N is a positive integer and N is less than M.

9. The method according to claim 1, wherein, The evaluation results corresponding to the target evaluation information include: A second prompt word is generated based on the target evaluation information and the target reference information; The second prompt word is input into the evaluation model to obtain the output evaluation result, which includes a first result and a second result. The first result indicates that the target result correctly understands the intent of the target question, and the second result indicates that the target result does not correctly understand the intent of the target question.

10. The method of claim 9, further comprising: In response to determining that the evaluation result is the second result, obtain the manual verification result for the second result; In response to determining that the manual verification result is an evaluation error, the target evaluation information is added to the knowledge base as the candidate reference information; In response to determining that the manual verification result is correct, the manual correction result for the target result is obtained, and the corrected target evaluation information is added to the knowledge base as the candidate reference information. The intent understanding model is then optimized based on the corrected target evaluation information.

11. The method of claim 9, further comprising: In response to determining that the evaluation result is the second result, the error type corresponding to the target evaluation information is determined and recorded; In response to the determination that the report generation conditions are met, the system counts the number of times different error types occurred within the most recent predetermined time period based on the recorded content, and generates a visual report based on the statistical results for display.

12. An evaluation apparatus for understanding the results of an intention, comprising: The module comprises a first acquisition module, a first screening module, and a first evaluation module. The first acquisition module is used to acquire target evaluation information, which includes a target question and a target result, wherein the target result is an intent understanding result generated by the intent understanding model for the target question; The first filtering module is used to determine the target reference information corresponding to the target evaluation information from the candidate reference information in the knowledge base; The first evaluation module is used to generate an evaluation result corresponding to the target evaluation information based on the target reference information. The evaluation result is used to explain whether the target result correctly understands the intent of the target problem.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-11.

Images