Automatic man-machine dialogue test and element analysis method based on dialogue content
By obtaining human-computer dialogue from the log database and using script processing and analysis, integrating scene use cases and labeling correct elements, the problem of incompatibility of different scenario tests in the existing technology is solved, and the accuracy and efficiency of automated human-computer dialogue tests are improved. Ensure the reliability and quality of the test results.
Patent Information
- Application Number
- CN202411079643.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-03
AI Technical Summary
The existing technology has failed to effectively test automated human-computer dialogues in different scenarios, resulting in the dialogue content that does not match the actual scenarios, affecting the accuracy and efficiency of the test results.
By obtaining human-computer dialogue from the log database, using shell scripts and Python scripts to process and analyze dialogue content, integrating scene use cases, annotating and adjusting correct elements, conducting tests and generating test reports, ensuring the accuracy and efficiency of test results.
It improves the accuracy and efficiency of test results, can analyze and test for different scenarios, reduce human errors, adjust robot programs in a timely manner, and ensure the quality of human-computer dialogue and the reliability of test results.
Smart Images

Figure CN119065966A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of artificial intelligence and relates to an automated human-computer dialogue testing technology, in particular to an automated human-computer dialogue testing and factor analysis method based on dialogue content. Background Art
[0002] Automated human-computer dialogue testing, also known as human-computer dialogue automated testing, refers to the use of automated testing technology and tools to test the functions, performance, stability, etc. of the human-computer dialogue system; automated testing can ensure that the conditions and steps of each test are consistent, avoiding test deviations caused by human factors; through carefully designed test cases, automated testing can cover more functions and scenarios, and improve the comprehensiveness of testing; automated testing reduces the possibility of human errors, making test results more accurate and reliable; automated testing can discover potential problems in the early stages of development, thereby avoiding problems from being magnified in the later stages and affecting the stability of the system; automated testing can be set as part of continuous integration to monitor the status of the system in real time to ensure the stability and reliability of the system.
[0003] The prior art (invention patent with application number 2020111551179) discloses a testing method for a human-computer dialogue system, which includes: inputting a voice instruction group into a standard system at least twice to obtain a standard return result; the voice instruction group includes at least 2 voice instructions; using the standard return result, determining a key information group corresponding to the voice instruction group; inputting the voice instruction group into the system to be tested to obtain a return result to be tested; determining the test result of the system to be tested according to the key information group and the return result to be tested; by inputting the voice instruction group into the standard system multiple times to obtain a standard return result, and using the standard return result to determine the key information group; so as to realize the recognition of the context in multiple rounds of dialogue; and then using the key information group and the return result to be tested to determine the test result of the system to be tested, the test of multiple rounds of dialogue in the human-computer dialogue system is realized; however, the prior art identifies the key information group and the return group to be tested through multiple rounds of dialogue to determine the test result. In actual applications, human-computer dialogues may occur in different scenarios. The prior art does not consider testing automated human-computer dialogues for different scenarios, resulting in the content of the dialogue not being consistent with the actual scenario.
[0004] The present invention provides an automated human-computer dialogue test and factor analysis method based on dialogue content to solve the above technical problems. Summary of the invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an automated human-computer dialogue test and factor analysis method based on dialogue content, which is used to solve the technical problem that the prior art does not consider testing automated human-computer dialogue for different scenarios, resulting in the content of the dialogue not being consistent with the actual scenario.
[0006] To achieve the above object, the first aspect of the present invention provides an automated human-computer dialogue test and factor analysis method based on dialogue content, comprising:
[0007] Step S1: Obtain several human-computer dialogues from the log database, and use shell scripts to process the human-computer dialogues to obtain several scenario use cases in fixed formats;
[0008] Step S2: labeling several scenario use cases to obtain correct elements; adjusting several scenario use cases according to the correct elements; uploading the adjusted scenario use cases to the same-level directory as the Python script;
[0009] Step S3: Use Python script to test the robot program to obtain a return dialogue; determine whether to continue testing based on the return dialogue; if yes, proceed to the next set of scenario dialogues; if no, exit the corresponding group and mark the result as failed;
[0010] Step S4: extract the dialogue elements in the returned dialogue and determine whether the dialogue element extraction is correct; if yes, mark it as correct; if no, mark it as the real result; analyze the element extraction pass rate of each scenario use case based on the dialogue elements;
[0011] Step S5: After all scenario use cases have been automatically tested, a test report is generated based on the test results and displayed to the user.
[0012] Preferably, the method of using shell script to process human-computer dialogue to obtain several scenario use cases in fixed formats includes:
[0013] Retrieve the human-computer dialogue in the log database, analyze the fields and formats of the human-computer dialogue; divide the human-computer dialogue into several dialogue contents; the human-computer dialogue includes the dialogue ID, user statement and robot statement;
[0014] Extract conversation IDs from several conversation contents, integrate conversation contents with the same conversation ID into a group of conversation examples, and output the conversation examples in a preset format to obtain several scenario use cases.
[0015] It should be noted that before processing the human-computer dialogue, it is necessary to pre-process the human-computer dialogue. When the human-computer dialogue lacks content of a certain content type, the missing content will be supplemented; otherwise, the human-computer dialogue is analyzed to see whether there is repetition. If so, the repeated content will be removed; the dialogue content includes the user's statement and the corresponding robot's statement.
[0016] The present invention integrates the conversation contents with the same conversation ID into the same group according to the conversation ID in the human-computer conversation, and outputs several scenario use cases according to a preset format. It can analyze the conversation for different scenarios, provide a data basis for subsequent testing, and is conducive to improving the accuracy of the test results.
[0017] Preferably, the adjusting of several scenario use cases according to the correct elements includes:
[0018] Retrieve the correct elements marked in several scenario use cases; obtain the element type table and match the correct elements with the element type table;
[0019] Determine whether there are any unmatched preset features in the feature class table; if yes, supplement the missing features; if no, continue analyzing the correct features.
[0020] It should be noted that the correct elements include the name of the person involved in the human-computer dialogue, telephone number, address, time, product name, intent type, etc.; different correct elements can be set according to different scenario use cases, and the correct elements can be adjusted according to actual conditions.
[0021] The present invention supplements the missing correct elements according to the correct elements corresponding to several scenario use cases, can eliminate the influence of other factors on the test results, is conducive to improving the accuracy of human-computer dialogue testing, and improves the efficiency of testing.
[0022] Preferably, the method of testing the robot program using a Python script to obtain a return dialogue includes:
[0023] Modify the basic information of the scenario use case in the Python script and the program address of the test robot; the basic information includes the document path and document name of the scenario use case and the document name of the generated result report;
[0024] Analyze scenario use cases using Python scripts. Group user statements in several scenario use cases according to conversation IDs and send them to the robot program. The robot program analyzes the user statements and obtains return conversations. The return conversations include conversation IDs, extracted elements, and conversation content.
[0025] The present invention uses Python scripts to test the robot program, and sets the test scenario use cases and report addresses, which is convenient for managing the scenario use cases and querying the test results.
[0026] Preferably, judging whether to continue the test according to the returned dialogue includes:
[0027] Retrieve the conversation ID and conversation content in the returned conversation; obtain the preset conversation table; match the conversation ID with the conversation table to obtain the standard content corresponding to the conversation content;
[0028] Compare the conversation content with the standard content to determine whether the conversation content is consistent with the standard content; if yes, continue testing; if not, stop testing the corresponding scenario use case.
[0029] The present invention compares the dialogue content in the dialogue returned by the robot with the corresponding standard content to determine whether to perform a test on the scenario use case. When an error occurs in the dialogue of a scenario use case, the next scenario use case can be tested first, which is beneficial to improving the efficiency of the test.
[0030] Preferably, the determining whether the dialogue element extraction is correct includes:
[0031] Retrieve the conversation ID and extract elements from the returned conversation, extract the corresponding correct elements according to the conversation ID; integrate the correct elements into an element matching table;
[0032] The extracted elements are matched with the element matching table in turn to determine whether the extracted elements are consistent with the corresponding correct elements in the element matching table. If so, the dialogue element extraction is determined to be correct; if not, the dialogue element extraction is determined to be incorrect.
[0033] The present invention analyzes whether the extracted elements in the returned dialogue are consistent with the correct elements, can timely determine the error points in the human-computer dialogue, and adjust the robot in time, which is beneficial to ensuring the quality of the human-computer dialogue.
[0034] Preferably, analyzing the element extraction pass rate of each scenario use case according to the dialogue elements includes:
[0035] Retrieve the conversation ID and the number of extracted elements in the returned conversation; analyze the extracted elements to obtain the number of correctly marked elements;
[0036] The extracted elements are divided according to the dialogue ID to obtain the number of elements and the correct number of several scenario use cases; the correct number is divided by the number of elements to obtain the element extraction pass rate corresponding to several scenario use cases.
[0037] The present invention divides the extracted elements into corresponding scenario use cases according to the dialogue ID in the returned dialogue, and analyzes the element extraction pass rates of several scenario use cases, which can evaluate the test of the robot and facilitate the subsequent adjustment of the human-computer dialogue.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. The present invention obtains a number of human-computer dialogues from a log database, integrates the dialogue contents with the same dialogue ID into the same group according to the dialogue ID in the human-computer dialogue, and outputs a number of scenario use cases according to a preset format, so as to analyze the dialogues for different scenarios, provide a data basis for subsequent testing, and help improve the accuracy of the test results; supplement the missing correct elements according to the correct elements corresponding to the several scenario use cases, and eliminate the influence of other factors on the test results, which is beneficial to improve the accuracy of the human-computer dialogue test and improve the efficiency of the test; use Python scripts to test the robot program, and set the scenario use cases and report addresses of the tests, so as to facilitate the management of the scenario use cases and query the test results.
[0040] 2. The present invention compares the dialogue content in the dialogue returned by the robot with the corresponding standard content to determine whether to test the scenario use case. When an error occurs in the dialogue of the scenario use case, the next scenario use case can be tested first, which is conducive to improving the efficiency of the test; by analyzing whether the extracted elements in the returned dialogue are consistent with the correct elements, the error points in the human-computer dialogue can be judged in time, and the robot can be adjusted in time, which is conducive to ensuring the quality of the human-computer dialogue; the extracted elements are divided into corresponding scenario use cases according to the dialogue ID in the returned dialogue, and the element extraction pass rate of several scenario use cases is analyzed, which can evaluate the test of the robot, which is conducive to the subsequent adjustment of the human-computer dialogue; BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0042] Figure 1 Schematic diagram of the specific steps of the method of the present invention;
[0043] Figure 2 A schematic diagram of the human-computer dialogue processing steps of the present invention;
[0044] Figure 3 It is a schematic diagram of the human-computer dialogue test analysis steps of the present invention. DETAILED DESCRIPTION
[0045] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] See also Figure 1 The first aspect of the present invention provides an automated human-computer dialogue test and factor analysis method based on dialogue content, including:
[0047] Step S1: Obtain several human-computer dialogues from the log database, and use shell scripts to process the human-computer dialogues to obtain several scenario use cases in fixed formats;
[0048] Step S2: labeling several scenario use cases to obtain correct elements; adjusting several scenario use cases according to the correct elements; uploading the adjusted scenario use cases to the same level directory as the python script;
[0049] Step S3: Use Python script to test the robot program to obtain a return dialogue; determine whether to continue testing based on the return dialogue; if yes, proceed to the next set of scenario dialogues; if no, exit the corresponding group and mark the result as failed;
[0050] Step S4: extract the dialogue elements in the returned dialogue and determine whether the dialogue element extraction is correct; if yes, mark it as correct; if no, mark it as the real result; analyze the element extraction pass rate of each scenario use case based on the dialogue elements;
[0051] Step S5: After all scenario use cases have been automatically tested, a test report is generated based on the test results and displayed to the user.
[0052] See also Figure 2 , obtain several human-computer dialogues from the log database, analyze the fields and formats of the human-computer dialogues; divide the human-computer dialogues into several dialogue contents; wherein the human-computer dialogues include dialogue IDs, user statements and robot statements; extract dialogue IDs from several dialogue contents, integrate dialogue contents with the same dialogue IDs into a group of dialogue examples, and output the dialogue examples in a preset format to obtain several scenario use cases.
[0053] For example: assuming that there are 3 groups of human-computer dialogues, analyze the fields and formats of the 3 groups of human-computer dialogues, and divide the human-computer dialogues into 30 groups of dialogue contents; analyze the 30 groups of dialogue contents to obtain 3 groups of scenarios with dialogue IDs 1, 2, and 3, integrate the dialogue contents with the same dialogue ID into a group of dialogue examples, and output 3 groups of scenario use cases; the first group of scenario use cases has a total of 8 groups of dialogue contents; the second group of scenario use cases has a total of 12 groups of dialogue contents; the third group of scenario use cases has a total of 10 groups of dialogue contents.
[0054] Label several scenario use cases to obtain the correct elements, obtain the feature type table, and match the correct elements with the feature type table; determine whether there are unmatched preset elements in the feature class table; if so, supplement the missing elements; if not, continue to analyze the correct elements.
[0055] For example: suppose that we analyze the first three groups of dialogue content in the first group of scenario use cases, extract the correct elements in the first three groups of dialogue content, and find that the location element is missing in the first group of dialogue content; the product element is missing in the third group of dialogue content; then identify the location element in the first group of dialogue content and mark it; and supplement the product element in the third group of dialogue.
[0056] Upload the adjusted scenario use case to the same-level directory as the Python script; modify the basic information of the scenario use case and the program address of the test robot in the Python script; the basic information includes the document path and document name of the scenario use case and the document name of the generated result report; use the Python script to analyze the scenario use case; group the user statements in several scenario use cases according to the dialogue ID and send them to the robot program, which analyzes the user statements and obtains the returned dialogue; the returned dialogue includes the dialogue ID, extracted elements and dialogue content.
[0057] See also Figure 3 , retrieve the conversation ID and conversation content in the returned conversation; obtain the preset conversation table; match the conversation ID with the conversation table to obtain the standard content corresponding to the conversation content; compare the conversation content with the standard content to determine whether the conversation content is consistent with the standard content; if yes, proceed to the next set of scene conversations; if not, exit the corresponding group and mark the result as failed.
[0058] For example: suppose that the dialogue content of the first group of scenario use cases is analyzed, and the dialogue table corresponding to the first group of scenario use cases is obtained; the dialogue content is compared with the standard content respectively, and it is found that the 8 groups of dialogue content in the first group of scenario use cases are consistent with the standard, then the second group of scenario use cases is tested;
[0059] Analyze the dialogue content of the second group of scenarios, obtain the dialogue table corresponding to the second group of scenario use cases, analyze the dialogue content in the second group of return dialogues with the standard content in turn, and in the fifth group of dialogues, find that the dialogue content in the return dialogue is inconsistent with the standard content, then exit the dialogue of the second group of scenario use cases, mark the test of the second group of scenario use cases as failed; and test the third group of scenario use cases;
[0060] If the dialogue content in the return dialogue of the scenario use case in Group 3 is consistent with the standard content, the element analysis is performed.
[0061] Retrieve the conversation ID and extracted elements in the returned conversation, and extract the corresponding correct elements according to the conversation ID; integrate the correct elements into an element matching table; match the extracted elements with the element matching table in turn to determine whether the extracted elements are consistent with the corresponding correct elements in the element matching table. If yes, mark them as correct; if not, mark them as true results.
[0062] For example: suppose that the 15 extracted elements extracted from the first group of scenario use cases are analyzed and the corresponding correct elements are extracted; the correct elements are integrated into an element matching table; the 15 extracted elements are matched with the element matching table in sequence, and the 5th, 7th, and 10th elements are inconsistent with the correct elements, and these elements are marked as true results; the remaining elements are marked as correct;
[0063] Now we analyze the 18 extracted elements from the third group of scenario use cases, extract the corresponding correct elements, and integrate the correct elements into an element matching table; match the 18 extracted elements with the element matching table in turn, and find that the 1st and 15th elements are inconsistent with the correct elements. Mark these elements as true results; mark the remaining elements as correct.
[0064] Retrieve the conversation ID and the number of extracted elements in the returned conversation; analyze the extracted elements to obtain the number of correct labels; divide the extracted elements according to the conversation ID to obtain the number of elements and the correct number of several scenario use cases; divide the correct number by the number of elements to obtain the element extraction pass rate corresponding to several scenario use cases; when all scenario use cases are automatically tested, generate a test report based on the test results and display it to the user.
[0065] For example, suppose that the element extraction pass rate of the first group of scenario use cases is calculated, and the extracted elements are divided according to the conversation ID to obtain 15 elements of several scenario use cases and 12 correct elements; the element extraction pass rate is 80%;
[0066] Now the element extraction pass rate of the third group of scenario use cases is calculated. The extracted elements are divided according to the dialogue ID to obtain 18 elements of several scenario use cases and 16 correct elements; the element extraction pass rate is 88.89%;
[0067] The test contents of the first and third groups of scenario use cases, as well as the element extraction pass rate, are integrated into a test report and displayed to the user; the test results of the second group of scenario use cases are displayed to the user as failures.
[0068] The working principle of the present invention is as follows: the present invention obtains a number of human-computer dialogues from a log database, uses a shell script to process the human-computer dialogues to obtain a number of scenario use cases in a fixed format; annotates a number of scenario use cases to obtain correct elements; adjusts a number of scenario use cases according to the correct elements; uploads the adjusted scenario use cases to a directory at the same level as a python script; uses a Python script to test a robot program to obtain a return dialogue; determines whether to continue testing according to the return dialogue; if yes, performs the next set of scenario dialogues; if not, exits the corresponding group and marks the result as failed; extracts dialogue elements from the return dialogue, and determines whether the dialogue element extraction is correct; if yes, marks it as correct; if not, marks the real result; analyzes the element extraction pass rate of each scenario use case according to the dialogue elements; when all scenario use cases are automatically tested, generates a test report according to the test results and displays it to the user.
[0069] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.
Claims
1. Automated human-computer dialogue testing and factor analysis methods based on dialogue content, It is characterized in that include: Step S1: Obtain several human-computer dialogues from the log database, and use shell scripts to process the human-computer dialogues to obtain several scenario use cases in fixed formats; Step S2: labeling several scenario use cases to obtain correct elements; adjusting several scenario use cases according to the correct elements; uploading the adjusted scenario use cases to the same-level directory as the Python script; Step S3: Use Python script to test the robot program and obtain the returned dialogue; Determine whether to continue the test based on the returned dialogue; If yes, proceed to the next set of scene dialogues; If not, exit the corresponding group and mark the result as failed; Step S4: extracting the dialogue elements in the returned dialogue, and determining whether the dialogue elements are extracted correctly; If yes, mark it as correct; If not, mark the true result; Analyze the element extraction pass rate of each scenario use case based on the dialogue elements; Step S5: After all scenario use cases have been automatically tested, a test report is generated based on the test results and displayed to the user.
2. The method for automated human-computer dialogue testing and factor analysis based on dialogue content according to claim 1, It is characterized in that The shell script is used to process the human-computer dialogue to obtain several scenario use cases in fixed formats, including: Retrieve the human-computer dialogue in the log database, analyze the fields and formats of the human-computer dialogue; divide the human-computer dialogue into several dialogue contents; the human-computer dialogue includes the dialogue ID, user statement and robot statement; Extract conversation IDs from several conversation contents, integrate conversation contents with the same conversation ID into a group of conversation examples, and output the conversation examples in a preset format to obtain several scenario use cases.
3. The method for automated human-computer dialogue testing and factor analysis based on dialogue content according to claim 1, It is characterized in that The adjustments to several scenario use cases are based on the correct elements, including: Retrieve the correct elements marked in several scenario use cases; obtain the element type table and match the correct elements with the element type table; Determine whether there are any unmatched preset features in the feature class table; if yes, supplement the missing features; if no, continue analyzing the correct features.
4. The method for automated human-computer dialogue testing and factor analysis based on dialogue content according to claim 1, It is characterized in that The method of testing the robot program using Python script to obtain a return dialogue includes: Modify the basic information of the scenario use case in the Python script and the program address of the test robot; the basic information includes the document path and document name of the scenario use case and the document name of the generated result report; Analyze scenario use cases using Python scripts. Group user statements in several scenario use cases according to conversation IDs and send them to the robot program. The robot program analyzes the user statements and obtains return conversations. The return conversations include conversation IDs, extracted elements, and conversation content.
5. The method for automated human-computer dialogue testing and factor analysis based on dialogue content according to claim 4, It is characterized in that The step of judging whether to continue the test according to the returned dialogue includes: Retrieve the conversation ID and conversation content in the returned conversation; obtain the preset conversation table; match the conversation ID with the conversation table to obtain the standard content corresponding to the conversation content; Compare the conversation content with the standard content to determine whether the conversation content is consistent with the standard content; if yes, continue testing; if not, stop testing the corresponding scenario use case.
6. The method for automated human-computer dialogue testing and factor analysis based on dialogue content according to claim 4, It is characterized in that The step of judging whether the extraction of dialogue elements is correct includes: Retrieve the conversation ID and extract elements from the returned conversation, extract the corresponding correct elements according to the conversation ID; integrate the correct elements into an element matching table; The extracted elements are matched with the element matching table in turn to determine whether the extracted elements are consistent with the corresponding correct elements in the element matching table. If so, the dialogue element extraction is determined to be correct; if not, the dialogue element extraction is determined to be incorrect.
7. The method for automated human-computer dialogue testing and factor analysis based on dialogue content according to claim 6, It is characterized in that The element extraction pass rate of each scenario use case analyzed based on the dialogue elements includes: Retrieve the conversation ID and the number of extracted elements in the returned conversation; analyze the extracted elements to obtain the number of correctly marked elements; The extracted elements are divided according to the dialogue ID to obtain the number of elements and the correct number of several scenario use cases; the correct number is divided by the number of elements to obtain the element extraction pass rate corresponding to several scenario use cases.
