Method, device and system for process automation test of intelligent conversation robot

Abstract

The invention provides a process automation test method, device and system for an intelligent dialogue robot, and the method comprises the steps: receiving a large model instruction corresponding to each node in a dialogue test process of the intelligent dialogue robot through a large model engine, carrying out the operation based on each large model instruction, and obtaining a large model output result, and inputting each large model output result into the intelligent dialogue robot, performing intention understanding by the intelligent dialogue robot according to each large model output result, obtaining an intention understanding result, and outputting and storing each intention understanding result. In addition, the correctness of all intention understanding results corresponding to all nodes in the dialogue test process can be checked. Based on the technical scheme provided by the invention, the test execution cost of the intelligent dialogue robot can be remarkably reduced, and the execution accuracy of the test process of the intelligent dialogue robot is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent dialogue robots, and more specifically, to a method, device and system for process automation testing of intelligent dialogue robots. Background Art

[0002] With the rapid development of artificial intelligence technology, various intelligent dialogue robots have emerged. Devices or systems such as intelligent outbound call robots, intelligent inbound call robots, and intelligent text customer service are becoming more and more widely used in people's daily lives, making people's use more convenient.

[0003] In the innovation process of intelligent conversational robots, after creating an intelligent conversational robot, in order to make it more accurate and convenient for people to use, it is usually necessary to test the execution process of the intelligent conversational robot. The test in the prior art usually adopts the method of manually making a phone call to interact with the intelligent conversational robot or simulating text input to interact with the intelligent conversational robot. This interaction method requires a large number of call tests to improve the coverage of the intelligent conversational robot process execution, which is relatively costly in terms of manpower, time, and economy.

[0004] Therefore, there is an urgent need to provide a method for efficiently testing the execution process of an intelligent conversational robot. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and system for automated process testing of an intelligent conversational robot, which adopts large model technology and uses the large model as a virtual user to interact with the intelligent conversational robot, so as to achieve automated testing of the correctness of process execution effect, information extraction results and intent understanding results.

[0006] According to one aspect of the present invention, the present invention provides a method for process automation testing of an intelligent conversational robot, the method comprising the following steps:

[0007] Step 1: The big model engine receives the big model instruction corresponding to each node in the intelligent dialogue robot dialogue test process;

[0008] Step 2: The large model engine performs operations based on each of the large model instructions and obtains a large model output result;

[0009] Step 3: The large model engine transmits each of the large model output results to the intelligent dialogue robot;

[0010] Step 4: The intelligent conversational robot understands the intention according to the output results of each of the large models and obtains the intention understanding result;

[0011] Step 5: The intelligent dialogue robot outputs and stores each intention understanding result;

[0012] Step 6: Verify the correctness of all intent understanding results corresponding to all nodes in the dialogue test process.

[0013] Furthermore, the large model instructions are instructions configured by the tester for the large model engine according to the specific testing purpose of the intelligent dialogue robot.

[0014] Furthermore, the dialogue test process includes one or more nodes, each of which is an interaction in the dialogue process with the intelligent dialogue robot.

[0015] Furthermore, the big model output result output by the big model engine to the intelligent dialogue robot simulates the user's words input manually.

[0016] Furthermore, after receiving the output result of the large model, the intelligent conversational robot performs intention understanding on the output result of the large model based on its own processing logic, and outputs the intention understanding result;

[0017] Furthermore, the intelligent dialogue robot stores the intention understanding result in a memory, and the memory may be a memory existing in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot.

[0018] Furthermore, the verification is performed by manual sampling, and the overall execution accuracy of the dialogue test process is calculated by sampling the intent understanding results corresponding to each of the nodes.

[0019] Furthermore, if an error in the test process execution is found during the verification, first confirm whether it is a process execution configuration error. If it is a process execution configuration error, correct the process execution configuration. If it is not a process execution configuration error, confirm whether the output result of the large model engine is not the expected result. If there is a problem with the output of the large model engine, modify the large model instruction.

[0020] According to another aspect of the present invention, the present invention provides a device for process automation testing of an intelligent conversational robot, the device comprising: a receiving unit, an execution unit, an output unit, a comprehension unit, a storage unit, and a verification unit;

[0021] The receiving unit is used for the large model engine to receive the large model instruction corresponding to each node in the dialogue test process of the intelligent dialogue robot;

[0022] The execution unit is used for the large model engine to perform operations based on each of the large model instructions and obtain a large model output result;

[0023] The output unit is used for the large model engine to transmit each of the large model output results to the intelligent dialogue robot;

[0024] The understanding unit is used for the intelligent dialogue robot to understand the intention according to each of the large model output results and obtain the intention understanding result;

[0025] The storage unit is used for the intelligent dialogue robot to output and store each intention understanding result;

[0026] The verification unit is used to verify the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

[0027] According to another aspect of the present invention, the present invention provides a system for process automation testing of an intelligent dialogue robot, the system comprising: a large model engine, an intelligent dialogue robot and a verification system, wherein:

[0028] The big model engine receives the big model instruction corresponding to each node in the intelligent dialogue robot dialogue test process, performs calculations based on each big model instruction and obtains a big model output result, and inputs each big model output result to the intelligent dialogue robot;

[0029] The intelligent conversational robot understands the intention and obtains the intention understanding result according to each output result of the large model, and outputs and stores each intention understanding result;

[0030] The verification system verifies the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

[0031] According to another aspect of the present invention, an electronic device includes:

[0032] Storage device;

[0033] one or more processors;

[0034] Among them, the storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for process automation testing of an intelligent dialogue robot.

[0035] According to another aspect of the present invention, a computer-readable storage medium stores a computer program, which, when executed, implements the above-mentioned method for process automation testing of an intelligent conversational robot.

[0036] According to another aspect of the present invention, a computer program product includes computer program instructions, which, when executed by a processor, are used to implement the above-mentioned method for process automation testing of an intelligent conversational robot.

[0037] Compared with the prior art, the method, device and system for automated process testing of an intelligent dialogue robot provided by the present invention, wherein the large model engine receives the large model instruction corresponding to each node in the dialogue test process of the intelligent dialogue robot, performs operations based on each of the large model instructions and obtains the large model output result, inputs each of the large model output results into the intelligent dialogue robot, and the intelligent dialogue robot performs intent understanding according to each of the large model output results and obtains the intent understanding result, outputs and stores each of the intent understanding results. In addition, the correctness of all intent understanding results corresponding to all nodes in the dialogue test process can also be verified. Based on the technical solution provided by the present invention, the test execution cost of the intelligent dialogue robot can be significantly reduced, and the execution accuracy of the intelligent dialogue robot test process can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 This is a flow chart of a method according to Embodiment 1 of the present invention;

[0040] Figure 2 This is a schematic diagram of a device according to Embodiment 2 of the present invention;

[0041] Figure 3 This is a system schematic diagram of Embodiment 3 of the present invention;

[0042] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0043] In order to make the problems, technical solutions and technical effects to be solved by the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments and obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0044] Before describing the exemplary embodiments of the present invention in more detail, it should be noted that although the flowchart of the present invention describes the operations as a sequential processing mode, many of the operation steps can be implemented in parallel or simultaneously, and the order of the operations can also be rearranged as needed. In addition, the processing can be terminated when the operation is completed, but there can also be additional steps not included in the drawings.

[0045] Embodiments of the present invention relate to artificial intelligence technologies such as large models. Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Large models, also known as large language models (LLM), refer to deep learning models trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and are an important path to artificial intelligence.

[0046] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0047] Embodiment 1:

[0048] Figure 1 1 is a flow chart of a method for process automation testing of an intelligent conversational robot proposed according to Embodiment 1 of the present invention, the method comprising the following steps S101-S106:

[0049] Step S101: The big model engine receives the big model instructions corresponding to each node in the intelligent dialogue robot dialogue test process.

[0050] In step S101, the large model instruction is received by the large model engine, and the large model instruction is an instruction configured by the tester for the large model engine according to the specific test purpose of the intelligent dialogue robot. For example, if you want to test the interaction of the intelligent dialogue robot with a satisfaction score, this interaction can be used as a node in the dialogue process with the intelligent dialogue robot. If the score that the intelligent dialogue robot can understand is 0-10 points, the large model instruction can be configured for the large model engine. After receiving the large model instruction, the large model engine will output the simulated scoring result and transmit the simulated scoring result to the intelligent dialogue robot for testing.

[0051] The dialogue test process may include one or more nodes, each of which is an interaction with the intelligent dialogue robot in the dialogue process. The process of automated testing of the intelligent dialogue robot process may include tests of various contents and forms. Some test processes may be completed with only one interaction, but some tests may require multiple interactions to complete. Therefore, the entire test process may include multiple nodes, and each node needs to be tested.

[0052] Step S102: the large model engine performs operations based on each of the large model instructions and obtains a large model output result.

[0053] In step S102, after receiving the large model instruction, the large model engine uses the large model algorithm therein to perform calculations to obtain a large model output result.

[0054] Step S103: The large model engine transmits each of the large model output results to the intelligent dialogue robot.

[0055] In step S103, after the large model engine obtains the large model output result through calculation, it transmits the large model output result to the intelligent dialogue robot. In each of the nodes, the large model output result output by the large model engine to the intelligent dialogue robot is a user statement that simulates manual input.

[0056] Step S104: The intelligent dialogue robot understands the intention according to the output results of each of the large models and obtains the intention understanding results.

[0057] In step S104, after receiving the output result of the large model, the intelligent dialogue robot performs intention understanding on the output result of the large model based on its own processing logic, and outputs the intention understanding result output. In addition, the execution path of the dialogue test process is recorded.

[0058] Step S105: The intelligent dialogue robot outputs and stores each intention understanding result.

[0059] In step S105, the intelligent dialogue robot outputs each of the intention understanding results and stores the output intention understanding results in a memory, which may be a memory in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot. The intention understanding results corresponding to all nodes in a dialogue test process are stored together.

[0060] Step S106: verifying the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

[0061] In step S106, the verification can be carried out in a manual sampling manner, by sampling the intent understanding results corresponding to each node, and calculating the overall execution accuracy of the dialogue test process. If a call with an execution error in the test process is found in the manual sampling, first confirm whether it is a process execution configuration error. If it is a process execution configuration error, correct the process execution configuration, and pay attention to whether the accuracy can be improved during the next test. If it is not a process execution configuration error, confirm whether the output result of the large model engine is not the expected result. If there is a problem with the output of the large model engine, modify the large model instruction, and check whether the accuracy can be improved after retesting.

[0062] Embodiment 2:

[0063] Figure 2 2 is a schematic diagram of the composition of a device unit for process automation testing of an intelligent conversational robot according to Embodiment 2 of the present invention. The function performed by the device corresponds to the method in Embodiment 1. The device includes: a receiving unit 201, an execution unit 202, an output unit 203, a comprehension unit 204, a storage unit 205, and a verification unit 206.

[0064] The receiving unit 201 is used for the large model engine to receive the large model instruction corresponding to each node in the intelligent dialogue robot dialogue test process;

[0065] The large model instruction is received by the large model engine, and the large model instruction is an instruction configured by the tester for the large model engine according to the specific test purpose of the intelligent dialogue robot. For example, if you want to test the interaction of the intelligent dialogue robot with a satisfaction score, this interaction can be used as a node in the dialogue process with the intelligent dialogue robot. If the score that the intelligent dialogue robot can understand is 0-10 points, the large model instruction can be configured for the large model engine. After receiving the large model instruction, the large model engine will output the simulated scoring result and transmit the simulated scoring result to the intelligent dialogue robot for testing.

[0066] The dialogue test process may include one or more nodes, each of which is an interaction with the intelligent dialogue robot in the dialogue process. The process of automated testing of the intelligent dialogue robot process may include tests of various contents and forms. Some test processes may be completed with only one interaction, but some tests may require multiple interactions to complete. Therefore, the entire test process may include multiple nodes, and each node needs to be tested.

[0067] The execution unit 202 is used for the large model engine to perform operations based on each of the large model instructions and obtain a large model output result;

[0068] After receiving the large model instruction, the large model engine uses the large model algorithm therein to perform calculations and obtain the large model output result.

[0069] The output unit 203 is used for the large model engine to transmit each of the large model output results to the intelligent dialogue robot;

[0070] After the large model engine obtains the large model output result through calculation, the large model output result is transmitted to the intelligent dialogue robot. Wherein, at each node, the large model output result output by the large model engine to the intelligent dialogue robot is a user statement that simulates manual input.

[0071] The understanding unit 204 is used by the intelligent dialogue robot to understand the intention according to each of the large model output results and obtain the intention understanding result.

[0072] After receiving the output result of the large model, the intelligent dialogue robot performs intention understanding on the output result of the large model based on its own processing logic, and outputs the intention understanding result output. In addition, the execution path of the dialogue test process is recorded.

[0073] The storage unit 205 is used for the intelligent dialogue robot to output and store each intention understanding result;

[0074] The intelligent dialogue robot outputs each of the intention understanding results, and stores the output intention understanding results in a memory, which may be a memory existing in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot. The intention understanding results corresponding to all nodes in a dialogue test process are stored together.

[0075] The checking unit 206 is used to check the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

[0076] The verification can be carried out in a manual sampling manner, by sampling the intent understanding results corresponding to each node, and calculating the overall execution accuracy of the dialogue test process. If a call with an execution error in the test process is found in the manual sampling, first confirm whether it is a process execution configuration error. If it is a process execution configuration error, correct the process execution configuration, and pay attention to whether the accuracy can be improved during the next test. If it is not a process execution configuration error, confirm whether the output result of the large model engine is not the expected result. If there is a problem with the output of the large model engine, modify the large model instruction, and check whether the accuracy can be improved after retesting.

[0077] Embodiment three:

[0078] Embodiment 3 of the present invention provides a system for process automation testing of an intelligent conversational robot. The functions performed by the system correspond to the method in Embodiment 1. The system includes: a large model engine 301, an intelligent conversational robot, and a verification system 302;

[0079] The big model engine 301 receives the big model instruction corresponding to each node in the intelligent dialogue robot dialogue test process, performs operations based on each big model instruction and obtains a big model output result, and transmits each big model output result to the intelligent dialogue robot;

[0080] The intelligent conversational robot 302 performs intention understanding and obtains intention understanding results according to each of the large model output results, and outputs and stores each of the intention understanding results;

[0081] The verification system 303 verifies the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

[0082] In addition, other embodiments of the present invention further provide an electronic device, which includes a storage device and one or more processors, wherein the storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as in Embodiment 1.

[0083] It should be noted that a part of the present invention can be applied as a computer program product, such as a computer program instruction, when it is executed by an intelligent electronic device (such as a smart mobile phone or a tablet computer, etc.), the method and / or technical solution according to the present invention can be called or provided through the operation of the intelligent electronic device. The program instruction for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of an intelligent electronic device that operates according to the program instruction. Here, according to an embodiment of the present invention, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present invention.

[0084] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which aspect, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the description of the above embodiments, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for process automation testing of an intelligent conversational robot, characterized in that: The method comprises the following steps: Step S101: The big model engine receives the big model instruction corresponding to each node in the intelligent dialogue robot dialogue test process; Step S102: the large model engine performs operations based on each of the large model instructions and obtains a large model output result; Step S103: the large model engine transmits each of the large model output results to the intelligent dialogue robot; Step S104: the intelligent dialogue robot understands the intention according to the output results of each large model and obtains the intention understanding result; Step S105: the intelligent dialogue robot outputs and stores each intention understanding result; Step S106: verifying the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

2. A method for process automation testing of an intelligent conversational robot according to claim 1, characterized in that: The dialogue test process includes one or more nodes, each of which is an interaction in the dialogue process with the intelligent dialogue robot.

3. A method for process automation testing of an intelligent conversational robot according to claim 2, characterized in that: The big model output result output by the big model engine to the intelligent dialogue robot is a user statement that simulates manual input.

4. The method for process automation testing of an intelligent conversational robot according to claim 1, characterized in that: The step S104 is specifically as follows: After receiving the output result of the large model, the intelligent dialogue robot performs intent understanding on the output result of the large model based on its own processing logic, and outputs the intent understanding result.

5. The method for process automation testing of an intelligent conversational robot according to claim 1, characterized in that: The verification is carried out by manual sampling, and the overall execution accuracy of the dialogue test process is calculated by sampling the intent understanding results corresponding to each node.

6. A method for process automation testing of an intelligent conversational robot according to claim 1 or 5, characterized in that: In the step S106, if an error in the execution of the test process is found during the verification, first confirm whether it is a process execution configuration error. If it is a process execution configuration error, correct the process execution configuration. If it is not a process execution configuration error, confirm whether the output result of the large model engine is not the expected result. If there is a problem with the output of the large model engine, modify the large model instruction.

7. A device for process automation testing of an intelligent conversational robot, characterized in that: The device comprises: a receiving unit, an executing unit, an output unit, a comprehension unit, a storage unit and a checking unit; The receiving unit is used for the large model engine to receive the large model instruction corresponding to each node in the dialogue test process of the intelligent dialogue robot; The execution unit is used for the large model engine to perform operations based on each of the large model instructions and obtain a large model output result; The output unit is used for the large model engine to transmit each of the large model output results to the intelligent dialogue robot; The understanding unit is used for the intelligent dialogue robot to understand the intention according to each of the large model output results and obtain the intention understanding result; The storage unit is used for the intelligent dialogue robot to output and store each intention understanding result; The verification unit is used to verify the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

8. A system for process automation testing of an intelligent conversational robot, the system comprising: Large model engine, intelligent dialogue robot and verification system, characterized by: The big model engine receives the big model instruction corresponding to each node in the intelligent dialogue robot dialogue test process, performs calculations based on each of the big model instructions and obtains a big model output result, and inputs each of the big model output results to the intelligent dialogue robot; The intelligent conversational robot understands the intention and obtains the intention understanding result according to each output result of the large model, and outputs and stores each intention understanding result; The verification system verifies the correctness of all intention understanding results corresponding to all nodes in the dialogue test process.

9. An electronic device, characterized in that: The device comprises: Storage device; one or more processors; Wherein, the storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method for process automation testing of an intelligent conversational robot as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed, implements a method for process automation testing of an intelligent conversational robot as described in any one of claims 1-6.

Images