Method, device and system for testing intention understanding accuracy of intelligent dialogue robot
Through large-scale model technology, the intent understanding accuracy of intelligent dialogue robots is automatically tested, which solves the problem of poor intent understanding accuracy and recall rates caused by relying on historical calls in the existing technology, and achieves more efficient intent judgment and service quality improvement.
Patent Information
- Application Number
- CN202411219129.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-27
AI Technical Summary
The prior art relies on historical call flows when testing the intent-understanding accuracy of intelligent dialogue robots, resulting in poor in intention-understanding and recall when the business is just launched, which is time-consuming and labor-intensive and incomplete coverage.
Using big model technology, the big model engine receives intent-related instructions, generates output results and transmits them to the intelligent dialogue robot. The intelligent dialogue robot performs intent understanding and statistics intent understanding accuracy, frees up the manpower and time to collect historical call users' statements, and improves intent Understand the coverage and accuracy of judgment.
The intention judgment ability and service comfort of intelligent dialogue robots are improved, and the accuracy and recall rate of intention understanding are improved through automated testing processes, and labor and time costs are reduced.
Smart Images
Figure CN119201693A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent dialogue robots, and more specifically, to a method, device and system for testing the accuracy of intention understanding of an intelligent dialogue robot. Background Art
[0002] With the rapid development of artificial intelligence technology, various intelligent dialogue robots have emerged. Devices or systems such as intelligent outbound call robots, intelligent inbound call robots, and intelligent text customer service are becoming more and more widely used in people's daily lives, making people's use more convenient.
[0003] In the process of innovation of intelligent conversational robots, after creating an intelligent conversational robot, in order to make it more accurate and convenient for people to use, it is usually necessary to test the accuracy of the intelligent conversational robot's intention understanding. The testing method of the prior art is usually for business personnel to count all user statements in the historical call flow, sort them out and submit them to the intelligent conversational robot as the test basis for the intelligent conversational robot's intention understanding results. This method needs to rely on historical call flow. When the business is just launched and there are not many historical calls, it often results in poor accuracy and recall of intention understanding judgment.
[0004] Therefore, there is an urgent need to provide a method for testing the accuracy of intent understanding of intelligent conversational robots. Summary of the invention
[0005] In view of the problems of insufficient testing of the intention understanding of intelligent dialogue robots and time-consuming, labor-intensive and incomplete coverage caused by collecting historical call flow to test the accuracy, the present invention provides a method, device and system for testing the accuracy of intention understanding of intelligent dialogue robots. The large model technology is adopted to input the intention-related statements output by the large model into the intelligent dialogue robot, and after the intelligent dialogue robot outputs the corresponding understanding results, the accuracy of intention understanding is statistically calculated. Through the scheme of the present invention, the manpower and time of collecting historical call user statements can be released, the coverage, accuracy and recall rate of the intention understanding judgment test can be improved, and then the intention judgment ability of the intelligent dialogue robot can be improved, and the service comfort can be improved.
[0006] According to one aspect of the present invention, the present invention provides a method for testing the accuracy of intention understanding of an intelligent conversational robot, the method comprising the following steps:
[0007] Step S101: The large model engine receives an intention-related instruction for testing the accuracy of intention understanding of the intelligent conversational robot;
[0008] Step S102: the large model engine performs operations based on each of the intention-related instructions and obtains output results corresponding to the intention-related instructions;
[0009] Step S103: the large model engine transmits the output result corresponding to the intention-related instruction to the intelligent dialogue robot;
[0010] Step S104: the intelligent dialogue robot uses its own understanding ability to understand the intention of the output result and obtains the intention understanding result;
[0011] Step S105: the intelligent dialogue robot extracts and stores information in the intention understanding result;
[0012] Step S106: Counting the accuracy of the intention understanding.
[0013] Furthermore, the intention-related instruction is a large model Prompt instruction.
[0014] Furthermore, the step S104 is specifically as follows: after receiving the output result corresponding to the intention-related instruction, the intelligent dialogue robot performs intention understanding on the output result corresponding to the intention-related instruction based on its own processing logic and obtains the intention understanding result.
[0015] Furthermore, the step S105 is specifically as follows: the intelligent dialogue robot extracts key information in the intention understanding and stores the key information in a memory, and the memory may be a memory existing in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot.
[0016] Furthermore, in the step S106, the information is obtained through a statistical device and the number of the intention understanding accuracy test and the intention understanding results are automatically counted, and the accuracy of the intention understanding is calculated based on the number of the intention understanding accuracy test and the intention understanding results.
[0017] Furthermore, in the step S106, the accuracy of the intention understanding can be statistically analyzed by manual sampling.
[0018] According to another aspect of the present invention, the present invention provides a device for testing the accuracy of intention understanding of an intelligent conversational robot, the device comprising: a receiving unit, an execution unit, an output unit, an understanding unit, a storage unit and a statistical unit;
[0019] The receiving unit is used for the large model engine to receive intention-related instructions for the intention understanding accuracy test of the intelligent dialogue robot;
[0020] The execution unit is used for the large model engine to perform operations based on each of the intention-related instructions and obtain output results corresponding to the intention-related instructions;
[0021] The output unit is used for the large model engine to transmit the output result corresponding to the intention-related instruction to the intelligent dialogue robot;
[0022] The understanding unit is used for the intelligent dialogue robot to call its own understanding ability to understand the intention of the output result and obtain the intention understanding result;
[0023] The storage unit is used for the intelligent dialogue robot to extract and store information in the intention understanding result;
[0024] The statistical unit is used to count the accuracy of the intention understanding.
[0025] According to another aspect of the present invention, the present invention provides a system for testing the accuracy of intention understanding of an intelligent dialogue robot, the system comprising: a large model engine, an intelligent dialogue robot and a statistical device, wherein:
[0026] The large model engine is used to receive intention-related instructions for the intention understanding accuracy test of the intelligent dialogue robot, perform operations based on each of the intention-related instructions and obtain output results corresponding to the intention-related instructions, and transmit the output results corresponding to the intention-related instructions to the intelligent dialogue robot;
[0027] The intelligent conversational robot is used to call upon its own understanding ability to understand the intention of the output result and obtain an intention understanding result, and to extract and store information in the intention understanding result;
[0028] The statistical device is used to count the accuracy of the intention understanding.
[0029] According to another aspect of the present invention, an electronic device includes:
[0030] Storage device;
[0031] one or more processors;
[0032] Among them, the storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for testing the accuracy of intention understanding of an intelligent dialogue robot.
[0033] According to another aspect of the present invention, a computer-readable storage medium stores a computer program thereon, which, when executed, implements the above-mentioned method for testing the accuracy of intention understanding of an intelligent conversational robot.
[0034] According to another aspect of the present invention, a computer program product includes computer program instructions, which, when executed by a processor, are used to implement the above-mentioned method for testing the accuracy of intention understanding of an intelligent conversational robot.
[0035] Compared with the prior art, the present invention provides a method, device and system for testing the accuracy of intention understanding of an intelligent dialogue robot, wherein the large model engine receives intention-related instructions for testing the accuracy of intention understanding of the intelligent dialogue robot; performs operations based on each of the intention-related instructions and obtains the output results corresponding to the intention-related instructions; transmits the output results corresponding to the intention-related instructions to the intelligent dialogue robot; the intelligent dialogue robot calls on its own understanding ability to understand the intention of the output result and obtains the intention understanding result; the intelligent dialogue robot extracts and stores the information in the intention understanding result; and counts the accuracy of the intention understanding. Based on the technical solution provided by the present invention, it is possible to free up manpower and time for collecting historical call user statements, improve the coverage, accuracy and recall rate of the intention understanding judgment test, and thus improve the intention judgment ability of the intelligent dialogue robot and improve service comfort. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0037] Figure 1 This is a flow chart of a method according to Embodiment 1 of the present invention;
[0038] Figure 2 This is a schematic diagram of a device according to Embodiment 2 of the present invention;
[0039] Figure 3 This is a system schematic diagram of Embodiment 3 of the present invention;
[0040] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION
[0041] In order to make the problems, technical solutions and technical effects to be solved by the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments and obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0042] Before describing the exemplary embodiments of the present invention in more detail, it should be noted that although the flowchart of the present invention describes the operations as a sequential processing mode, many of the operation steps can be implemented in parallel or simultaneously, and the order of the operations can also be rearranged as needed. In addition, the processing can be terminated when the operation is completed, but there can also be additional steps not included in the drawings.
[0043] Embodiments of the present invention relate to the fields of artificial intelligence technology such as large models and the field of intelligent dialogue robot technology. Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Large models, also known as large language models (LLM), refer to deep learning models trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and are an important path to artificial intelligence.
[0044] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.
[0045] Embodiment 1:
[0046] Figure 1 1 is a flow chart of a method for testing the accuracy of intention understanding of an intelligent conversational robot according to Embodiment 1 of the present invention. The method comprises the following steps S101-S106:
[0047] Step S101: The large model engine receives intent-related instructions for testing the accuracy of intent understanding of the intelligent conversational robot.
[0048] In step S101, there are many intent understanding accuracy tests for the intelligent conversational robot, and for each intent understanding accuracy test, a corresponding intent-related instruction needs to be input into the big model engine. The intent-related instruction is a big model Prompt instruction, and the big model engine can be ChatGPT, Wenxin Yiyan or other self-developed big models.
[0049] Step S102: The large model engine performs operations based on each of the intention-related instructions and obtains output results corresponding to the intention-related instructions.
[0050] In step S102, after receiving each of the intention-related instructions, the big model engine uses the big model algorithm therein to perform operations to obtain the output result corresponding to the intention-related instructions. The output result is the intention-related user statement output by the big model engine after the operation based on the intention-related instructions.
[0051] Step S103: The large model engine transmits the output result corresponding to the intention-related instruction to the intelligent dialogue robot.
[0052] In the step S103, after the large model engine obtains the output result corresponding to the intention-related instruction through calculation, the output result corresponding to the intention-related instruction is transmitted to the intelligent dialogue robot.
[0053] Step S104: The intelligent dialogue robot uses its own understanding ability to understand the intention of the output result and obtain an intention understanding result.
[0054] In step S104, after receiving the output result corresponding to the intention-related instruction, the intelligent dialogue robot performs intention understanding on the output result corresponding to the intention-related instruction based on its own processing logic and obtains the intention understanding result. In this step, the intelligent dialogue robot calculates and obtains the intention understanding result corresponding to all user statements.
[0055] Step S105: The intelligent dialogue robot extracts and stores information from the intention understanding result.
[0056] In step S105, after the intelligent dialogue robot understands the intention, the intelligent dialogue robot extracts key information in the intention understanding and stores the key information in a memory. The memory can be a memory existing in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot.
[0057] Step S106: Counting the accuracy of the intention understanding.
[0058] In step S106, the information can be obtained through a special statistical device and the number of the intent understanding accuracy test and the intent understanding result can be automatically counted, and the accuracy of the intent understanding can be calculated based on the number of the intent understanding accuracy test and the intent understanding result. The above process can also be completed by manual sampling. In addition, the recall rate of the intent understanding result can also be counted.
[0059] Embodiment 2:
[0060] Figure 220 is a schematic diagram of the unit composition of a device for testing the accuracy of intention understanding of an intelligent conversational robot according to Embodiment 2 of the present invention. The function performed by the device corresponds to the method in Embodiment 1. The device includes: a receiving unit 201, an execution unit 202, an output unit 203, an understanding unit 204, a storage unit 205, and a statistical unit 206.
[0061] The receiving unit 201 is used for the large model engine to receive intention-related instructions for the intention understanding accuracy test of the intelligent dialogue robot.
[0062] There will be many intent understanding accuracy tests for the intelligent conversational robot, and for each intent understanding accuracy test, the corresponding intent-related instructions need to be input into the big model engine. The intent-related instructions are big model Prompt instructions, and the big model engine can be ChatGPT, Wenxin Yiyan or other self-developed big models.
[0063] The execution unit 202 is used for the large model engine to perform operations based on each of the intention-related instructions and obtain output results corresponding to the intention-related instructions.
[0064] After receiving each of the intention-related instructions, the large model engine uses the large model algorithm therein to perform operations to obtain the output results corresponding to the intention-related instructions. The output results are the intention-related user statements output by the large model engine after the operations based on the intention-related instructions.
[0065] The output unit 203 is used by the large model engine to transmit the output result corresponding to the intention-related instruction to the intelligent dialogue robot.
[0066] After the large model engine obtains the output result corresponding to the intention-related instruction through calculation, the output result corresponding to the intention-related instruction is transmitted to the intelligent dialogue robot.
[0067] The understanding unit 204 is used for the intelligent dialogue robot to call its own understanding ability to understand the intention of the output result and obtain the intention understanding result.
[0068] After receiving the output result corresponding to the intention-related instruction, the intelligent dialogue robot performs intention understanding on the output result corresponding to the intention-related instruction based on its own processing logic and obtains the intention understanding result. In this step, the intelligent dialogue robot calculates and obtains the intention understanding result corresponding to all user statements.
[0069] The storage unit 205 is used by the intelligent dialogue robot to extract and store information in the intention understanding result.
[0070] After the intelligent dialogue robot understands the intention, the intelligent dialogue robot extracts key information in the intention understanding and stores the key information in a memory. The memory can be a memory existing in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot.
[0071] The statistical unit 206 is used to count the accuracy of the intention understanding.
[0072] The information can be obtained through a special statistical device and the number of the intent understanding accuracy test and the intent understanding result can be automatically counted, and the accuracy of the intent understanding can be calculated based on the number of the intent understanding accuracy test and the intent understanding result. The above process can also be completed by manual sampling. In addition, the recall rate of the intent understanding result can also be counted.
[0073] Embodiment three:
[0074] Embodiment 3 of the present invention provides a system for testing the accuracy of intention understanding of an intelligent conversational robot. The functions performed by the system correspond to the method in Embodiment 1. The system includes: a large model engine 301, an intelligent conversational robot 302, and a statistical device 303;
[0075] The large model engine 301 is used to receive intention-related instructions for the intention understanding accuracy test of the intelligent dialogue robot, perform operations based on each of the intention-related instructions and obtain output results corresponding to the intention-related instructions, and transmit the output results corresponding to the intention-related instructions to the intelligent dialogue robot;
[0076] The intelligent conversational robot 302 is used to use its own understanding ability to understand the intention of the output result and obtain the intention understanding result, and extract and store the information in the intention understanding result;
[0077] The statistical device 303 is used to count the accuracy of the intention understanding.
[0078] In addition, other embodiments of the present invention further provide an electronic device, which includes a storage device and one or more processors, wherein the storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as in Embodiment 1.
[0079] It should be noted that a part of the present invention can be applied as a computer program product, such as a computer program instruction, when it is executed by an intelligent electronic device (such as a smart mobile phone or a tablet computer, etc.), the method and / or technical solution according to the present invention can be called or provided through the operation of the intelligent electronic device. The program instruction for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of an intelligent electronic device that operates according to the program instruction. Here, according to an embodiment of the present invention, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present invention.
[0080] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which aspect, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the description of the above embodiments, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
Claims
1. A method for testing the accuracy of intention understanding of an intelligent conversational robot, characterized in that: The method comprises the following steps: Step S101: The large model engine receives an intention-related instruction for testing the accuracy of intention understanding of the intelligent conversational robot; Step S102: the large model engine performs operations based on each of the intention-related instructions and obtains output results corresponding to the intention-related instructions; Step S103: the large model engine transmits the output result corresponding to the intention-related instruction to the intelligent dialogue robot; Step S104: the intelligent dialogue robot uses its own understanding ability to understand the intention of the output result and obtains the intention understanding result; Step S105: the intelligent dialogue robot extracts and stores information in the intention understanding result; Step S106: Counting the accuracy of the intention understanding.
2. A method for testing the accuracy of intention understanding of an intelligent conversational robot according to claim 1, characterized in that: The intent-related instruction is a large model Prompt instruction.
3. The method for testing the accuracy of intention understanding of an intelligent conversational robot according to claim 1, characterized in that: The step S104 is specifically as follows: After receiving the output result corresponding to the intention-related instruction, the intelligent dialogue robot performs intent understanding on the output result corresponding to the intention-related instruction based on its own processing logic and obtains the intention understanding result.
4. The method for testing the accuracy of intention understanding of an intelligent conversational robot according to claim 1, characterized in that: The step S105 specifically includes: The intelligent dialogue robot extracts key information in intent understanding and stores the key information in a memory, which may be a memory existing in the intelligent dialogue robot or a memory independent of the intelligent dialogue robot.
5. The method for testing the accuracy of intention understanding of an intelligent conversational robot according to claim 1, characterized in that: In the step S106, the information is obtained through a statistical device and the number of intention understanding accuracy tests and the intention understanding results are automatically counted, and the accuracy of the intention understanding is calculated based on the number of intention understanding accuracy tests and the intention understanding results.
6. The method for testing the accuracy of intention understanding of an intelligent conversational robot according to claim 1, characterized in that: In the step S106, the accuracy of the intention understanding may be calculated by manual sampling.
7. A device for testing the accuracy of intention understanding of an intelligent conversational robot, characterized in that: The device comprises: a receiving unit, an executing unit, an output unit, a comprehension unit, a storage unit and a statistical unit; The receiving unit is used for the large model engine to receive intention-related instructions for the intention understanding accuracy test of the intelligent dialogue robot; The execution unit is used for the large model engine to perform operations based on each of the intention-related instructions and obtain output results corresponding to the intention-related instructions; The output unit is used for the large model engine to transmit the output result corresponding to the intention-related instruction to the intelligent dialogue robot; The understanding unit is used for the intelligent dialogue robot to call its own understanding ability to understand the intention of the output result and obtain the intention understanding result; The storage unit is used for the intelligent dialogue robot to extract and store information in the intention understanding result; The statistical unit is used to count the accuracy of the intention understanding.
8. A system for testing the accuracy of intention understanding of an intelligent conversational robot, the system comprising: Large model engine, intelligent dialogue robot and statistical equipment, characterized by: The large model engine is used to receive intention-related instructions for the intention understanding accuracy test of the intelligent dialogue robot, perform operations based on each of the intention-related instructions and obtain output results corresponding to the intention-related instructions, and transmit the output results corresponding to the intention-related instructions to the intelligent dialogue robot; The intelligent conversational robot is used to call upon its own understanding ability to understand the intention of the output result and obtain an intention understanding result, and to extract and store information in the intention understanding result; The statistical device is used to count the accuracy of the intention understanding.
9. An electronic device, characterized in that: The device comprises: Storage device; one or more processors; Wherein, the storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a method for testing the accuracy of intention understanding of an intelligent dialogue robot as described in any one of claims 1-6.
10. A computer-readable storage medium having a computer program stored thereon, which, when executed, implements a method for testing the accuracy of intention understanding of an intelligent conversational robot as described in any one of claims 1-6.