A test method, device and equipment

Abstract

The embodiments of this specification provide a testing method, device and equipment that can be used in the field of artificial intelligence. The method is applied to an automated testing system, in which test cases containing test scenario contextual relationships are stored, and the method includes: receiving a first voice stream broadcast by the system under test; obtaining a first expected test result and a first response content from the test case; verifying the first voice stream and the first expected test result, and recording the verification result; sending the first response content to the system under test; the first response content is used to enable the system under test to perform the next round of interaction; determining whether the context in the test case has been tested; and obtaining all verification results when the test is completed. The embodiments of this specification can be used to implement automated testing of intelligent customer service robots and improve testing efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a testing method, device and equipment. Background Art

[0002] With the continuous development of technology, the use of intelligent customer service robots has become more and more common in all walks of life. However, intelligent customer service robots need to be tested before delivery so that they can meet business needs.

[0003] In the prior art, the test of intelligent customer service robots is mainly conducted by testers submitting outbound call tasks for a certain business scenario in the intelligent customer service robots, and then the intelligent customer service robots make calls to the testers for interaction. This method can only be done manually. When a large number of intelligent customer service robots need to be tested, there is not enough manpower to make calls at the same time, so stress testing cannot be performed, resulting in low test efficiency.

[0004] Therefore, the industry is in urgent need of a technical solution that can solve the above technical problems. Summary of the invention

[0005] The embodiments of this specification provide a testing method, device and equipment, which can realize automated testing of intelligent customer service robots and improve testing efficiency.

[0006] The test method, device and equipment provided in this specification are implemented in the following ways.

[0007] A testing method is applied to an automated testing system, wherein the automated testing system stores test cases containing test scenario contextual relationships, the method comprising: receiving a first voice stream broadcast by a system under test; obtaining a first expected test result and a first response content from the test case; verifying the first voice stream and the first expected test result, and recording the verification result; sending the first response content to the system under test; the first response content is used to enable the system under test to perform a next round of interaction; determining whether the context in the test case has been tested; and obtaining all verification results when the test is completed.

[0008] A testing device includes: a receiving module for receiving a first voice stream broadcast by a system under test; a first acquisition module for acquiring a first expected test result and a first response content from the test case; a verification module for verifying the first voice stream and the first expected test result, and recording the verification result; a sending module for sending the first response content to the system under test; the first response content is used to enable the system under test to perform a next round of interaction; a judgment module for judging whether the context in the test case has been tested; and a second acquisition module for obtaining all verification results when the test is completed.

[0009] A testing device includes at least one processor and a memory storing computer executable instructions, wherein when the processor executes the instructions, the steps of any method embodiment in the embodiments of this specification are implemented.

[0010] A computer-readable storage medium stores computer instructions, which, when executed, implement the steps of any method embodiment in the embodiments of this specification.

[0011] This specification provides a test method, device and equipment. In some embodiments, the voice stream broadcast by the system under test can be received, the expected test results and response content can be obtained from the test case, and then the voice stream and the expected test results can be verified, and the verification results can be recorded. Furthermore, the response content can be sent to the system under test so that the system under test performs the next round of interaction according to the response content until the context test in the test case is completed and all verification results are obtained. Since the present application is to realize the automated testing of the intelligent customer service voice robot system containing contextual relationships by building an automated testing system based on voice streams, it can reduce manpower input and improve testing efficiency. The implementation scheme provided in this specification can be used to realize automated testing of intelligent customer service robots and improve testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of this specification, constitute a part of this specification, and do not constitute a limitation of this specification. In the drawings:

[0013] Figure 1 It is a flow chart of an embodiment of a testing method provided in this specification;

[0014] Figure 2 It is a schematic diagram of a specific embodiment of a test method provided in this specification;

[0015] Figure 3 It is a schematic diagram of the module structure of an embodiment of a testing device provided in this specification;

[0016] Figure 4This is a hardware structure block diagram of an embodiment of a test server provided in this specification. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments in this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of this specification.

[0018] The following is an example of a specific application scenario to illustrate the implementation plan of this specification. Specifically, Figure 1 It is a flow chart of an embodiment of a test method provided in this specification. Although this specification provides the method operation steps or device structure shown in the following embodiments or drawings, the method or device may include more or fewer operation steps or module units after partial combination based on routine or no creative labor.

[0019] An implementation scheme provided in this specification can be applied to a client, a server, etc. The client can include a terminal device, such as a smart phone, a tablet computer, etc. The server can include a single computer device, or a server cluster composed of multiple servers, or a server structure of a distributed system, etc.

[0020] It should be noted that the following embodiment description does not limit the technical solutions in other application scenarios that can be extended based on this specification. Figure 1 As shown, an embodiment of a testing method provided in this specification can be applied to an automated testing system, wherein the automated testing system stores test cases containing test scenario context relationships. The method may include the following steps.

[0021] S0: Receive the first voice stream broadcast by the system under test.

[0022] Among them, the system under test can be an intelligent customer service voice robot or a system that needs to test voice functions.

[0023] In some embodiments, before the automated testing system receives the first voice stream broadcast by the system under test, it may first send a task creation request to the system under test so that the system under test starts a call task.

[0024] In some implementation scenarios, the automated test system can be triggered automatically or manually to send a task creation request to the system under test. For example, in some implementation scenarios, the test time can be set in advance in the automated test system, so that the automated test system can be automatically triggered to send a task creation request to the system under test when the test time is reached. In other implementation scenarios, the automated test system can be manually triggered to send a task creation request to the system under test by the tester clicking a preset button on the automated test system. Of course, the above is only an exemplary description, and the method of triggering the automated test system to send a task creation request to the system under test is not limited to the above example. Technical personnel in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0025] In some implementation scenarios, after receiving the task creation request sent by the automated test system, the system under test can start the call task and broadcast the voice stream to the automated test system. For example, in some implementation scenarios, the automated test system can send a task creation request to the voice robot under test according to the test case, start the call task, and then the voice robot broadcasts the voice stream according to the preset logic.

[0026] In some implementation scenarios, the system under test may send a task creation request to the automated testing system, wherein the task creation request includes a first voice stream broadcast by the system under test.

[0027] In some implementation scenarios, the first voice stream may be inherent to the system under test and may not be reflected in the test case. For example, for an outbound call task, an opening statement will be broadcast after the call is connected, such as "Are you xxx?" At this time, "Are you xxx?" can be understood as the first voice stream. Of course, the above is only an exemplary explanation, and the first voice stream is not limited to the above examples. Technical personnel in the relevant field may make other changes based on the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0028] S2: Obtain a first expected test result and a first response content from the test case.

[0029] In an embodiment of the present specification, after receiving the first voice stream broadcast by the system under test, the first expected test result and the first response content can be obtained from the test case. Among them, the expected test result can be used to verify whether the received voice stream is accurate, that is, whether the voice stream broadcast by the system under test is accurate. In some implementation scenarios, verifying whether the received voice stream is accurate may include verifying whether the information content meets preset conditions, verifying whether the logic jump meets preset rules, etc. Among them, the preset conditions may include whether the similarity of the information content reaches a preset value, such as 85%, 90%, etc. The preset rules may include whether the logic jump meets a preset logical order, etc. The response content can be used to enable the system under test to enter the next round of interaction.

[0030] In some embodiments, test cases containing test scenario contexts are pre-stored in the automated testing system. In some implementation scenarios, since the tester is familiar with each scenario process in the system under test, before testing the system under test, the tester can prepare different coverage cases for different processes (or scenarios), and then store them in advance in the automated testing system. The coverage cases prepared for different processes (or scenarios) can be understood as test cases containing test scenario contexts. It should be noted that each group of test cases can be understood as a situation for the tested scenario of the system under test. Every time the automated testing system receives a sentence (i.e., voice stream) from the system under test, it can take out an answer (i.e., response content) and an expected test result from the test case.

[0031] For example, in some implementation scenarios, the first voice stream received by the automated testing system is question 1, the first expected test result obtained from the test case is expected test result 1, and the first response content is answer 1. At this time, expected test result 1 can be used to verify whether question 1 broadcast by the system under test is accurate, and answer 1 can be used for subsequent interaction with the system under test.

[0032] In some embodiments, the automated testing system may obtain the first expected test result and the first response content from the test case according to the contextual logical order in the test case. Of course, the above is only an exemplary description, and the method of obtaining the first expected test result and the first response content is not limited to the above example. Technicians in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0033] S4: Verify the first voice stream and the first expected test result, and record the verification result.

[0034] In some embodiments, after obtaining the first expected test result and the first response content from the test case, the first voice stream and the first expected test result can be verified and the verification result can be recorded. The verification result can include whether the information content meets the preset conditions, whether the logic jump meets the preset rules, etc.

[0035] In some embodiments, verifying the first voice stream against the first expected test result and recording the verification result may include: converting the first voice stream into a first text; verifying the first text against the first expected test result and recording the verification result.

[0036] In some implementation scenarios, since the expected test results obtained from the test case are stored in text information, the received voice stream can be converted into text by voice recognition before verification with the received voice stream, and then the converted text can be compared with the expected test results. Of course, in some implementation scenarios, after receiving the voice stream sent by the system under test, it can be converted into text by voice recognition, and then the expected test results and response content can be obtained from the test case accordingly. The technical personnel in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0037] In some embodiments, verifying the first text with the first expected test result and recording the verification result may include: comparing whether the logical jump between the first text and the first expected test result meets preset rules; comparing whether the content included in the first text and the first expected test result meets preset conditions.

[0038] In some implementation scenarios, the comparison between the first text and the first expected test result to determine whether the logic jump satisfies the preset rule can be performed by obtaining the corresponding logic structure and comparing whether the logic structure is consistent, thereby determining whether the logic jump satisfies the preset rule. The preset rule can be set according to the actual scenario, and this specification does not limit this.

[0039] In some implementation scenarios, the first text is compared with the first expected test result to see whether the content meets the preset conditions. The similarity between the two can be calculated, and then it can be determined whether the similarity meets the preset conditions. Among them, the preset condition can be a preset similarity, such as 85%, 90%, etc., which is not limited in this specification. Of course, the above is only an exemplary description, and the method of verifying the first text and the first expected test result is not limited to the above example. Technical personnel in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0040] In some embodiments, when the content included in the first text and the first expected test result meets the preset conditions, it can be said that the content broadcast by the tested system is relatively accurate, and the recorded verification result can be "the content broadcast by the first voice stream is accurate". Otherwise, it means that the content broadcast by the tested system is inaccurate, and the recorded verification result can be "the content broadcast by the first voice stream is inaccurate". When the logical jump between the first text and the first expected test result meets the preset rules, it can be said that the content broadcast by the tested system meets the preset jump logic, and the recorded verification result can be "the first voice stream satisfies the logical jump". Otherwise, the recorded verification result can be "the first voice stream does not meet the logical jump". Of course, the above is only an exemplary description, and the recording method of the verification results is not limited to the above examples. Technical personnel in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0041] S6: Send the first response content to the system under test; the first response content is used to enable the system under test to perform a next round of interaction.

[0042] In the embodiment of the present specification, after verifying the first voice stream and the first expected test result and recording the verification result, the first response content can be sent to the system under test so that the system under test can perform the next round of interaction.

[0043] In some embodiments, before sending the first response content to the system under test, the first response content may be converted into a first response voice stream, and then the first response voice stream may be sent to the system under test.

[0044] In order to improve the interaction efficiency, in some implementation scenarios, before sending the response content to the system under test, it can be converted into a voice stream and then sent to the system under test.

[0045] In some implementation scenarios, the automated test system may include a voice conversion module, so that the response content can be converted into a voice stream using the voice conversion module and then sent to the system under test.

[0046] S8: Determine whether the context in the test case has been tested.

[0047] S10: When the test is completed, obtain all verification results.

[0048] In some embodiments, after the first response content is sent to the system under test, it can be determined whether the context in the test case has been tested.

[0049] In some embodiments, when it is determined that the test is complete, all verification results may be obtained.

[0050] In some embodiments, when it is determined that the test is not completed, steps S0-S8 may be repeatedly performed until the context of the entire test case has completed the test and all verification results have been obtained.

[0051] For example, in some implementation scenarios, when it is determined that the test is not completed, a second voice stream broadcast by the system under test can be received; accordingly, a second expected test result and a second response content can be obtained from the test case; the second voice stream and the second expected test result are verified, and the verification result is recorded; the second response content is sent to the system under test; the second response content is used to enable the system under test to perform the next round of interaction; it is determined whether the context in the test case has been tested; and when the test is completed, all verification results are obtained.

[0052] In some embodiments, after all verification results are obtained, statistics may be collected on all verification results to obtain the test results of the system under test in the scenario.

[0053] In some embodiments, after all verification results are obtained, a request to view the test results may also be received; and the test results may be displayed.

[0054] In some embodiments, after the test results are displayed, the tester can check them to ensure the accuracy of the test.

[0055] In the embodiments of this specification, an automated testing system can be built based on container technology, which can support large-scale concurrency, thereby achieving stress testing of the intelligent customer service voice robot system (conducting a large number of concurrent interactions at the same time, such as hundreds or thousands).

[0056] The embodiments of this specification can efficiently perform large-scale stress testing based on voice streams.

[0057] In the embodiments of this specification, the automated test system and the system under test interact via voice streams, and there is no mutual interference between different calls, thereby solving the problem of mutual interference between multiple calls in voice interaction mode and providing the prerequisite for multiple calls to be concurrent.

[0058] In the embodiments of this specification, an automated testing system is built based on containerization technology, which can facilitate the rapid deployment of multiple systems and implement stress testing of the intelligent customer service voice robot system.

[0059] Of course, the above is only an exemplary description, and the embodiments of this specification are not limited to the above examples. Technical personnel in the relevant field may make other changes based on the technical essence of this application. However, as long as the functions and effects achieved are the same or similar to those of this application, they should be included in the scope of protection of this application.

[0060] The above test method is explained below with a specific implementation scenario. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application. Among them, the automated testing device based on voice stream can be understood as an automated testing system, and the system under test can be an intelligent customer service voice robot. The automated testing device based on voice stream may include a test case library, a voice conversion module, a voice recognition module, and a test result collection module. Among them, the test case library is used to store coverage cases (test cases) in different scenarios, the voice conversion module can be used to convert the response content into a voice stream, the voice recognition module can be used to convert the voice stream into text, and the test result collection module can be used to record the verification results. The automated testing device based on voice stream can be built based on container technology and supports large-scale concurrency to achieve stress testing of the intelligent customer service voice robot. As Figure 2 As shown, Figure 2 It is a schematic diagram of a specific embodiment of a testing method provided in this specification, and this embodiment may specifically include the following steps.

[0061] S101: The tester imports the automated test case containing the context relationship.

[0062] S102: The voice stream-based automated testing device imports the case into a test case library.

[0063] S103: The voice stream-based automated testing device sends a task creation request to the intelligent customer service voice robot according to the test case to start the call task.

[0064] S104: The intelligent customer service voice robot broadcasts the voice stream according to the preset logic.

[0065] S105: The voice stream-based automated testing device calls a voice recognition module to convert the received voice stream into text.

[0066] S106: The voice stream-based automatic testing device extracts the response content and expected test results in the test case from the test case.

[0067] S107: The voice stream-based automated testing device compares the recognized text with the expected test result, and saves the comparison result in a test result collection module.

[0068] S108: The voice stream-based automated testing device calls a voice conversion module to convert the response content in the test case into a voice stream.

[0069] S109: The voice stream-based automated testing device broadcasts the converted voice stream to the intelligent customer service voice robot.

[0070] Steps S101 to S109 are repeated until the entire test context is completed and all comparison results are obtained.

[0071] In this embodiment, after obtaining all comparison results (also called verification results), the tester can view and verify them.

[0072] In the above embodiment, an automated test system sends a task creation request to an intelligent customer service voice robot according to a test case to start a call task for exemplary explanation. In some implementation scenarios, the intelligent customer service voice robot may also send a task creation request to the automated test system to start a call task, wherein the task creation request includes the intelligent customer service voice robot broadcasting a voice stream according to a preset logic. After receiving the task creation request, the automated test system may obtain a test case for the corresponding scenario. Further, the automated test system may call a voice recognition module to convert the received voice stream into text, extract the response content and expected test results in the test case from the test case, compare the recognized text with the expected test results, and save the comparison results to the test result collection module. Finally, the voice conversion module is called to convert the response content in the test case into a voice stream and broadcast it to the intelligent customer service voice robot until the entire test context ends and all comparison results are obtained.

[0073] From the above description, it can be seen that the embodiment of the present application can receive the voice stream broadcast by the system under test, obtain the expected test results and response content from the test case, and then verify the voice stream and the expected test results, and record the verification results. Furthermore, the response content can be sent to the system under test so that the system under test performs the next round of interaction according to the response content until the context test in the test case is completed and all verification results are obtained. Since the present application is to realize the automated testing of the intelligent customer service voice robot system containing contextual relationships by building an automated testing system based on voice streams, it can reduce manpower input and improve testing efficiency.

[0074] In this specification, each embodiment of the above method is described in a progressive manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. For relevant parts, refer to the partial description of the method embodiment.

[0075] Based on the above-mentioned test method, one or more embodiments of this specification also provide a test device. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device in combination with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the device are similar, the implementation of the specific device of the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0076] Specifically, Figure 3 is a schematic diagram of a module structure of an embodiment of a test device provided in this specification, such as Figure 3 As shown, a testing device provided in this specification may include: a receiving module 120 , a first obtaining module 122 , a checking module 124 , a sending module 126 , a judging module 128 , and a second obtaining module 130 .

[0077] The receiving module 120 may be used to receive a first voice stream broadcasted by the system under test;

[0078] A first acquisition module 122 may be configured to acquire a first expected test result and a first response content from the test case;

[0079] A verification module 124 may be used to verify the first voice stream and the first expected test result, and record the verification result;

[0080] The sending module 126 may be used to send the first response content to the system under test; the first response content is used to enable the system under test to perform a next round of interaction;

[0081] The judgment module 128 may be used to judge whether the context in the test case has been tested;

[0082] The second acquisition module 130 may be used to acquire all verification results when the test is completed.

[0083] It should be noted that the above-mentioned device may also include other implementation modes according to the description of the method embodiment. The specific implementation mode may refer to the description of the relevant method embodiment, which will not be described one by one here.

[0084] The present specification also provides an embodiment of a testing device, including a processor and a memory for storing processor executable instructions, wherein the instructions, when executed by the processor, implement the following steps: receiving a first voice stream broadcast by the system under test; obtaining a first expected test result and a first response content from the test case; verifying the first voice stream and the first expected test result, and recording the verification result; sending the first response content to the system under test; the first response content is used to enable the system under test to perform the next round of interaction; determining whether the context in the test case has been tested; and obtaining all verification results when the test is completed.

[0085] It should be noted that the above-mentioned device may also include other implementations according to the description of the method or device embodiment. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0086] The method embodiments provided in this specification can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 4 1 is a hardware structure block diagram of an embodiment of a test server provided in this specification. The server may be a test device or a test equipment in the above embodiment. Figure 4 As shown, the server 10 may include one or more (only one is shown in the figure) processors 100 (the processor 100 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 200 for storing data, and a transmission module 300 for communication functions. A person skilled in the art can understand that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components shown in the figure may also include other processing hardware, such as a database or multi-level cache, GPU, or other hardware with Figure 4 Different configurations are shown.

[0087] The memory 200 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the test method in the embodiment of this specification, and the processor 100 executes various functional applications and data processing by running the software programs and modules stored in the memory 200. The memory 200 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 200 may further include a memory remotely arranged relative to the processor 100, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0088] The transmission module 300 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission module 300 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 300 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0089] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0090] The methods or devices described in the above embodiments provided in this specification can implement business logic through computer programs and record them on storage media, and the storage media can be read and executed by computers to achieve the effects of the solutions described in the embodiments of this specification. The storage medium may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The storage medium may include: devices that use electrical energy to store information, such as various types of memories, such as RAM, ROM, etc.; devices that use magnetic energy to store information, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and U disks; devices that use optical methods to store information, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.

[0091] The above-mentioned test method or device embodiment provided in this specification can be implemented by a processor in a computer executing corresponding program instructions, such as using the C++ language of the Windows operating system to implement it on a PC, on a Linux system, or other methods such as using Android or iOS system programming languages ​​to implement it on a smart terminal, as well as based on the processing logic of a quantum computer.

[0092] It should be noted that the devices, equipment, and systems described in the specification may also include other implementation methods according to the description of the relevant method embodiments. The specific implementation methods can refer to the description of the corresponding method embodiments, which will not be described one by one here.

[0093] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0094] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of some modules can be implemented in the same or more software and / or hardware, and the modules that implement the same function can also be implemented by a combination of multiple sub-modules or sub-units.

[0095] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and systems according to the embodiments of the present invention. It should be understood that it can be implemented by computer program instructions, and these computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the specified function. These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, and the instruction device is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] Those skilled in the art will appreciate that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may be provided in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware.

[0097] The above is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the scope of the claims.

Claims

1. A testing method, It is characterized in that Applied to an automated testing system, the automated testing system storing test cases containing test scenario context relationships, the method comprising: receiving a first voice stream broadcasted by the system under test; Obtaining a first expected test result and a first response content from the test case; Verifying the first voice stream and the first expected test result, and recording the verification result; Sending the first response content to the system under test; the first response content is used to enable the system under test to perform a next round of interaction; Determine whether the context in the test case has been tested; When the test is completed, obtain all verification results; Wherein, the determining whether the context in the test case has been tested further includes: When the test is not completed, receiving a second voice stream broadcasted by the system under test, where the second voice stream does not interfere with the first voice stream, and the system under test includes an intelligent customer service voice robot including a contextual relationship; Correspondingly, a second expected test result and a second response content are obtained from the test case; the second voice stream and the second expected test result are verified, and the verification result is recorded; the second response content is sent to the system under test; and the second response content is used to enable the system under test to perform the next round of interaction.

2. The method according to claim 1, It is characterized in that Before receiving the first voice stream broadcasted by the system under test, the method includes: A task creation request is sent to the system under test, so that the system under test starts the calling task.

3. The method according to claim 1, It is characterized in that The verifying the first voice stream and the first expected test result, and recording the verification result, includes: Converting the first speech stream into a first text; Verify the first text and the first expected test result, and record the verification result.

4. The method according to claim 3, It is characterized in that The verifying the first text and the first expected test result and recording the verification result includes: Comparing the first text with the first expected test result to see whether the logic jump satisfies a preset rule; Compare the first text with the first expected test result to see whether the content meets a preset condition.

5. The method according to claim 1, It is characterized in that Before sending the first response content to the system under test, the method includes: Converting the first response content into a first response voice stream; The first answer voice stream is sent to the system under test.

6. The method according to claim 1, It is characterized in that After obtaining all verification results, the method further includes: receiving a request to view test results; The test results are presented.

7. A testing device, It is characterized in that include: A receiving module, used for receiving a first voice stream broadcasted by the system under test; A first acquisition module, used to acquire a first expected test result and a first response content from a test case; A verification module, used for verifying the first voice stream and the first expected test result, and recording the verification result; A sending module, used for sending the first response content to the system under test; the first response content is used to enable the system under test to perform a next round of interaction; A judgment module, used to judge whether the context in the test case has been tested; The second acquisition module is used to obtain all verification results when the test is completed; Wherein, the determining whether the context in the test case has been tested further includes: When the test is not completed, receiving a second voice stream broadcasted by the system under test, where the second voice stream does not interfere with the first voice stream, and the system under test includes an intelligent customer service voice robot including a contextual relationship; Correspondingly, a second expected test result and a second response content are obtained from the test case; the second voice stream and the second expected test result are verified, and the verification result is recorded; the second response content is sent to the system under test; and the second response content is used to enable the system under test to perform the next round of interaction.

8. A test device, It is characterized in that The method comprises at least one processor and a memory storing computer executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the instructions.

9. A computer-readable storage medium, It is characterized in that Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images