Automatic testing method, system and electronic equipment for intelligent answering system

Abstract

The present disclosure provides an automatic testing method, system and electronic device for an intelligent answering system. The automatic testing method for an intelligent answering system comprises: responding to a test case generation instruction, obtaining one or more answer rule trees corresponding to the intelligent answering system to be tested, wherein each of the answer rule trees corresponds to a question type; automatically generating test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases; and regularly testing the intelligent answering system to be tested using the test cases corresponding to the one or more question types. The embodiments of the present disclosure can improve the efficiency of the test case generation task caused by the version change of the intelligent answering system.

Description

Background Art

[0002] As a large-scale integrated software system, the intelligent response system's interactive mode is based on open domain natural language. It supports users to express their intentions through natural language, and the system outputs natural language responses based on user intentions. The main goal of the automated testing of the intelligent response system is to test the correctness of the business response answers when users consult about business, so as to ensure the normal business response function. The process of executing business responses within the intelligent response system is to output the corresponding business response answers based on user intentions (such as modifying orders, missing goods and missing accessories, etc.) combined with the slot values ​​input by the user (such as order numbers, etc.). Each user intention classification (type of question) has a corresponding answer rule tree to cover all answerable businesses.

[0003] In actual applications, the answer rule trees for business responses often reach hundreds, and each answer rule tree has an average of dozens of branches, requiring thousands of test cases to cover all businesses. In related technologies, test cases are often created manually, and input scripts, expected system response scripts, and slot values ​​required by the system are manually configured. Due to system version upgrades or new system launches, the system's answer rule tree will change, and at this time, it is often necessary to adjust the input language of the test case and the expected system response language. Therefore, the task of modifying the test cases of the intelligent response system is often very arduous and inefficient.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an automatic testing method for an intelligent answering system and an automatic testing system for an intelligent answering system, so as to at least overcome the problem of heavy test case modification tasks and low efficiency of the intelligent answering system caused by the limitations and defects of related technologies to a certain extent.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an automatic testing method for an intelligent answering system, comprising: responding to a test case generation instruction, obtaining one or more answer rule trees corresponding to the intelligent answering system to be tested, wherein each of the answer rule trees corresponds to a question type; automatically generating test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases; and regularly testing the intelligent answering system to be tested using the test cases corresponding to the one or more question types.

[0007] In an exemplary embodiment of the present disclosure, the automatic generation of test cases corresponding to one or more question types based on the one or more answer rule trees includes: obtaining multiple answer branches corresponding to the answer rule tree, each of the answer branches starting from the root node of the answer rule tree, and each of the answer branches ending at a leaf node of the answer rule tree; when there is a slot-filling node in the answer branch, automatically generating a simulated slot value according to the position of the slot-filling node in the corresponding answer branch, recording the simulated slot value and the slot-filling rhetorical dialogue corresponding to the slot-filling node; recording the slot-filling rhetorical dialogue corresponding to each answer branch, the simulated slot value and the answer dialogue corresponding to the leaf node.

[0008] In an exemplary embodiment of the present disclosure, the automatic generation of test cases corresponding to one or more question types based on the one or more answer rule trees includes: automatically obtaining multiple trigger scripts corresponding to the one or more question types based on the online operation data of the intelligent answering system; recording the corresponding multiple trigger scripts and multiple answer branches for each of the question types to generate test cases corresponding to the multiple question types.

[0009] In an exemplary embodiment of the present disclosure, the periodic use of the test cases corresponding to the one or more question types to test the intelligent answering system to be tested includes: obtaining a test case, selecting a trigger speech from multiple trigger speech corresponding to the test case and inputting it into the intelligent answering system to be tested to obtain the response speech of the intelligent answering system to be tested; if the response speech is a slot-filling rhetorical question speech, when the slot-filling rhetorical question speech meets the corresponding record of the test case, inputting the simulated slot value corresponding to the slot-filling rhetorical question speech and continuing to obtain the response speech; if the response speech is an answer speech, when the answer speech meets the corresponding record of the test case, ending the test of the test case. The test case is tested and the test success and the current test times of the test case are recorded; if the slot-filling rhetorical question or the answer speech does not match the corresponding record of the test case, determine whether the test times of the test case reaches a preset threshold; if the test times of the test case does not reach the preset threshold, add one to the test times of the test case, and re-select a trigger speech from the multiple trigger speeches of the test case and input it into the intelligent answering system to be tested; if the test times of the test case reaches the preset threshold, end the test of the test case and record the test failure; stop the test when it is determined that the testing of all the test cases corresponding to the one or more answer rule trees has been completed.

[0010] In an exemplary embodiment of the present disclosure, the inputting of the simulated slot value corresponding to the slot-filling rhetorical question and continuing to obtain the response speech includes: after inputting the simulated slot value, judging the branch node corresponding to the simulated slot value; sending a branch selection instruction to the intelligent response system to be tested according to the branch node corresponding to the simulated slot value, so as to control the intelligent response system to be tested to enter the branch node corresponding to the simulated slot value, and return the response speech corresponding to the branch node.

[0011] In an exemplary embodiment of the present disclosure, it also includes: automatically generating a test report based on the test results of the test cases corresponding to the one or more question types, and the test report includes the test results of each test case, the number of tests, and the question and answer experience score of the intelligent answering system to be tested.

[0012] In an exemplary embodiment of the present disclosure, the question-answering experience score of the intelligent answering system to be tested is obtained based on the test success ratio of multiple test cases, the number of interactions corresponding to the successfully tested test cases, and the number of test cases corresponding to each question type.

[0013] According to a second aspect of an embodiment of the present disclosure, an automatic testing system for an intelligent answering system is provided, including: a test case automatic generation module, configured to respond to a test case generation instruction, obtain one or more answer rule trees corresponding to the intelligent answering system to be tested, wherein each of the answer rule trees corresponds to a question type; automatically generate test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases; an automatic testing module, configured to periodically test the intelligent answering system to be tested using the test cases corresponding to the one or more question types.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0015] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the automatic testing method for an intelligent response system as described in any one of the above.

[0016] The disclosed embodiment automatically generates test cases according to the answer rule tree of the intelligent answering system to be tested, and automatically uses the test cases to test the intelligent answering system to be tested. It can automatically generate test cases and automatically test the intelligent answering system when the intelligent answering system is newly launched or the version is updated, which greatly reduces the workload of modifying the test cases of the intelligent answering system and can effectively improve the efficiency of automated testing of the intelligent answering system.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 is a flow chart of an automatic testing method for an intelligent answering system in an exemplary embodiment of the present disclosure.

[0020] Figure 2 yes Figure 1 Schematic diagram of the answer branches shown.

[0021] Figure 3 The sub-flow chart of step S102 in one embodiment.

[0022] Figure 4 This is a sub-flow chart of step S103 in one embodiment.

[0023] Figure 5 yes Figure 4 Another flow chart of the process shown.

[0024] Figure 6 FIG. 6 is a block diagram of an automatic testing system 600 for an intelligent response system in an exemplary embodiment of the present disclosure.

[0025] Figure 7 It is a system structure diagram in which a simulation slot value management module participates in the work in one embodiment of the present disclosure.

[0026] Figure 8 It is a functional schematic diagram of a simulation slot value management module in one embodiment of the present disclosure.

[0027] Fig. 9 It is a schematic diagram of the working details of each module in the system 600 in one embodiment of the present disclosure.

[0028] Fig.10 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0030] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 The flowchart of the automatic testing method for the intelligent answering system in the exemplary embodiment of the present disclosure is schematically shown.

[0033] refer to Figure 1 , the automatic testing method 100 for the intelligent response system may include:

[0034] Step S101, responding to a test case generation instruction, obtaining one or more answer rule trees corresponding to the intelligent answering system to be tested, wherein each of the answer rule trees corresponds to a question type;

[0035] Step S102, automatically generating test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases;

[0036] Step S103: regularly testing the intelligent answering system to be tested using test cases corresponding to the one or more question types.

[0037] The disclosed embodiment automatically generates test cases according to the answer rule tree of the intelligent answering system to be tested, and automatically uses the test cases to test the intelligent answering system to be tested. It can automatically generate test cases and automatically test the intelligent answering system when the intelligent answering system is newly launched or the version is updated, which greatly reduces the workload of modifying the test cases of the intelligent answering system and can effectively improve the efficiency of automated testing of the intelligent answering system.

[0038] Next, each step of the automatic testing method 100 for the intelligent response system is described in detail.

[0039] The method provided by the embodiment of the present disclosure may be executed by an automatic testing system.

[0040] In step S101, in response to a test case generation instruction, one or more answer rule trees corresponding to the intelligent answering system to be tested are obtained, wherein each of the answer rule trees corresponds to a question type.

[0041] In the disclosed embodiment, the test case generation instruction can be automatically generated when the intelligent answering system to be tested is online or the version is updated. Specifically, an automatic detection program can be set to detect whether multiple answer rule trees of the intelligent answering system to be tested have been updated (updates include additions, deletions, or modifications) when it is detected that the intelligent answering system to be tested is online or the version is updated. If there is an update, a test case generation instruction is generated to instruct the automatic testing system to automatically generate test cases for one or more answer rule trees or all answer rule trees that have been updated.

[0042] By automatically triggering the test case generation instruction, the test case can be updated in time when the intelligent response system is updated, which greatly reduces the workload of test case update and improves the efficiency of test case update.

[0043] An answer rule tree corresponds to a question type and includes multiple answer branches. Table 1 is an example of answer branches.

[0044] Table 1:

[0045]

[0046] Figure 2 yes Figure 1 Schematic diagram of the answer branches shown.

[0047] refer to Figure 2 As shown in Table 1, the first step (start) of all answer branches corresponds to the root node of the answer rule tree, which is triggered by the triggering phrase "I want to change my order". When the intelligent answering system receives the triggering phrase, it determines that the question type is "order modification" and automatically enters the root node 21 of the answer rule tree 200 corresponding to the question type.

[0048] In the first step, the triggering speech triggers a response speech of the answer branch, and the type of the response speech is a slot-filling rhetorical question speech. It is worth noting that in the embodiment of the present disclosure, the response speech includes slot-filling rhetorical question speech and answer speech, that is, the system's response includes slot-filling rhetorical question and final answer: if the system cannot give a final answer, it outputs a slot-filling rhetorical question speech to require the questioner to provide more information (slot value) until the system can give a final answer and outputs the answer speech.

[0049] In the second step, the first slot value is provided for the order number slot 22 (manually configured in the related technology), so that the system enters the "unmodifiable" branch based on the judgment of the first slot value, reaches the order attribute modification slot 23, and returns to the slot filling rhetorical dialogue again.

[0050] In the third step, the second slot value is provided for the order attribute slot 23, so that the system enters the "Modify Phone" branch based on the judgment of the second slot value and provides the corresponding answer "Answer D". At this point, the answer branch reaches a leaf node (answer word) of the answer rule tree, and the question and answer ends.

[0051] Figure 2 The test case corresponding to the answer branch shown consists of a triggering statement, a first slot value, a second slot value, and an answer statement.

[0052] In step S102, one or more test cases corresponding to question types are automatically generated according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases.

[0053] Figure 3 It is a schematic diagram of sub-steps of step S102.

[0054] refer to Figure 3 , step S102 may include:

[0055] Step S1021, obtaining multiple answer branches corresponding to the answer rule tree, where the beginning of each answer branch is the root node of the answer rule tree, and the end of each answer branch is a leaf node of the answer rule tree;

[0056] Step S1022, when there is a slot-filling node in the answer branch, automatically generate a simulated slot value according to the position of the slot-filling node in the corresponding answer branch, and record the simulated slot value and the slot-filling rhetorical question corresponding to the slot-filling node;

[0057] Step S1023, recording the slot-filling rhetorical questions corresponding to each of the answer branches, the simulated slot values, and the answer words corresponding to the leaf nodes.

[0058] In the disclosed embodiment, a simulated slot value (mock value) can be automatically generated according to the slot filling node to automatically generate a test case. In one embodiment, a high-performance hash algorithm can be used to generate the simulated slot value in real time.

[0059] First, the answer rule tree is disassembled into multiple answer branches, each of which corresponds to a leaf node (answer phrase). Figure 2 The answer rule tree shown can be decomposed into four answer branches. Then, the corresponding slot value is generated for the slot filling node in each answer branch. Figure 2 For example, at the order number slot 22, the first simulation slot value and the second simulation slot value (order number) can be generated according to the number of branches of the slot filling node, and each simulation slot value corresponds to two answer branches; at the modified order attribute slot 23, the third simulation slot value and the fourth simulation slot value are continued to be generated for each answer branch; at the modified order attribute slot 24, the fifth simulation slot value and the sixth simulation slot value are generated for each answer branch.

[0060] Finally, the following four answer branches are obtained: "Start-first simulated slot value-third simulated slot value-answer C", "Start-first simulated slot value-fourth simulated slot value-answer D", "Start-second simulated slot value-fifth simulated slot value-answer A", "Start-second simulated slot value-sixth simulated slot value-answer B". At the same time, the slot-filling rhetorical words corresponding to each slot-filling node are recorded for each answer branch.

[0061] In one embodiment, multiple triggering scripts corresponding to one or more question types can be automatically obtained based on the online operation data of the intelligent answering system, and then the corresponding multiple triggering scripts and multiple answer branches can be recorded for each question type to generate test cases corresponding to multiple question types.

[0062] For example, user dialogues that can trigger questions of multiple question types can be collected from online operation data, and multiple user dialogues that can trigger questions of corresponding types, i.e., trigger dialogues, are recorded for each answer rule tree. Table 2 schematically shows the trigger dialogues corresponding to two question types.

[0063] Table 2

[0064]

[0065] Next, the triggering words are combined with the above answer branches to generate test cases corresponding to the question type. Table 3 schematically shows the test cases corresponding to the "Modify Order" type of questions.

[0066] Table 3

[0067]

[0068] If there are ten types of triggering words corresponding to one question type, taking the above four answer branches as an example, there are ten triggering words corresponding to each answer branch, that is, four test cases can be generated based on the four answer branches, and the triggering words for each test case are ten. The diversification of triggering words corresponding to a test case is conducive to better and more flexibly realizing simulation testing.

[0069] After automatically generating test cases for one or more question types corresponding to one or more answer rule trees, these test cases can be automatically used to test the intelligent answering system to be tested.

[0070] In step S103, the intelligent answering system to be tested is tested using test cases corresponding to the one or more question types at regular intervals.

[0071] Figure 4 This is a sub-flow chart of step S103 in one embodiment.

[0072] refer to Figure 4 , step S103 may include:

[0073] Step S1031, obtaining a test case, selecting a triggering speech from multiple triggering speech corresponding to the test case and inputting it into the intelligent response system to be tested to obtain a response speech of the intelligent response system to be tested;

[0074] Step S1032, if the response speech is a slot-filling rhetorical question speech, when the slot-filling rhetorical question speech meets the corresponding record of the test case, input the simulated slot value corresponding to the slot-filling rhetorical question speech and continue to obtain the response speech;

[0075] Step S1033, if the response speech is an answer speech, when the answer speech matches the corresponding record of the test case, the test of the test case is terminated and the test success and the current test times of the test case are recorded;

[0076] Step S1034, if the slot-filling rhetorical questioning words or the answer words do not match the corresponding record of the test case, determine whether the number of tests of the test case reaches a preset threshold;

[0077] Step S1035: if the test number of the test case does not reach the preset threshold, the test number of the test case is increased by one, and a triggering speech is selected from the multiple triggering speech of the test case and input into the intelligent answering system to be tested;

[0078] Step S1036, if the test number of the test case reaches the preset threshold, end the test of the test case and record the test failure;

[0079] Step S1037, stopping the test when it is determined that the test of all the test cases corresponding to the one or more answer rule trees has been completed.

[0080] Figure 4 The embodiment shown can also be implemented by Figure 5 The branch diagram shown is used to represent it.

[0081] like Figure 5 As shown, process 500 may include:

[0082] Step S501, obtaining a test case;

[0083] Step S502: input a triggering speech corresponding to the test case into the intelligent answering system to be tested;

[0084] Step S503, obtaining the response words;

[0085] Step S504, determine the type of response speech, if it is a slot-filling rhetorical question speech, proceed to step S505, if it is an answer speech, proceed to step S506;

[0086] Step S505, determining whether the slot-filling rhetorical questioning technique conforms to the test case record, if so, proceeding to step S507 to input the simulated slot value and then returning to step S503, if not, proceeding to step S509;

[0087] Step S506, determine whether the answer is consistent with the test case record, if yes, go to step S508, if not, go to step S509;

[0088] Step S508, record the success of the test case and the number of times the test case is tested;

[0089] Step S509, determining whether the test times of the test case reaches the threshold value, if not, proceeding to step S510 to increase the test times by one and then returning to step S502, if yes, proceeding to step S511 to record that the test case fails;

[0090] Step S512, determine whether all test cases have been tested, if not, return to step S501, if yes, end the test.

[0091] Figure 4 and Figure 5 The test cases shown in are all test cases corresponding to one or more problem types corresponding to the test case generation instruction in step S101.

[0092] Table 4 is the test process of a test case, corresponding to the test case shown in Table 3.

[0093] Table 4

[0094]

[0095] Since the intelligent answering system will judge the input slot value, when the simulated slot value is input to the slot filling node, in order to ensure that the simulated slot value can make the answer branch into the correct node, the embodiment of the present disclosure is also arranged to judge the branch node corresponding to the simulated slot value after the simulated slot value is input, and then send a branch selection instruction to the intelligent answering system to be tested according to the branch node corresponding to the simulated slot value, so as to control the intelligent answering system to be tested to enter the branch node corresponding to the simulated slot value, and return the answer words corresponding to the branch node.

[0096] Determining the branch node corresponding to the simulated slot value can be achieved by setting a Bloom filter for each branch node. The simulated slot value corresponding to each branch can be recorded in the Bloom filter corresponding to the branch, and the Bloom filter can be stored in a distributed cache (such as a Redis cache). Then, when or after filling in the simulated slot value, the matching result of each Bloom filter is obtained, and the intelligent answering system to be tested is controlled to enter the branch corresponding to the Bloom filter according to the matching result.

[0097] In one embodiment of the present disclosure, a test report can also be automatically generated based on the test results of the test cases corresponding to one or more question types, and the test report includes the test results of each test case, the number of tests, and the question-answering experience score of the intelligent answering system to be tested. The question-answering experience score of the intelligent answering system to be tested is obtained based on the test success ratio of multiple test cases, the number of interactions corresponding to the test cases that have been successfully tested, and the number of test cases corresponding to each question type.

[0098] The test report provided by the embodiment of the present disclosure includes the test results of each test case, the evaluation score of the interactive experience and the evaluation score of the overall experience. The test report can be presented in a chart, for example.

[0099] The test results may be displayed, for example, by displaying the success and failure marks of all test cases and the interaction logs of failed cases, so as to find system bugs.

[0100] In the process of forming the interactive experience evaluation score, the higher the proportion of successful test cases, the higher the evaluation score; the fewer the number of interactions in the successful test cases, the higher the evaluation score; the more test cases corresponding to a question type, the better the business response experience; and the higher the credibility of the evaluation score, the higher the evaluation score. The experience evaluation score can be calculated by quantifying the above influences, and the comparison of evaluation scores of different question types or the comparison of evaluation scores of the same question type in different versions of the intelligent response system can be displayed through charts, so as to facilitate the analysis of experience evaluation changes of a single question type and assist operators in optimizing the classification response logic.

[0101] The calculation formula for the experience evaluation score of question type I may be, for example:

[0102] Score(I)=(1+TAScore(I)*SRate(I))*lg(CNumber(I)) (1)

[0103] Among them, TAScore(I) is the truncated average reward score of problem type I: each successful test case in the classification has an initial reward score of 100 points, and a single test case interaction is deducted 1 point for each additional round of interaction. After obtaining the reward score of each case, the highest and lowest scores are removed to calculate the total score, which is divided by the number of cases to calculate the truncated average reward score. SRate(I) is the test success rate of problem type I, that is, the proportion of successful test cases in the test cases corresponding to this problem type. CNumber(I) is the number of test cases of problem type I.

[0104] The above scoring can provide an overall experience evaluation of the intelligent response system, that is, add up the evaluation scores of all question types, and display the fluctuations of the overall experience of different versions of the intelligent response system in a chart, so that testers can analyze system problems based on the test report according to the failure cases and experience evaluation fluctuations. Table 5 is an example of calculating the evaluation experience score.

[0105] Table 5

[0106]

[0107] In this embodiment, the launch of the intelligent answering system or the modification of the answer rule tree will trigger the creation of an automated test case, and a timer (such as Jenkins) will periodically trigger the test execution module to periodically execute the test case and automatically generate a test report.

[0108] To summarize, the disclosed embodiment automatically generates test cases by automatically disassembling the answer rule tree and automatically generating simulated slot values, replaces manual generation or modification of test cases with machine automation, reduces manual maintenance work, and introduces an evaluation algorithm in the test report, thereby providing operators with reference data for improving the response experience.

[0109] Corresponding to the above method embodiments, the present disclosure also provides an automatic testing system for an intelligent response system, which can be used to execute the above method embodiments.

[0110] Figure 6 A block diagram of an automatic testing system for an intelligent response system in an exemplary embodiment of the present disclosure is schematically shown.

[0111] refer to Figure 6 , the automatic test system 600 for the intelligent response system may include:

[0112] The test case automatic generation module 601 is configured to respond to the test case generation instruction, obtain one or more answer rule trees corresponding to the intelligent answering system to be tested, wherein each of the answer rule trees corresponds to a question type; automatically generate test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases;

[0113] The automatic testing module 602 is configured to periodically test the intelligent answering system to be tested using test cases corresponding to the one or more question types.

[0114] In an exemplary embodiment of the present disclosure, the test case automatic generation module 601 is configured to: obtain multiple answer branches corresponding to the answer rule tree, each of the answer branches starts at the root node of the answer rule tree, and each of the answer branches ends at a leaf node of the answer rule tree; when there is a slot-filling node in the answer branch, automatically generate a simulated slot value according to the position of the slot-filling node in the corresponding answer branch, record the simulated slot value and the slot-filling rhetorical words corresponding to the slot-filling node; record the slot-filling rhetorical words corresponding to each answer branch, the simulated slot value and the answer words corresponding to the leaf node.

[0115] In an exemplary embodiment of the present disclosure, the test case automatic generation module 601 is configured to: automatically obtain multiple triggering scripts corresponding to the one or more question types based on the online operation data of the intelligent answering system; record the corresponding multiple triggering scripts and multiple answer branches for each of the question types to generate test cases corresponding to the multiple question types.

[0116] In an exemplary embodiment of the present disclosure, the automatic testing module 602 is configured to: obtain a test case, select a trigger speech from multiple trigger speeches corresponding to the test case and input it into the intelligent response system to be tested to obtain the response speech of the intelligent response system to be tested; if the response speech is a slot-filling rhetorical question speech, when the slot-filling rhetorical question speech meets the corresponding record of the test case, input the simulated slot value corresponding to the slot-filling rhetorical question speech and continue to obtain the response speech; if the response speech is an answer speech, when the answer speech meets the corresponding record of the test case, end the test of the test case and record the success of the test and The current test times of the test case; if the slot-filling rhetorical question or the answer words do not match the corresponding record of the test case, determine whether the test times of the test case reaches a preset threshold; if the test times of the test case does not reach the preset threshold, add one to the test times of the test case, and reselect a trigger words from the multiple trigger words of the test case and input it into the intelligent answering system to be tested; if the test times of the test case reaches the preset threshold, end the test of the test case and record the test failure; stop the test when it is determined that the testing of all the test cases corresponding to the one or more answer rule trees has been completed.

[0117] In an exemplary embodiment of the present disclosure, the automatic testing module 602 is configured to: after inputting the simulated slot value, determine the branch node corresponding to the simulated slot value; send a branch selection instruction to the intelligent response system to be tested according to the branch node corresponding to the simulated slot value, so as to control the intelligent response system to be tested to enter the branch node corresponding to the simulated slot value, and return the response speech corresponding to the branch node.

[0118] In an exemplary embodiment of the present disclosure, a test report generation module 603 is also included, which is configured to automatically generate a test report based on the test results of the test cases corresponding to the one or more question types, and the test report includes the test results of each test case, the number of tests, and the question and answer experience score of the intelligent answering system to be tested.

[0119] In an exemplary embodiment of the present disclosure, the question-answering experience score of the intelligent answering system to be tested is obtained based on the test success ratio of multiple test cases, the number of interactions corresponding to the successfully tested test cases, and the number of test cases corresponding to each question type.

[0120] In one embodiment, a separate simulation slot value management module can be used to implement functions such as generating simulation slot values ​​and processing input of simulation slot values. The simulation slot value management module can be used to generate simulation slot values, store simulation slot values ​​(stored in the Bloom filter corresponding to each branch), identify the branch corresponding to the simulation slot value through the Bloom filter during testing, and control the intelligent response module to directly enter the branch corresponding to the simulation slot value.

[0121] Figure 7 It is a system structure diagram in which a simulation slot value management module participates in the work in one embodiment of the present disclosure.

[0122] refer to Figure 7 , the test case automatic generation module 601 obtains the answer rule tree from the intelligent answering system 700, and calls the simulated slot value management module 604 to generate the simulated slot value, wherein the simulated slot value management module 604 generates the simulated slot value according to the interaction information with the intelligent answering system 700. The test case automatic generation module 601 generates a test case set and stores it for the automatic test module 602 to call. The automatic test module 602 interacts with the intelligent answering system 700 to perform the test. When encountering a slot filling node during the interaction process, the simulated slot value management module 604 is called to obtain the simulated slot value and fill the slot. The simulated slot value management module 604 controls the intelligent answering system 700 to enter the branch corresponding to the simulated slot value according to the slot filling situation. After the test is completed, the automatic test module 602 sends the test results to the test report generation module 603, and the test report generation module 603 generates a test report.

[0123] In one embodiment, the simulated slot value management module 604 is always running during the test process of the intelligent response system 700. When a slot filling node of the intelligent response system 700 obtains a slot value, the simulated slot value management module 604 determines whether the slot value is a simulated slot value. If it is a simulated slot value, the intelligent response system 700 is controlled to enter the corresponding branch. If it is not a simulated slot value, the intelligent response system 700 is not controlled, and the intelligent response system 700 judges the current slot value by itself and enters the branch according to the judgment result. This situation corresponds to the intelligent response system 700 receiving an externally called slot value or a manually input slot value during manual testing.

[0124] Figure 8 Schematic diagram of the functions of the simulation slot value management module 604 in one embodiment of the present disclosure.

[0125] refer to Figure 8In order modification problems, when a slot filling node is encountered, the intelligent response system determines that the slot value needs to be obtained. At this time, the simulated slot value management module 604 can be called to generate simulated slot values ​​and store them in the Bloom filter corresponding to each branch and simulated slot value. The slot value manually input or externally called can also be obtained. Next, the simulated slot value management module 604 determines whether the slot value received by the intelligent response system is a simulated slot value. If so, it enters the processing flow of the corresponding branch according to the corresponding branch of the simulated slot value (judged by the Bloom filter); if not, no processing is performed, and the intelligent response system obtains the slot value obtained in the order number slot, and judges the order status corresponding to the slot value in the next step, and enters the processing flow of the corresponding branch.

[0126] Fig. 9 It is a schematic diagram of the working details of each module in the system 600 in one embodiment of the present disclosure.

[0127] refer to Fig. 9 In the test case automatic generation module 601, step S901 triggers the test case automatic generation process by changing the answer rule tree or the intelligent answer system. In step S902, the classification trigger corpus is obtained and stored; in step S903, the answer rule tree is disassembled and the answer branch is stored; in step S904, the simulated slot value management module 604 is called to obtain the simulated slot value corresponding to each answer branch in the answer rule tree; in step S905, the simulated slot value management module 604 generates the simulated slot value; in step S906, the simulated slot value management module 604 stores the simulated slot value in the Bloom filter corresponding to the slot filling node judgment branch and stores the slot value in the Bloom filter cache; in step S907, a classification test case set is generated and stored according to the classification trigger corpus, the answer branch and the simulated slot value corresponding to each answer branch.

[0128] In the automatic test module 602, the timer triggers the automatic test process through step S922. In step S908, the classified test case set stored in the test case automatic generation module 601 is read; in step S909, an interaction is initiated with the intelligent answering system 700, and in step S910, it is determined whether the answering language of the intelligent answering system 700 is a slot-filling rhetorical question. If so, it proceeds to step S911 to obtain the simulated slot value. When it is determined in step S912 that the test number threshold of the test case has not been reached, it returns to step S909 to output the simulated slot value to the intelligent answering system 700. After obtaining the slot value, the intelligent answering system 700 calls the simulated slot value management module 604 through the slot filling node 71, enters step S913 to traverse all request slot filling nodes in turn to determine the Bloom filter corresponding to the branch (stored in the cache), and determines whether the slot value is a simulated slot value in step S914. If yes, enter step S915 to return the branch identifier corresponding to the simulated slot value and then return it to the slot filling node 71. If not, directly return the result to the slot filling node 71, and the slot filling node 71 controls the intelligent answering system to enter the next branch. If it is not a slot filling question, enter step S917 to determine whether it is the expected answer (the answer speech corresponding to the test case). If not, enter step S918 to switch the trigger speech and return to step S912; if yes, enter step S919 to return the test success information. When step S912 determines that the test number threshold of the current test case is reached, it enters step S920 to return the test failure information.

[0129] After receiving the test success information or the test failure information, the automatic test module 602 determines whether there are test cases to be executed. If yes, it returns to step S908 to continue to obtain test cases; if not, it enters the test report generation module 603 to generate a test report.

[0130] To sum up, the automatic testing system for the intelligent answering system provided by the embodiments of the present disclosure can automatically generate test cases or automatically test the intelligent answering system when the intelligent answering system is updated, and automatically generate test reports, which can greatly improve the testing efficiency of the intelligent answering system and reduce the workload of updating test cases.

[0131] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0132] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0133] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0134] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present invention will be described. Fig.10 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0135] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, and a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010).

[0136] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the following steps: Figure 1 Follow the steps shown in .

[0137] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0138] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0139] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0140] The electronic device 1000 may also communicate with one or more external devices 1100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0141] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0142] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0143] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0144] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0146] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0147] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0148] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0149] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. An automatic testing method for an intelligent answering system, characterized in that: include: In response to the test case generation instruction, one or more answer rule trees corresponding to the intelligent answering system to be tested are obtained, wherein each of the answer rule trees corresponds to a question type; Automatically generate test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases, and the test cases include triggering speech, slot-filling rhetorical speech corresponding to each slot-filling node, and simulated slot values ​​and answer speech; Regularly testing the intelligent answering system to be tested using test cases corresponding to the one or more question types; The step of periodically using the test cases corresponding to the one or more question types to test the intelligent answering system to be tested includes: Obtain a test case, select a triggering speech from multiple triggering speech corresponding to the test case and input it into the intelligent response system to be tested to obtain the response speech of the intelligent response system to be tested; If the answering speech is a slot-filling rhetorical questioning speech, when the slot-filling rhetorical questioning speech meets the corresponding record of the test case, input the simulated slot value corresponding to the slot-filling rhetorical questioning speech and continue to obtain the answering speech; If the response speech is an answer speech, when the answer speech matches the corresponding record of the test case, the test of the test case is terminated and the success of the test and the current test number of the test case are recorded; If the slot-filling rhetorical questioning technique or the answering technique does not match the corresponding record of the test case, determining whether the number of tests of the test case reaches a preset threshold; If the number of tests of the test case does not reach the preset threshold, the number of tests of the test case is increased by one, and a triggering speech is selected from multiple triggering speech of the test case and input into the intelligent answering system to be tested; If the number of tests for the test case reaches the preset threshold, the test of the test case is terminated and the test failure is recorded; The test is stopped when it is determined that the test of all the test cases corresponding to the one or more answer rule trees has been completed.

2. The automatic testing method for an intelligent response system according to claim 1, characterized in that: The step of automatically generating one or more test cases corresponding to the question types according to the one or more answer rule trees comprises: Acquire multiple answer branches corresponding to the answer rule tree, where the beginning of each answer branch is a root node of the answer rule tree, and the end of each answer branch is a leaf node of the answer rule tree; When there is a slot-filling node in the answer branch, automatically generate a simulated slot value according to the position of the slot-filling node in the corresponding answer branch, and record the simulated slot value and the slot-filling rhetorical question corresponding to the slot-filling node; Record the slot-filling rhetorical questions, simulated slot values ​​and answer words corresponding to each answer branch, and the leaf nodes.

3. The automatic testing method for an intelligent response system according to claim 2, characterized in that: The step of automatically generating one or more test cases corresponding to the question types according to the one or more answer rule trees comprises: Automatically obtain multiple triggering scripts corresponding to the one or more question types according to the online operation data of the intelligent answering system; For each of the question types, corresponding multiple triggering scripts and multiple answer branches are recorded to generate test cases corresponding to the multiple question types.

4. The automatic testing method for an intelligent response system according to claim 1, characterized in that: The step of inputting the simulated slot value corresponding to the slot filling rhetorical questioning speech and continuing to obtain the answering speech comprises: After inputting the simulated slot value, determining a branch node corresponding to the simulated slot value; According to the branch node corresponding to the simulated slot value, a branch selection instruction is sent to the intelligent response system to be tested to control the intelligent response system to be tested to enter the branch node corresponding to the simulated slot value and return the response speech corresponding to the branch node.

5. The automatic testing method for an intelligent response system according to claim 1, characterized in that: Also includes: A test report is automatically generated according to the test results of the test cases corresponding to the one or more question types, wherein the test report includes the test results of each test case, the number of tests, and the question-answering experience score of the intelligent answering system to be tested.

6. The automatic testing method for an intelligent response system according to claim 5, characterized in that: The question-and-answer experience score of the intelligent answering system to be tested is obtained based on the test success ratio of multiple test cases, the number of interactions corresponding to the successful test cases, and the number of test cases corresponding to each question type.

7. An automatic testing system for an intelligent answering system, characterized in that: include: A test case automatic generation module is configured to respond to a test case generation instruction, obtain one or more answer rule trees corresponding to the intelligent answering system to be tested, wherein each of the answer rule trees corresponds to a question type; automatically generate test cases corresponding to one or more question types according to the one or more answer rule trees, wherein each question type corresponds to multiple test cases, and the test cases include triggering speech, slot-filling rhetorical speech corresponding to each slot-filling node, and simulated slot values ​​and answer speech; An automatic testing module, configured to periodically test the intelligent answering system to be tested using test cases corresponding to the one or more question types; Among them, the automatic testing module is configured to: obtain a test case, select a trigger speech from multiple trigger speeches corresponding to the test case and input it into the intelligent response system to be tested to obtain the response speech of the intelligent response system to be tested; if the response speech is a slot-filling rhetorical question speech, when the slot-filling rhetorical question speech meets the corresponding record of the test case, input the simulated slot value corresponding to the slot-filling rhetorical question speech and continue to obtain the response speech; if the response speech is an answer speech, when the answer speech meets the corresponding record of the test case, end the test of the test case and record the success of the test and the test case. The current number of tests; if the slot-filling rhetorical question or the answer speech does not match the corresponding record of the test case, determine whether the number of tests for the test case has reached a preset threshold; if the number of tests for the test case has not reached the preset threshold, add one to the number of tests for the test case, and reselect a trigger speech from the multiple trigger speeches of the test case and input it into the intelligent answering system to be tested; if the number of tests for the test case reaches the preset threshold, end the test of the test case and record the test failure; stop the test when it is determined that the testing of all test cases corresponding to the one or more answer rule trees has been completed.

8. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the automatic testing method for an intelligent response system according to any one of claims 1 to 6 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the automatic testing method for an intelligent response system as described in any one of claims 1 to 6.

Images