System and method for testing single-wheel and multi-wheel effects of text robot

Abstract

The invention discloses a system and a method for testing single-round and multi-round effects of a text robot, and the method comprises the steps: taking a test set as a unified drive entrance, enabling a tester to only need to input a to-be-verified single-round problem or multi-round dialogue and an expected result, and carrying out the writing and uploading of the to-be-verified single-round problem or multi-round dialogue; the test configuration unit can select a single round or multiple rounds and respectively associate a knowledge question-answer library or a workflow verbal skill, so that the original test behaviors needing to be manually input one by one are converted into tasks which can be scheduled by a system in batches; on the basis, the test execution and judgment unit drives the test set in batches and obtains an actual result which can be directly used for verification, at least outputs a standard question identifier and a similarity score in a single round, at least outputs a verbal skill node label in multiple rounds, and then compares the verbal skill node label with an expected result to generate a judgment result; therefore, the test conclusion does not depend on manual subjective judgment and manual recording any more, but is automatically obtained by a unified comparison rule.

Description

Technical Field

[0001] This application relates to the field of dialogue system testing technology, and in particular to a system and method for single-turn and multi-turn effect testing of text robots. Background Technology

[0002] With the development of artificial intelligence and natural language processing technologies, text robots (such as dialogue systems for customer service inquiries, business processing, knowledge-based Q&A, and process guidance) are widely used in government affairs, finance, e-commerce, and operational support scenarios. Text robots typically include both single-turn question-and-answer capabilities based on knowledge bases / FAQs and multi-turn dialogue / workflow capabilities based on dialogue nodes and branching logic. To ensure effective deployment and continuous iteration quality, enterprises usually need to evaluate and perform regression testing on the accuracy of text robot responses and the correctness of dialogue flow under different corpora, different knowledge base versions, and different dialogue workflow versions.

[0003] In existing technologies, the effectiveness verification of text-based chatbots largely relies on manual testing. In single-round testing, testers often need to input test questions one by one, manually record the standard questions or answers the chatbot answers correctly, and compare them with the expected results. In multi-round testing, testers need to input user statements one round at a time according to a preset dialogue path, manually observe the response content in each round and whether the dialogue enters the expected branch / node, and backtrack to locate the cause of the problem when deviations occur. The above manual testing methods have drawbacks such as low test execution efficiency, large amount of repetitive work, non-standard result recording and statistics, and high regression testing costs. At the same time, due to the round dependencies and complex branches in multi-round dialogues, the manual verification process is more susceptible to subjective factors, making it difficult to locate "from which round the deviation from expectations" in a timely and accurate manner, thus affecting the quality control and delivery efficiency of text-based chatbot version iterations.

[0004] Therefore, in the effectiveness testing of text robots, single-round question-and-answer testing requires manual input and verification of each question, resulting in low efficiency and high retesting costs. Multi-round dialogue testing, due to round dependence and complex branches, makes it difficult to locate deviations and the evaluation results are highly subjective, which has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a system and method for single-round and multi-round performance testing of text robots, aiming to solve the problems in the existing technology of text robot performance testing. Single-round question-and-answer testing requires manual input and verification of each question, resulting in low efficiency and high retesting costs. Multi-round dialogue testing is difficult to locate deviations due to round dependence and branch complexity, and the evaluation results are highly subjective.

[0006] In a first aspect, a method for single-round and multi-round performance testing of a text robot, the method comprising: The system includes a test set management unit, a test configuration unit, a test execution and judgment unit, and a process recording and output unit; The test set management unit is used to provide test set templates and receive uploaded test sets. The test set contains multiple test data, and each test data contains test input and the expected result corresponding to the test input. The test configuration unit is used to select the test type, which includes at least single-round testing and multi-round testing, and is used to associate a knowledge question and answer base in single-round testing and associate a workflow script in multi-round testing; The test execution and judgment unit is used to drive the text robot to obtain actual results in batches based on the test set, and compare the actual results with the expected results to generate a judgment result. The actual results of a single round of testing include at least standard question identifiers and similarity scores, and the actual results of multiple rounds of testing include utterance node tags. The process recording and output unit is used to record test process information and output test result data.

[0007] Optionally, in the above scheme, the test set template provided by the test set management unit is used to distinguish between single-round test data and multi-round test data; The single-round test data includes the question text and the corresponding expected standard question identifier; the multi-round test data includes the user input sequence arranged by round and the corresponding expected dialogue node label sequence for each round.

[0008] Optionally, in the above scheme, the process recording and output unit includes a progress control unit; The progress control unit is used to output the total number of tests, the number of tests completed, the number of tests passed, and the number of tests failed during the test execution process, and is also used to output the judgment result entries corresponding to the completed test data and the detailed information of the entries; The progress control unit is also used to respond to termination commands to stop the processing of subsequent unexecuted test data and to identify incomplete test data.

[0009] Optionally, in the above scheme, the process recording and output unit further includes an exception recording unit; the exception recording unit is used to record exception information such as timeout, no response, or call error that occurs during test execution, and to associate and store the exception information with the corresponding test data.

[0010] Optionally, in the above scheme, the process recording and output unit may further include a summary report unit; The summary report unit is used to generate summary results and export test result reports based on the executed test data after the test is completed or terminated; wherein, the summary results include at least the start time, end time, number of executions, and number of failed tests; The summary report unit is used to group and summarize the executed test data according to the same expected results, and output a detailed list for each group, including at least the number of tests and the number of passes for the group; wherein, the expected result is the expected standard question identifier in a single round of testing, and the expected dialogue node label in a multi-round test. For test data that is determined to be unsuccessful, the test result report also includes actual hit information. In a single round of testing, the actual hit information includes the standard question identifier and similarity score of the actual hit. In a multi-round test, it includes the inconsistent rounds and the actual speech node labels of the corresponding rounds. The test result report includes at least the test input, expected result, actual result and judgment result, and includes similarity score in single-round tests and speech node tags in multi-round tests.

[0011] Secondly, a method for single-round and multi-round performance testing of a text robot is provided, the method comprising: Step 1: Obtain a test set containing test inputs and expected results, and select the test type; associate with a knowledge question and answer base for single-round testing, and associate with workflow scripts for multi-round testing; Step 2: Drive the text robot in batches based on the test set to obtain the actual results corresponding to each test data; Step 3: Compare the actual results with the expected results to generate a judgment result and generate test result data; the test result data includes at least: test input, expected result, actual result, and judgment result; wherein, the actual result of a single round of testing includes standard question identifiers and similarity scores, and the actual result of multiple rounds of testing includes verbal node tags; and for test data judged as failing, the test result data also includes actual hit information, the actual hit information of a single round of testing includes the actual hit standard question identifiers and similarity scores, and the actual hit information of multiple rounds of testing includes inconsistent rounds and the actual verbal node tags of the corresponding rounds; Step 4: Output the test result data.

[0012] Optionally, in the above scheme, step 2 includes: In a single round of testing, recall is performed based on the question text in the vector retrieval library corresponding to the associated knowledge question and answer library, and the standard question identifier with the highest similarity and its similarity score are obtained as the actual result of the single round of testing; In multi-round testing, a session context is established for the user input sequence and input is entered into the associated workflow scripts in rounds. The output script node label after each round of input is obtained as the actual result of this round.

[0013] Optionally, in the above scheme, step 3 includes: In a single round of testing, the standard question identifier with the highest similarity is compared with the expected standard question identifier to generate a judgment result for the single round of testing; In multiple rounds of testing, the actual word node labels in each round are compared with the expected word node labels in the corresponding round to generate round-by-round judgment results.

[0014] Optionally, in the above scheme, the method further includes: During test execution, the total number of tests, the number of tests completed, the number of tests passed, and the number of tests failed are output. The judgment result entries corresponding to the completed test data and the detailed information of the entries are also output. Upon receiving a termination command, the processing of any remaining unexecuted test data is halted, and incomplete test data is identified.

[0015] Optionally, in the above scheme, the method further includes: Record exception information such as timeouts, no response, or call errors that occur during test execution and associate them with the corresponding test data; Step 4 includes: After the test is completed or terminated, a summary result is generated based on the executed test data, including at least the start time, end time, number of tests executed, and number of tests that failed. The executed test data are grouped and summarized according to the same expected results, and a test result report is generated. Each group includes the number of tests, the number of passes, and the corresponding detailed list. The expected result is the expected standard question identifier in a single round of testing and the expected dialogue node label in a multi-round test. For test data that is determined to be unsuccessful, the actual hit information is recorded in the test result report. The actual hit information includes the standard question identifier and similarity score of the actual hit in a single round of testing, and includes the inconsistent rounds and the actual speech node labels of the corresponding rounds in a multi-round test. The test result report includes at least the test input, expected result, actual result and judgment result, and includes similarity score in single-round tests and speech node tags in multi-round tests.

[0016] Compared with the prior art, this application has at least the following beneficial effects: This application uses a test set (test input + expected result) as a unified driving entry point. Testers only need to write and upload the single-turn question or multi-turn dialogue input to be verified and its expected result. The test configuration unit can then select single-turn or multi-turn input and associate it with the knowledge question and answer base or workflow scripts respectively, thus transforming the test behavior that originally required manual input line by line into a task that can be batch-scheduled by the system. On this basis, the test execution and judgment unit batch drives the test set and obtains actual results that can be directly used for verification. The single-turn input outputs at least the standard question identifier and similarity score, and the multi-turn input outputs at least the script node label. These results are then compared with the expected results to generate a judgment result. Therefore, the test conclusion no longer relies on manual subjective judgment and manual recording, but is automatically obtained by a unified comparison rule. At the same time, the process recording and output unit outputs the execution process and test result data in a structured manner, so that the same batch of test data can be repeatedly run on different versions of knowledge bases or script workflows and the results can be compared. Therefore, it can be deduced that the system can reduce the repetitive work caused by manual data entry, recording and verification, and support single-round hit verification and multi-round process landing point verification through locatable actual results such as "standard question identifier / similarity" and "node label". This solves the problems of low efficiency of single-round testing, high cost of retesting and difficulty in branch verification and deviation location in multi-round testing in the existing technology. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a system module for single-round and multi-round performance testing of a text robot, provided in one embodiment of this application. Figure 2 This is a schematic diagram illustrating a round-based effect test analysis according to an embodiment of this application. Figure 3 A flowchart of a round-based effect testing system provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Currently, text robots are mainly divided into two types: one is a single-turn dialogue text robot that relies on FAQs. Its working logic is to use the language understanding capabilities of a large model to perform semantic parsing on the user's input questions, and then match them in the FAQ knowledge base to find the most similar standard questions and provide the answers to the user. The other type is a multi-turn dialogue text robot that relies on a conversational workflow. It interacts with users based on a series of pre-set conversational scripts and fixed process logic. The conversational workflow breaks down the dialogue into multiple steps according to specific business scenarios and interaction needs. Each step has clear triggering conditions and corresponding response scripts. When a user initiates a dialogue, the robot will guide the user step by step according to the set process, triggering the corresponding conversational nodes based on the user's response, thereby completing the entire interaction process.

[0020] Currently, single-round dialogue testing relies on manual input of each question into the question-and-answer database, followed by manual recording of the returned standard question ID and answer. The correctness of the returned result must then be marked against the given expected outcome. When it's necessary to repeatedly verify this data, the entire manual testing process must be executed again.

[0021] During multi-turn dialogue testing, it's not only necessary to record the robot's responses round by round, but also to verify that each round's corresponding exit node in the overall workflow meets expectations. When a round's test result deviates from expectations, the troubleshooting process becomes significantly more difficult: because multi-turn dialogues are strongly correlated with previous rounds, an anomaly in a particular round might be a chain reaction caused by a deviation in a previous round's interaction. Therefore, it's essential to trace the entire process back from the beginning of the dialogue, checking the transitions between rounds to pinpoint the problematic round.

[0022] Whether it is a single-round or multi-round corpus, the corpus must be manually entered one by one (round by round) during the test, and the returned standard question answers, speech nodes and other information must be manually recorded. Then, the correctness is marked and the accuracy is calculated by comparing with the expected results.

[0023] When the program needs to be optimized and updated, and the same batch of data needs to be retested, the entire manual operation must be performed from the beginning, with repeated data entry and verification. If problems occur, the entire process must be traced back.

[0024] In one embodiment, such as Figure 1 As shown, a system for single-round and multi-round effect testing of a text robot is provided. The system includes a test set management unit, a test configuration unit, a test execution and judgment unit, and a process recording and output unit. The test set management unit is used to provide test set templates and receive uploaded test sets. A test set contains multiple test data sets, and each test data set contains test inputs and the expected results corresponding to the test inputs. The test configuration unit is used to select the test type, which includes at least single-round testing and multi-round testing. It is also used to associate a knowledge question and answer base in single-round testing and a workflow script in multi-round testing. The test execution and judgment unit is used to drive the text robot to obtain actual results in batches based on the test set, and compare the actual results with the expected results to generate judgment results. The actual results of a single round of testing include at least the standard question identifier and similarity score, and the actual results of multiple rounds of testing include the utterance node label. The process recording and output unit is used to record test process information and output test result data.

[0025] This embodiment provides a system for single-round and multi-round performance testing of a text robot. The system can be deployed as a server-side application, a cloud service, or a localized testing tool. The system uses a "test task" as the basic operating unit. A test task is determined by the imported test set, the selected test type, and the bound test object, and generates traceable test result data during execution.

[0026] In its implementation, the test set management unit is used to import and manage test data. Testers can download templates, fill in test data, and upload them. The system parses the test set into structured records and saves them to storage media (such as a database or file storage) for subsequent batch scheduling and execution. To ensure task execution, the test set management unit can perform basic checks on field completeness, data type, and round number consistency, and provide locatable error messages for data that does not meet the conditions.

[0027] The test configuration unit is used to establish a "test type - test object" binding relationship. Single-round tests are bound to a knowledge question and answer base (e.g., a specific FAQ version or its corresponding vector retrieval index), while multi-round tests are bound to workflow dialogue (e.g., a specific workflow version or state machine configuration of the dialogue flow engine). This binding relationship is passed to the execution and decision unit in the form of task parameters, enabling it to correctly route to the corresponding question and answer retrieval interface or workflow engine instance during execution.

[0028] The test execution and judgment unit is used to perform batch execution and automatic judgment on test data in the test set. In a single-round scenario, the execution and judgment unit submits the question text to a knowledge question-answering base or its retrieval service, obtaining a standard question identifier and similarity score as the actual result. In a multi-round scenario, the execution and judgment unit submits user input round by round within the same conversation context, obtaining utterance node tags as the actual result in each round. Subsequently, the execution and judgment unit performs a comparison based on the expected results of the test set, generating a pass / fail judgment result and writing it into the test result data.

[0029] The process recording and output unit provides the ability to organize and output results during and after the execution of a test task. It can save the input, expected, actual, and judgment results of each test data in a structured form, and supports output as report files, result query interfaces, or visual list pages, facilitating subsequent review and record keeping.

[0030] This embodiment unifies single-round and multi-round testing into a single system framework of "test set driven - configuration binding - batch execution - automatic judgment - result output", which can replace the manual input and verification method and improve the reusability, traceability and scalability of test tasks.

[0031] In this embodiment, the test set template provided by the test set management unit is used to distinguish between single-round test data and multi-round test data, wherein: Single-round test data includes the question text and the corresponding expected standard question identifier; The multi-round test data includes user input sequences arranged by round and corresponding expected dialogue node label sequences for each round.

[0032] To ensure the system can stably handle both single-round and multi-round testing scenarios, this embodiment designs the test set template as a data structure that can distinguish between the two types of test data. The test set template can be in tabular form (such as CSV / Excel) or structured text form (such as JSON), but the meaning of its fields remains consistent so that the system can parse them uniformly.

[0033] In a single round of testing, each test data item contains at least the question text and the expected standard question identifier. The question text drives the retrieval of the knowledge question-answering base, and the expected standard question identifier is used to compare and determine the actual hit result. In engineering implementation, the expected standard question identifier can be the standard question ID, knowledge entry ID, or their equivalent unique code within the knowledge base.

[0034] In the multi-round test data, each test data set contains at least a user input sequence arranged by round and a sequence of expected dialogue node labels for each round. The user input sequence drives the workflow dialogue round by round, and the expected node labels for each round are used to verify the landing point of each round of dialogue. In implementation, the user input sequence and the expected labels for each round can be expressed using "multi-column round fields" (input_1 / input_2… and expect_1 / expect_2…) or "delimiter encoded sequences". The system parses them into a round list during import and verifies the consistency of the round count.

[0035] Regarding the definition of "round-by-round expectation" in multi-round testing, the system does not require that expectation labels be marked for all rounds. If the business only focuses on key rounds or the final exit, labels can be filled in only for key rounds in the template, and placeholders or reused labels can be set for non-key rounds. The system can still complete round-by-round comparison or key round comparison according to the configuration strategy.

[0036] This embodiment defines the minimum necessary fields and structural boundaries of single-round and multi-round test data at the template level, ensuring that the test data can be stably parsed and automatically executed by the system. This reduces the risk of non-executability caused by data preparation errors from the source and improves the standardization and maintainability of the test set.

[0037] In this embodiment, the process recording and output unit includes a progress control unit; The progress control unit is used to output the total number of tests, the number of tests completed, the number of tests passed, and the number of tests that failed during the test execution process. It is also used to output the judgment result items corresponding to the completed test data and the detailed information of the items. The progress control unit is also used to respond to termination commands to stop the processing of subsequent unexecuted test data and to identify incomplete test data.

[0038] In batch testing tasks, to facilitate monitoring of execution status and rapid location of problematic data, this embodiment sets up a progress control unit as part of the process recording and output unit. The progress control unit can be linked with the task scheduler and result storage module to form an output structure of "task-dimensional statistics + item-dimensional details".

[0039] In its implementation, the progress control unit can maintain the following statistics based on the task status table or task memory status: total number of tests, number of tests executed, number of tests passed, and number of tests failed. These statistics are updated after each test data point is evaluated and output externally via polling, push, or query interfaces. If the system uses concurrent execution, consistency of the statistics updates can be ensured through atomic counting or transaction updates.

[0040] The progress control unit is also used to output the judgment result items corresponding to the completed test data and the detailed information of the items. At the item level, it can output test data identifiers, judgment status, hit criteria identifiers and similarity scores for single-round scenarios, or node labels and round information for multi-round scenarios; at the detailed level, it can output complete requests / responses, comparison details, inconsistency locations, relevant log indexes, etc., so that testers can locate the source of the problem without leaving the system.

[0041] Upon receiving a termination command, the progress control unit can stop subsequent scheduling by setting a task termination flag or updating the task status to "terminating / terminated". For test data that has not yet been executed, the system marks it as incomplete; for test data that is being executed, either a "safe termination" strategy (stop scheduling new data and allow data in transit to complete) or a "forced termination" strategy (interrupt the call and record the reason for the interruption) can be used, and either strategy can be implemented.

[0042] This embodiment provides task-level progress statistics, completed item output, and termination control capabilities during execution, making batch testing observable and controllable. This facilitates quick understanding of the overall status, locating specific problem data, and stopping the task in a timely manner when necessary.

[0043] In this embodiment, the process recording and output unit further includes an exception recording unit; the exception recording unit is used to record exception information such as timeouts, no response, or call errors that occur during test execution, and to associate and store the exception information with the corresponding test data.

[0044] To differentiate between different types of problems such as "judgment failure" and "execution exception," this embodiment introduces an exception recording unit in the process recording and output unit. The exception recording unit is coupled with the execution and judgment process, capturing and recording exceptions on critical paths that call the robot or related services.

[0045] In practical implementation, the exception logging unit can cover at least three types of exceptions: call timeout, no response, and call error. Call timeout can be resolved by setting a request timeout period and recording the exception type and time consumption information after the timeout; no response can be identified and recorded by returning an empty response, an invalid format response, or a response with missing key fields; call error can record network errors, server error codes, authentication failures, parsing failures, etc., and save the error codes, error messages, and necessary context parameters.

[0046] Anomaly records are linked to test data and stored using test data identifiers or task sequence numbers. The system can display anomaly information in the results details, or write anomaly information as an independent field into the test results data or a separate anomaly list for subsequent statistics and screening.

[0047] The anomaly recording unit can also work in conjunction with retry strategies. For example, a limited number of retries can be configured for transient network fluctuations, and the number of retries and the final result can be recorded; if the retry still fails, the test data is terminated in an abnormal state while retaining complete abnormal link information.

[0048] This embodiment records timeouts, no response, and call errors in a structured manner and associates them with specific test data, enabling the testing system to accurately distinguish between "business logic deviations" and "execution chain anomalies," thereby improving the efficiency of problem localization and the reliability of result interpretation.

[0049] In this embodiment, the process recording and output unit further includes a summary report unit; The summary report unit is used to generate summary results and export test result reports based on the executed test data after the test is completed or terminated; the summary results include at least the start time, end time, number of tests executed, and number of tests that failed; The summary report unit is used to group and summarize the executed test data according to the same expected results, and output a detailed list for each group, including at least the number of tests and the number of passes for the group; where the expected result is the expected standard question identifier in a single round of testing, and the expected dialogue node label in a multi-round test. For test data that is judged to be unsuccessful, the test result report also includes actual hit information. In a single round of testing, the actual hit information includes the standard question identifier and similarity score of the actual hit. In a multi-round test, it includes the inconsistent rounds and the actual speech node labels of the corresponding rounds. The test results report should include at least the test input, expected results, actual results, and judgment results, and should include similarity scores in single-round tests and verbal node tags in multi-round tests.

[0050] To output analyzable and traceable conclusions after the test task is completed or terminated, this embodiment sets up a summary report unit to summarize and group the executed test data. The summary report unit can be generated all at once when the task is completed, or it can be incrementally maintained during the task and fixed at the end.

[0051] The summary results should include at least the start time, end time, number of tests executed, and number of tests that failed. The start time and end time can be taken from the task metadata or the task status transition timestamp; the number of tests executed can be defined as the number of test data that have generated judgment results or abnormal results; the number of tests that failed can be defined as the number of tests that failed (abnormalities can be counted separately or included in the failure list according to configuration).

[0052] In terms of grouping and summarizing, the summary report unit aggregates the executed test data according to the same expected results, and outputs the number of tests, the number of passes, and a detailed list for each group. To align with single-round / multi-round scenarios, the expected results for single-round tests use the expected standard question identifier as the grouping key, while the expected results for multi-round tests use the expected utterance node label as the grouping key. The detailed list must at least include the input, expected, actual, and judgment information for each test data point within the group to support further analysis and verification.

[0053] For test data deemed unsuccessful, the report further records the actual hit information. In a single round of testing, the actual hit information includes the standard question identifier and similarity score of the actual hit; in multi-round testing, the actual hit information includes the rounds where inconsistencies occurred and the actual dialogue node labels for that round. This design aims to provide sufficient location clues for subsequent analysis without repeating the testing.

[0054] The test result report can be implemented as a multi-sheet spreadsheet, a CSV file, or structured data exported via an interface. In addition to the minimum set of fields, the report fields can be expanded to include information such as exception type, call time, version number, and session ID to adapt to different engineering scenarios.

[0055] This embodiment simultaneously provides task-level summaries, grouped statistics aggregated according to expected results, and actual hit information output when a pass is not achieved. This allows the test results to not only provide a macro-level assessment of overall performance but also to quickly drill down to specific expected items and deviations, thereby enhancing the interpretability and analyzability of the results.

[0056] like Figure 2 As shown, this embodiment provides an automated verification method based on a test set for evaluating the response accuracy of a text robot in single-turn and multi-turn dialogue scenarios. The system includes the following modules: For the initial test set, download the test question template provided by the system, fill in the single-round or multi-round test dataset according to the format, and upload it to the system. The test data can come from user inquiries in actual production, or self-designed questions and dialogue scenarios. Select test type: single round / multiple rounds. Manually select the corresponding test type in the system based on the uploaded test sets. When selecting a knowledge question and answer base / workflow, for single-round testing, you need to specify the large model knowledge question and answer base to be used in this test; for multi-round testing, you need to specify the workflow script to be used in this test. To begin testing, after completing the configuration, click the "Start Test" button in the system, and the system will automatically execute the test task. Vector library recall: In a single round of testing, the vector library is used to recall the standard question with the highest matching degree and similarity score based on the submitted test question. Test progress display: After the test starts, the system will display the test progress, total number of test sets, number of tests completed, number of successful tests, and number of failures in real time. It also supports terminating the test. After termination, the number of completed tests and the number of uncompleted tests will be displayed. Error capture: If errors such as timeout or no response occur during execution, the system will record them. Test results output: During task execution, the system displays the real-time results of completed test questions and supports viewing result details. Results Summary: After the test task is completed, the system will automatically summarize the total number of tests, accuracy rate, number of failures, and start and end times of the tests. Additionally, it will provide a separate summary for data with the same expected results, displaying not only the accuracy rate corresponding to that expected result but also allowing users to view the specific results for each test question under that expected result. The system generates structured test results. After all tests are completed, the system will automatically generate a comprehensive report containing core indicators such as actual results, pass rate, and similarity score, making it easy to quickly view the overall test situation and compare detailed results.

[0057] The test results output includes the following feature rules: Single-round test: When the expected standard question is hit, the similarity score is automatically displayed; when the expected standard question is not hit, the actual standard question hit and the corresponding similarity score are automatically displayed. Multiple rounds of testing: When the expected dialogue node is hit, the dialogue node label is automatically displayed; when the expected dialogue node is not hit, the actual dialogue node label is automatically displayed.

[0058] In one embodiment, such as Figure 3 As shown, the following is combined Figure 3 This article introduces a workflow for testing interactive effects using the above method. The specific steps are as follows: Step 0: Download the test template provided by the system, and organize the test data and corresponding expected results. Step 1: Depending on the dataset type, choose either single-round or multi-round testing. Step 2: Upload the test dataset prepared in Step 0, and select the knowledge question-and-answer base or workflow to be tested. Step 3: The system will summarize and display the uploaded test questions based on the expected results. Step 4: Select a desired result in the page list to view all corresponding test questions. You can also delete or add test questions. Step 5: Click "Start Test" to execute the test task. Step 6: The test can be terminated at any time during the test. After termination, the system will automatically calculate and display the pass rate and related details based on the completed test data. Step 7: After the dataset test is completed, the system generates a downloadable results report containing core indicators such as the actual results for each test question, the overall pass rate, and the similarity score.

[0059] like Figure 2As shown, the effectiveness testing process in this embodiment includes a test initialization phase and a test execution phase. In the test initialization phase, the system initializes the test set to form single-round / multi-round test data and determines the test type as a single-round test or a multi-round test. In the test execution phase, the system selects the corresponding test object according to the selected test type; for single-round tests, it selects a knowledge question-and-answer base, and for multi-round tests, it selects workflow dialogue. It also receives a start test command to initiate the test task. When the selected type is a single-round test, the system further performs vector library recall processing to obtain the hit results corresponding to the test input. During the test task execution, the system outputs test progress information and outputs test results after the test ends. Subsequently, it performs branch processing based on the test execution status: when the test fails, it performs exception handling; when the test succeeds, it summarizes the test results in detail and generates structured test results for storage or subsequent output.

[0060] like Figure 3 As shown, the test task execution steps in this embodiment include: providing a test template for download and organizing the test corpus and corresponding expected results based on the template; selecting single-round or multi-round testing according to the test dataset type; uploading the organized test dataset and selecting the knowledge question-and-answer base or workflow script corresponding to this test; the system summarizes and outputs the test questions based on the expected results; the system provides filtering of expected results and query output of corresponding test questions, and supports adding or deleting test questions to update the test dataset; receiving a start test command to execute the test task; terminating the test task when a termination command is received during the test, and performing statistical output based on the completed test data, including the pass rate and related detailed information; generating a downloadable test result report after the test is completed, the report including the actual results corresponding to each test question, the overall pass rate, and indicator information related to the judgment.

[0061] This embodiment automates test tasks, replacing manual testing and reducing manual operation costs. Before testing, only the test corpus and expected results need to be maintained according to the template, improving the reusability of the test set and making maintenance more convenient. The system supports testing of single-turn and multi-turn dialogue scenarios and can be adapted to knowledge question-and-answer bases (single-turn) and workflow scripts (multi-turn), meeting the verification needs of different dialogue scenarios and improving the versatility of the method. The testing process is automatically executed by the system program and the results are summarized, providing a quantitative basis for evaluating the accuracy of responses and avoiding subjective judgment bias.

[0062] This embodiment provides templates for maintaining test corpora and expected results. Single-round testing retrieves the highest-matching standard question similarity score from a vector library. Multi-round testing uses phrasal node tags to specify the exit point for each round, ensuring the process proceeds as expected. For failed tests, single-round testing provides the actual hit standard question similarity score; multi-round testing provides the actual hit phrasal node tags. The system automatically generates core metrics such as overall pass rate and pass rate for individual expected results based on the uploaded test set.

[0063] In one embodiment, a method for single-round and multi-round performance testing of a text robot is provided, the method comprising: Step 1: Obtain a test set containing test inputs and expected results, and select the test type; associate with a knowledge question and answer base for single-round testing, and associate with workflow scripts for multi-round testing; Step 2: Drive the text robot in batches based on the test set to obtain the actual results corresponding to each test data; Step 3: Compare the actual results with the expected results to generate a judgment result and test result data. The test result data includes at least: test input, expected result, actual result, and judgment result. The actual result of a single round of testing includes the standard question identifier and similarity score, while the actual result of multiple rounds of testing includes the verbal node label. For test data that is judged as failing, the test result data also includes actual hit information. The actual hit information of a single round of testing includes the standard question identifier and similarity score of the actual hit, while the actual hit information of multiple rounds of testing includes the inconsistent rounds and the actual verbal node label of the corresponding round. Step 4: Output the test result data.

[0064] This embodiment provides a method for single-round and multi-round performance testing of a text robot, used to drive the execution flow of the aforementioned system. This method takes a test set as input and test result data as output, completing single-round and multi-round evaluations through automated execution and comparison.

[0065] In step 1, the test set is obtained and the test type is selected, while the test objects are associated and bound. Single-round tests are bound to a knowledge question-and-answer base, serving as the execution object for single-round retrieval and question-and-answer; multi-round tests are bound to workflow scripts, serving as the execution object for multi-round conversation progression and node output. This step can be completed through user interface configuration or API parameter passing, forming a task context.

[0066] In step 2, the system drives the text robot to execute in batches based on the test set. This can be implemented by executing sequentially according to the test data order, or concurrently if resource constraints are met. In concurrent scenarios, the round order of the same multi-round test data must be consistent with the session context. The execution output forms the actual results corresponding to each test data, providing a basis for subsequent comparisons.

[0067] In step 3, the actual results are compared with the expected results to generate a judgment result and test result data. The test result data includes at least the test input, expected result, actual result, and judgment result, and retains corresponding fields in single-round and multi-round scenarios (single-round includes standard question identifier and similarity score, multi-round includes utterance node label). When the judgment is not passed, the actual hit information is also recorded for subsequent location deviation reasons and review of the landing point.

[0068] In step 4, the test results data is output. The output can be a report file, an interface response, or persistent query results. The output process can be linked with summary and group statistics so that the overall conclusion and detailed evidence can be obtained in a single output.

[0069] This embodiment defines test configuration, batch execution, automatic judgment, and structured output in a closed loop through methodological step-by-step definition. This provides a repeatable and verifiable data foundation for single-round and multi-round evaluation processes, improving the standardization and consistency of the testing process.

[0070] In this embodiment, step 2 includes: In a single round of testing, recall is performed based on the question text in the vector retrieval library corresponding to the associated knowledge question and answer library, and the standard question identifier with the highest similarity and its similarity score are obtained as the actual result of the single round of testing; In multi-round testing, a session context is established for the user input sequence and input is entered into the associated workflow scripts in rounds. The output script node label after each round of input is obtained as the actual result of this round.

[0071] For single-round testing scenarios, this embodiment obtains verifiable actual results in step 2 through the retrieval capabilities of the knowledge question-answering base. In engineering implementation, vector retrieval can be used: after vectorizing the question text, candidate standard questions are retrieved from the index, and the identifier of the standard question with the highest similarity and its similarity score are output.

[0072] In practical implementation, the knowledge question-answering base can provide a unified search interface, returning Top 1 or Top K candidates and their relevance scores. To ensure the determinism of the judgment, this embodiment uses the Top 1 hit result as the actual result of this single-round test data, and writes the similarity score as a traceable field into the test result data. The similarity score can be cosine similarity, inner product score, or other relevance scores; the method is not limited.

[0073] In step 3, the system compares the Top 1 standard question identifier with the expected standard question identifier given in the test set to generate a judgment result. If they do not match, the actual hit information (the actual hit standard question identifier and the similarity score) is recorded according to the definition in this embodiment for subsequent analysis of questions such as "which standard question was wrongly hit and what the similarity level is".

[0074] This embodiment uses "standard question identifier + similarity score" as the actual result of a single round and judges accordingly, so that the single round test has a clear and verifiable judgment basis, and retains enough information to locate recall bias and adjust knowledge base configuration when it fails.

[0075] In this embodiment, step 3 includes: in a single-round test, comparing the standard question identifier with the expected standard question identifier with the highest similarity to generate a single-round test judgment result; in a multi-round test, comparing the actual speech node labels of each round with the expected speech node labels of the corresponding round to generate a round-by-round judgment result.

[0076] For multi-round testing scenarios, this embodiment establishes a session context for each piece of multi-round test data in step 2 and inputs user statements according to round. The session context can be represented by the session ID, process instance ID, or state object returned by the workflow engine, which is used to ensure that multiple rounds of input proceed within the same dialogue instance.

[0077] After each round of input, the system reads the corresponding output dialogue node label from the workflow dialogue engine as the actual result. The node label is used to identify the current endpoint of the workflow and can be a node ID, node code, or other unique identifier. If a dialogue has multiple label output formats, the main label field can be agreed upon as the comparison basis, or the labels can be selected for comparison according to a preset priority.

[0078] In step 3, the system compares the actual node labels in each round with the expected node labels for the corresponding round to generate a round-by-round judgment result. If a discrepancy occurs, the round with the discrepancy and the actual node label for that round are recorded as actual hit information. The system can choose to continue executing subsequent rounds to fully record the path after deviation, or it can stop executing subsequent rounds after the first discrepancy and record the reason for stopping. The specific decision can be determined by the implementation strategy.

[0079] In this embodiment, the method further includes: During test execution, the total number of tests, the number of tests completed, the number of tests passed, and the number of tests failed are output. The judgment result entries corresponding to the completed test data and the detailed information of the entries are also output. Upon receiving a termination command, the processing of any remaining unexecuted test data is halted, and incomplete test data is identified.

[0080] To achieve observability and controllability of the batch testing process at the method level, this embodiment outputs progress statistics and details of completed items during test execution. In practice, the statistics are updated after each test data item is executed and evaluated, and the resulting item is appended to the result set. Testers can obtain the latest progress and details of completed items in real time through a query interface or user interface.

[0081] Regarding the detailed output of entries, single-round entries can display the hit standard question identifier, similarity score, and expected comparison information; multi-round entries can display the node label trajectory for each round, the inconsistency rounds, and expected comparison information. The detailed output can also include anomaly log information, enabling testers to identify system stability issues or data problems before the task is completed.

[0082] Regarding termination control, upon receiving a termination command, the method stops scheduling subsequent unexecuted test data and marks it as incomplete. For test data currently being executed, options include either saving the completed data to the database or immediately interrupting the process and recording it as an exception; the specific implementation can be determined by the system's security and real-time requirements. The results of completed parts are retained and can be output after termination to obtain interim conclusions in resource-constrained or emergency situations.

[0083] This embodiment enhances the manageability and real-time visibility of batch testing by introducing progress output, item detail output, and termination control during the method execution process, enabling the testing process to locate problems while executing and to be stopped in a timely manner as needed.

[0084] In this embodiment, the method further includes: recording exception information such as timeouts, no response, or call errors that occur during test execution and associating it with the corresponding test data.

[0085] Step 4 includes: after the test is completed or terminated, generating a summary result based on the executed test data, which includes at least the start time, end time, number of tests executed, and number of tests that failed; The executed test data are grouped and summarized according to the same expected results, and a test result report is generated. Each group includes the number of tests, the number of passes, and the corresponding detailed list. The expected result is the expected standard question identifier in a single round of testing and the expected dialogue node label in a multi-round test. For test data that is judged to be unsuccessful, the actual hit information is recorded in the test result report. The actual hit information includes the standard question identifier and similarity score of the actual hit in a single round of testing, and the inconsistent rounds and the actual speech node labels of the corresponding rounds in a multi-round test. The test results report should include at least the test input, expected results, actual results, and judgment results, and should include similarity scores in single-round tests and verbal node tags in multi-round tests.

[0086] In this embodiment, when outputting test result data in step 4, a more refined output organization method is introduced, which is combined with the recording of abnormal information. After the abnormal information is associated with the test data, it can be exported in the form of fields or independent lists during the output stage, thereby ensuring that the output results include "judgment dimension information" and "execution link information".

[0087] The output first includes a task-level summary result. This summary result is derived from the executed test data and task metadata, and includes at least the start time, end time, number of tests executed, and number of tests that failed. This summary result can be used as a summary field on the report homepage or returned by the API to quickly understand the overall status of this test task.

[0088] The output further includes grouped summaries and test result reports based on the same expected results. The system aggregates data using expected results as the grouping key: single rounds use the expected standard question identifier as the grouping key, and multi-rounds use the expected utterance node label as the grouping key. Each group outputs the number of tests and the number of passes, along with a detailed list associated with that group; the detailed list includes at least the test input, expected results, actual results, and judgment results to support drill-down verification.

[0089] For test data deemed unsuccessful, the output phase records the actual hit information in the report. For single rounds, it records the standard question identifier and similarity score of the actual hit; for multiple rounds, it records the rounds with inconsistencies and the actual node labels for that round. Using this information, users can determine "where the deviation occurred and in which round" without re-running the test, and can combine similarity or node landing points to correct the direction.

[0090] Test result reports can be output as downloadable files or structured interfaces. Files can be organized using multiple sheets (summary sheet, group sheet, detail sheet, exception sheet), and interfaces can use paginated queries and support filtering by group keys. There are no restrictions on the output format.

[0091] This embodiment upgrades test results from "single-point judgment" to a structured delivery that can be summarized and evaluated, grouped and analyzed, and drilled down to locate by providing a combined output of summary, grouping and actual hit information in the output stage, thereby improving the usability of results and the efficiency of problem location.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A system for single-round and multi-round performance testing of a text robot, characterized in that, The system includes a test set management unit, a test configuration unit, a test execution and judgment unit, and a process recording and output unit; The test set management unit is used to provide test set templates and receive uploaded test sets. The test set contains multiple test data, and each test data contains test input and the expected result corresponding to the test input. The test configuration unit is used to select the test type, which includes at least single-round testing and multi-round testing, and is used to associate a knowledge question and answer base in single-round testing and associate a workflow script in multi-round testing; The test execution and judgment unit is used to drive the text robot to obtain actual results in batches based on the test set, and compare the actual results with the expected results to generate a judgment result. The actual results of a single round of testing include at least standard question identifiers and similarity scores, and the actual results of multiple rounds of testing include utterance node tags. The process recording and output unit is used to record test process information and output test result data.

2. The system according to claim 1, characterized in that, The test set templates provided by the test set management unit are used to distinguish between single-round test data and multi-round test data. The single-round test data includes the question text and the corresponding expected standard question identifier; the multi-round test data includes the user input sequence arranged by round and the corresponding expected dialogue node label sequence for each round.

3. The system according to claim 1, characterized in that, The process recording and output unit includes a progress control unit; The progress control unit is used to output the total number of tests, the number of tests completed, the number of tests passed, and the number of tests failed during the test execution process, and is also used to output the judgment result entries corresponding to the completed test data and the detailed information of the entries; The progress control unit is also used to respond to termination commands to stop the processing of subsequent unexecuted test data and to identify incomplete test data.

4. The system according to claim 3, characterized in that, The process recording and output unit also includes an exception recording unit; The exception recording unit is used to record exception information such as timeouts, no response, or call errors that occur during test execution, and to associate and store the exception information with the corresponding test data.

5. The system according to claim 3, characterized in that, The process recording and output unit also includes a summary report unit; The summary report unit is used to generate summary results and export test result reports based on the executed test data after the test is completed or terminated; wherein, the summary results include at least the start time, end time, number of executions, and number of failed tests; The summary report unit is used to group and summarize the executed test data according to the same expected results, and output a detailed list for each group, including at least the number of tests and the number of passes for the group; wherein, the expected result is the expected standard question identifier in a single round of testing, and the expected dialogue node label in a multi-round test. For test data that is determined to be unsuccessful, the test result report also includes actual hit information. In a single round of testing, the actual hit information includes the standard question identifier and similarity score of the actual hit. In a multi-round test, it includes the inconsistent rounds and the actual speech node labels of the corresponding rounds. The test result report includes at least the test input, expected result, actual result and judgment result, and includes similarity score in single-round tests and speech node tags in multi-round tests.

6. A method for single-round and multi-round performance testing of a text robot, characterized in that, The method includes: Step 1: Obtain a test set containing test inputs and expected results, and select the test type; associate with a knowledge question and answer base for single-round testing, and associate with workflow scripts for multi-round testing; Step 2: Drive the text robot in batches based on the test set to obtain the actual results corresponding to each test data; Step 3: Compare the actual results with the expected results to generate a judgment result and generate test result data; the test result data includes at least: test input, expected result, actual result, and judgment result; wherein, the actual result of a single round of testing includes standard question identifiers and similarity scores, and the actual result of multiple rounds of testing includes verbal node tags; and for test data judged as failing, the test result data also includes actual hit information, the actual hit information of a single round of testing includes the actual hit standard question identifiers and similarity scores, and the actual hit information of multiple rounds of testing includes inconsistent rounds and the actual verbal node tags of the corresponding rounds; Step 4: Output the test result data.

7. The method according to claim 6, characterized in that, Step 2 includes: In a single round of testing, recall is performed based on the question text in the vector retrieval library corresponding to the associated knowledge question and answer library, and the standard question identifier with the highest similarity and its similarity score are obtained as the actual result of the single round of testing; In multi-round testing, a session context is established for the user input sequence and input is entered into the associated workflow scripts in rounds. The output script node label after each round of input is obtained as the actual result of this round.

8. The method according to claim 6, characterized in that, Step 3 includes: In a single round of testing, the standard question identifier with the highest similarity is compared with the expected standard question identifier to generate a judgment result for the single round of testing; In multiple rounds of testing, the actual word node labels in each round are compared with the expected word node labels in the corresponding round to generate round-by-round judgment results.

9. The method according to claim 6, characterized in that, The method further includes: During test execution, the total number of tests, the number of tests completed, the number of tests passed, and the number of tests failed are output. The judgment result entries corresponding to the completed test data and the detailed information of the entries are also output. Upon receiving a termination command, the processing of any remaining unexecuted test data is halted, and incomplete test data is identified.

10. The method according to claim 9, characterized in that, The method further includes: Record exception information such as timeouts, no response, or call errors that occur during test execution and associate them with the corresponding test data; Step 4 includes: After the test is completed or terminated, a summary result is generated based on the executed test data, including at least the start time, end time, number of tests executed, and number of tests that failed. The executed test data are grouped and summarized according to the same expected results, and a test result report is generated. Each group includes the number of tests, the number of passes, and the corresponding detailed list. The expected result is the expected standard question identifier in a single round of testing and the expected dialogue node label in a multi-round test. For test data that is determined to be unsuccessful, the actual hit information is recorded in the test result report. The actual hit information includes the standard question identifier and similarity score of the actual hit in a single round of testing, and includes the inconsistent rounds and the actual speech node labels of the corresponding rounds in a multi-round test. The test result report includes at least the test input, expected result, actual result and judgment result, and includes similarity score in single-round tests and speech node tags in multi-round tests.

Images