Automatic testing method and device
Through automated testing methods, including data management, project management, model Q&A execution, result comparison, result analysis and test report generation, the problems of inefficient traditional testing methods and insufficient coverage scenarios are solved, and efficient and economical testing and model performance optimization are achieved.
Patent Information
- Application Number
- CN202411943808.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-03-07
AI Technical Summary
Traditional testing methods rely on manual verification, are inefficient and difficult to cover all possible Q&A scenarios, making it difficult to guarantee the accuracy and stability of command-type Q&A systems.
It provides an automated testing method, including data management, project management, model Q&A execution, result comparison, result analysis and test report generation, etc., through the automated testing framework, it significantly reduces manual intervention, improves testing efficiency, and improves model performance through feature matching technology and in-depth analysis.
Improve testing efficiency, significantly reduce manual intervention, improve model performance, provide strong support for optimizing model performance, and reduce test costs.
Smart Images
Figure CN119576797A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and specifically provides an automated testing method and device. Background Art
[0002] With the rapid development of artificial intelligence technology, large models are increasingly used in question-answering systems. However, as an important form of question-answering systems, the accuracy and stability of command-based question-answering are directly related to the user experience and actual application effect of the system.
[0003] Traditional testing methods often rely on manual verification of results, which is not only inefficient but also difficult to cover all possible question-and-answer scenarios. Summary of the invention
[0004] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides an automated testing method with strong practicability.
[0005] A further technical task of the present invention is to provide an automated testing device that is rationally designed, safe and applicable.
[0006] The technical solution adopted by the present invention to solve the technical problem is:
[0007] An automated testing method comprises the following steps:
[0008] S1, the data management module is responsible for the preparation of test data;
[0009] S2, the project management module is responsible for the overall management of the project;
[0010] S3, the model question-answering execution module is responsible for executing the question-answering task using the processed test data and obtaining the output results of the model;
[0011] S4, the result comparison module compares the output of the model with the expected correct answer;
[0012] S5, the result analysis module conducts in-depth analysis of the test results;
[0013] S6. The test report generation module automatically generates a detailed test report based on the test results.
[0014] Furthermore, in step S3, the model question-answering execution module interacts with the model interface to trigger the operation of the model and collect output results;
[0015] The model question-answering execution module processes multiple test data simultaneously, and each data is processed by an independent thread.
[0016] Furthermore, in step S3, it includes:
[0017] (1) Data input: accepting test data processed by the data management module;
[0018] (2) Model interaction: communicating with the model interface to trigger the model to process input data;
[0019] (3) Result collection: capturing the output results generated by the model to provide a basis for subsequent comparison and analysis;
[0020] (4) Data format conversion: convert the test data into an appropriate format according to the requirements of the model interface;
[0021] (5) Interface call: Use appropriate HTTP requests to send test data to the model interface. The request usually contains the input data and necessary parameters required by the model.
[0022] (6) Result analysis: parse the model output results from the response returned by the model interface;
[0023] (7) Exception handling: During the interface call and result parsing process, the module handles various exceptions and generates corresponding error logs;
[0024] (8) Thread pool management: The module uses a thread pool to manage multiple threads. The size of the thread pool is adjusted according to the system resources and test requirements;
[0025] (9) Task allocation: Each test data is encapsulated as a task and put into the task queue. The threads in the thread pool take the task from the task queue and execute it;
[0026] (10) Synchronization and mutual exclusion: During multi-threaded execution, it is necessary to ensure that access to shared resources is safe;
[0027] (11) Result collection and integration: After completing the task, each thread puts the output results of the model into the result set. After the module is tested, the result set is integrated.
[0028] Furthermore, in step S4, feature matching technology is used to extract key features from the model output and the correct answer for comparison one by one.
[0029] Furthermore, when performing a one-by-one comparison, the specific process is as follows:
[0030] (1) Feature extraction: First, text processing is performed. For text-type output results and correct answers, natural language processing technology is used to extract key features. Then, numerical processing is performed. For numerical output results, the numerical value itself is directly extracted or numerical comparison is performed.
[0031] (2) Comparison algorithm calculation: First, perform an exact match. For simple output results, use an exact match algorithm to compare whether the output result is completely consistent with the correct answer.
[0032] Then, similarity calculation is performed. For complex output results, a similarity calculation algorithm is used to evaluate the similarity between the output result and the correct answer;
[0033] Finally, feature matching, combining the text processing and numerical processing methods, extracts key features from the output results and the correct answers, and compares them using a feature matching algorithm;
[0034] (3) Result analysis: First, perform accuracy calculation. Based on the comparison results, calculate the accuracy of the model, that is, the proportion of correct answers. Then, perform recall calculation. For some tasks, it is also necessary to calculate the recall rate, that is, the proportion of correct answers correctly identified by the model to all correct answers. Finally, perform error analysis. For errors in the comparison results, perform a detailed error analysis to identify problems with the model and provide guidance for subsequent optimization.
[0035] Furthermore, in step S5, the data analysis module performs an in-depth analysis of the test results, including the error type, error distribution and stability of the model.
[0036] Furthermore, in step S6, the test report generation module automatically generates a detailed test report based on the test results, including a test overview, accuracy assessment and stability analysis, and records the test process, test results and test conclusions, providing an important reference for optimizing model performance.
[0037] An automated testing device, comprising: at least one memory and at least one processor;
[0038] The at least one memory is used to store a machine-readable program;
[0039] The at least one processor is used to call the machine-readable program to execute an automated testing method.
[0040] Compared with the prior art, the automated testing method and device of the present invention have the following outstanding beneficial effects:
[0041] The present invention can improve test efficiency and can significantly reduce manual intervention and improve test efficiency through an automated test framework.
[0042] Improve model performance: By analyzing the test results, we can find the shortcomings of the model in the question-answering task, providing strong support for optimizing model performance.
[0043] Reducing testing costs: Compared with traditional testing methods, the automated testing method of the present invention can reduce testing costs and improve the economic benefits of testing. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Attached Figure 1 It is a flow chart of an automated testing method;
[0046] Attached Figure 2 It is a schematic diagram of the project management page in an automated testing method. DETAILED DESCRIPTION
[0047] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] A best embodiment is given below:
[0049] like Figure 1-2 As shown, an automated testing method in this embodiment has the following steps:
[0050] S1, the data management module is responsible for the preparation of test data;
[0051] The data management module is responsible for the preparation of test data (such as data collection, cleaning, etc.). This module is the basis of testing work and ensures the accuracy and completeness of test data.
[0052] S2, the project management module is responsible for the overall management of the project;
[0053] The project management module is responsible for the overall management of the project, including editing project information, setting up the test environment, monitoring the test progress, etc. This module ensures the orderly progress of the test process and provides necessary project management support.
[0054] S3, the model question-answering execution module is responsible for executing the question-answering task using the processed test data and obtaining the output results of the model;
[0055] This module interacts with the model interface, triggers the model to run and collects the output results. In order to improve the test efficiency, the model question and answer execution module supports multi-threaded execution. The module can process multiple test data at the same time, and each data is processed by an independent thread.
[0056] The specific process is as follows:
[0057] (1) Data input: receiving processed test data from the data management module, which usually includes key information such as user questions and expected answers;
[0058] (2) Model interaction: communicating with the model interface to trigger the model to process input data;
[0059] (3) Result collection: Capturing the output results generated by the model to provide a basis for subsequent comparison and analysis.
[0060] (4) Data format conversion: According to the requirements of the model interface, the test data is converted into an appropriate format. For example, if the model interface requires data in JSON format, the module will convert the test data into a JSON string;
[0061] (5) Interface call: Use an appropriate HTTP request (such as a POST request) to send the test data to the model interface. The request usually contains the input data and necessary parameters required by the model.
[0062] (6) Result parsing: Parsing the model output results from the response returned by the model interface, such as parsing JSON, XML, or other format data;
[0063] (7) Exception handling: During the interface call and result parsing process, the module needs to be able to handle various exceptions, such as network errors, model operation errors, etc., and generate corresponding error logs;
[0064] (8) Thread pool management: The module uses a thread pool to manage multiple threads. The size of the thread pool can be adjusted according to the system resources and test requirements.
[0065] (9) Task allocation: Each test data is encapsulated as a task and put into the task queue. The threads in the thread pool take tasks from the task queue and execute them.
[0066] (10) Synchronization and mutual exclusion: During multi-threaded execution, it is necessary to ensure that access to shared resources is safe. Therefore, the module uses synchronization mechanisms to ensure mutual exclusion access between threads.
[0067] (11) After each thread completes its task, it puts the model’s output into a result set. After the module is tested, it integrates the result set for subsequent comparison and analysis.
[0068] S4, the result comparison module compares the output of the model with the expected correct answer;
[0069] The result comparison module compares the model output with the expected correct answer to analyze the model's accuracy. This module uses feature matching technology to extract key features from the model output and the correct answer for one-by-one comparison, thereby more accurately and comprehensively evaluating the model's performance.
[0070] The specific process is as follows:
[0071] (1) Feature extraction: First, text processing is performed. For text-type output results and correct answers, natural language processing techniques (such as word segmentation and part-of-speech tagging) are used to extract key features. Then, numerical processing is performed. For numerical output results, the numerical value itself is directly extracted or numerical comparison is performed.
[0072] (2) Comparison algorithm calculation: First, perform an exact match. For simple output results, use an exact match algorithm to compare whether the output result is completely consistent with the correct answer.
[0073] Then, similarity calculation is performed. For complex output results, a similarity calculation algorithm (such as cosine similarity, Jaccard similarity, etc.) is used to evaluate the similarity between the output result and the correct answer;
[0074] Finally, feature matching, combining the text processing and numerical processing methods, extracts key features from the output results and the correct answers, and compares them using feature matching algorithms (such as rule-based matching, machine learning-based matching, etc.);
[0075] (3) Result analysis: First, perform accuracy calculation. Based on the comparison results, calculate the accuracy of the model, that is, the proportion of correct answers. Then, perform recall calculation. For some tasks, it is also necessary to calculate the recall rate, that is, the proportion of correct answers correctly identified by the model to all correct answers. Finally, perform error analysis. For errors in the comparison results, perform a detailed error analysis to identify problems with the model and provide guidance for subsequent optimization.
[0076] S5, the result analysis module conducts in-depth analysis of the test results;
[0077] The data analysis module conducts in-depth analysis of the test results, including the model's error type, error distribution, stability, etc. Through data analysis, potential problems and improvement directions of the model can be discovered, providing data support for optimizing model performance.
[0078] S6. The test report generation module automatically generates a detailed test report based on the test results;
[0079] The test report generation module automatically generates a detailed test report based on the test results, including a test overview, accuracy assessment, stability analysis, etc. It records the test process, test results and test conclusions, providing an important reference for optimizing model performance.
[0080] Based on the above method, an automated testing device in this embodiment includes: at least one memory and at least one processor;
[0081] The at least one memory is used to store a machine-readable program;
[0082] The at least one processor is used to call the machine-readable program to execute an automated testing method.
[0083] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.
[0084] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An automated testing method, characterized in that: The steps are as follows: S1, the data management module is responsible for the preparation of test data; S2, the project management module is responsible for the overall management of the project; S3, the model question-answering execution module is responsible for executing the question-answering task using the processed test data and obtaining the output results of the model; S4, the result comparison module compares the output of the model with the expected correct answer; S5, the result analysis module conducts in-depth analysis of the test results; S6. The test report generation module automatically generates a detailed test report based on the test results.
2. An automated testing method according to claim 1, characterized in that: In step S3, the model question-answering execution module interacts with the model interface to trigger the running of the model and collect output results; The model question-answering execution module processes multiple test data at the same time, and each data is processed by an independent thread.
3. An automated testing method according to claim 2, characterized in that: In step S3, it includes: (1) Data input: accepting test data processed by the data management module; (2) Model interaction: communicating with the model interface to trigger the model to process input data; (3) Result collection: capturing the output results generated by the model to provide a basis for subsequent comparison and analysis; (4) Data format conversion: convert the test data into an appropriate format according to the requirements of the model interface; (5) Interface call: Use appropriate HTTP requests to send test data to the model interface. The request usually contains the input data and necessary parameters required by the model. (6) Result analysis: parse the model output results from the response returned by the model interface; (7) Exception handling: During the interface call and result parsing process, the module handles various exceptions and generates corresponding error logs; (8) Thread pool management: The module uses a thread pool to manage multiple threads. The size of the thread pool is adjusted according to the system resources and test requirements; (9) Task allocation: Each test data is encapsulated as a task and put into the task queue. The threads in the thread pool take the task from the task queue and execute it; (10) Synchronization and mutual exclusion: During multi-threaded execution, it is necessary to ensure that access to shared resources is safe; (11) Result collection and integration: After completing the task, each thread puts the output results of the model into the result set. After the module is tested, the result set is integrated.
4. An automated testing method according to claim 3, characterized in that: In step S4, feature matching technology is used to extract key features from the model output and the correct answer and compare them one by one.
5. An automated testing method according to claim 4, characterized in that: When performing a one-by-one comparison, the specific process is as follows: (1) Feature extraction: First, text processing is performed. For text-type output results and correct answers, natural language processing technology is used to extract key features. Then, numerical processing is performed. For numerical output results, the numerical value itself is directly extracted or numerical comparison is performed. (2) Comparison algorithm calculation: First, perform an exact match. For simple output results, use an exact match algorithm to compare whether the output result is completely consistent with the correct answer. Then, similarity calculation is performed. For complex output results, a similarity calculation algorithm is used to evaluate the similarity between the output result and the correct answer; Finally, feature matching, combining the text processing and numerical processing methods, extracts key features from the output results and the correct answers, and compares them using a feature matching algorithm; (3) Result analysis: First, perform accuracy calculation. Based on the comparison results, calculate the accuracy of the model, that is, the proportion of correct answers. Then, perform recall calculation. For some tasks, it is also necessary to calculate the recall rate, that is, the proportion of correct answers correctly identified by the model to all correct answers. Finally, perform error analysis. For errors in the comparison results, perform a detailed error analysis to identify problems with the model and provide guidance for subsequent optimization.
6. An automated testing method according to claim 5, characterized in that: In step S5, the data analysis module performs an in-depth analysis of the test results, including the error type, error distribution and stability of the model.
7. An automated testing method according to claim 6, characterized in that: In step S6, the test report generation module automatically generates a detailed test report based on the test results, including a test overview, accuracy assessment, and stability analysis, and records the test process, test results, and test conclusions, providing an important reference for optimizing model performance.
8. An automated testing device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.
Citation Information
Cited By
Model testing method and device
CN121858463A