Automatic testing system and method capable of being arranged and executed in batches

Abstract

The invention provides an automatic test system and method capable of being arranged and executed in batches, and the system comprises a test case arrangement plan table which is used for customizing the starting time, the precondition and the dependency relationship of each test item, whether execution is terminated or not when the test item fails, grouping execution, result archiving, product dependency and user-defined labels; the default configuration management module is used for parameterizing execution and supporting simple configuration to realize visual operation of the test script; the software framework supports an object-oriented language and provides multi-process processing to improve execution efficiency; and the distributed structure is used for realizing concurrent parallel execution across a plurality of processors or machines and supporting synchronous import and concurrent asynchronous execution of a plurality of test items. Through the support of multi-process processing and a distributed structure, the advantages of a multi-core processor and computing resources of cross-network nodes can be fully utilized, multiple test items are executed in parallel, the execution efficiency of automatic testing is greatly improved, and the overall testing period is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing automation and intelligentization, and in particular to a batch programmable execution automation testing system and method. Background Art

[0002] In the field of software testing automation and intelligence, with the acceleration of software development cycle and the improvement of software quality requirements, automated testing has become a key link in ensuring software quality and accelerating product launch; automated testing allows test cases to be automatically executed by writing scripts, which not only improves testing efficiency but also reduces human errors; however, in actual operations, testers need to arrange a large number of test cases according to specific logic for execution; there are various relationships between these test cases, such as dependency, causal, before-and-after, and synchronization.

[0003] Deficiencies of existing technology: 1. Inefficient manual calls: In the existing automated testing process, the execution of test cases often requires manual calls to the corresponding test scripts. This manual operation is not only time-consuming, but also cannot guarantee that the test cases can be executed correctly in the expected order, resulting in test results that may not meet expectations.

[0004] 2. Lack of flexibility and customization: Current automated testing tools usually do not support flexible adjustment of test case execution strategies, such as startup time, preconditions, dependencies, etc. It is difficult for testers to customize the test process according to specific needs, which limits the flexibility of testing work.

[0005] 3. Low execution efficiency: Due to differences in hardware performance in different test environments, existing automated testing solutions generally use a single-process or multi-threaded approach to execute tests. This approach cannot fully utilize the advantages of multi-core processors, resulting in low test efficiency, especially when dealing with a large number of test cases.

[0006] 4. Insufficient resource utilization: Existing solutions fail to effectively utilize computing resources for parallel processing, especially in distributed environments, and cannot achieve efficient collaboration across nodes, making the overall testing process slow and inflexible.

[0007] Therefore, the prior art has deficiencies and needs further improvement. Summary of the invention

[0008] In view of the problems existing in the prior art, the present invention provides a batch programmable execution automated testing system and method.

[0009] To achieve the above object, the specific scheme of the present invention is as follows: The present invention provides a batch programmable execution automated testing system, the system comprising: Test case scheduling table, used to customize the start time, preconditions, dependencies, whether to terminate execution upon failure, group execution, result archiving, product dependencies and user-defined tags of each test item; The default configuration management module is used for parameterized execution and supports simple configuration to visually run test scripts; Software framework that supports object-oriented languages ​​and provides multi-process processing to improve execution efficiency; Distributed structure, used to achieve concurrent parallel execution across multiple processors or machines, supporting synchronous import and concurrent asynchronous execution of multiple test items.

[0010] Furthermore, the default configuration management module also allows users to save commonly used configuration templates so as to quickly apply them to new test plans.

[0011] Furthermore, the software framework includes: Test file loading module, used to read the test batch plan file and convert it into a form that can be processed by the program; Test plan preparation module, including test case retrieval and filtering, precondition sorting, condition verification checking, and execution command generation; Test plan parsing module, used to parse the test plan and schedule test cases; Batch execution module, used to coordinate the execution of concurrent test cases; Report generation module collects test results and generates reports.

[0012] Furthermore, the software framework also includes: Logging module, used to record log information during the test process; The result merging module aggregates the results of each test case into a complete report and converts it into the final format for viewing and analysis.

[0013] The present invention also provides a batch programmable automated testing method, based on the above automated testing system, the method comprises the steps of: S1, create a test batch plan file, which contains specific test cases and their execution order information; S2, reading the above plan file through the test file loading module; S3, schedules the execution of test cases according to the plan file, and executes test cases concurrently in a multi-process manner; S4, generates logs and records the test status during execution; S5, collect test results and generate reports.

[0014] Further, in step S1, Adjust the test execution strategy according to different test environments to ensure efficient execution under different hardware conditions; Provides a user-friendly interface for testers to manage and monitor the testing process.

[0015] Furthermore, it also includes: in step S1, Allow users to customize test case labels for easy classification management and query.

[0016] Furthermore, it also includes: in step S3, For each test item, test cases can be automatically generated and executed according to the preset schedule; Dynamically handle dependencies between test cases to ensure the correct execution order; When a test item fails and is set to terminate execution early, the execution of subsequent related test items is stopped.

[0017] Furthermore, it also includes: in step S3, Distributed deployment is adopted to realize the synchronous execution of test cases on multiple machines, thereby improving the overall testing efficiency; Supports multiple execution modes, including synchronous, asynchronous, sequential, and batch execution.

[0018] Furthermore, it also includes: in step S5, After the test is completed, the results of all test cases are automatically merged and the final test report is generated; Automatically archive individual test results for easy subsequent review and analysis; Save log information to a log file for further review and troubleshooting.

[0019] The technical solution of the present invention has the following beneficial effects: 1. Improve test efficiency: Through the support of multi-process processing and distributed structure, it can fully utilize the advantages of multi-core processors and computing resources across network nodes to realize parallel execution of multiple test items, which greatly improves the execution efficiency of automated testing and shortens the overall test cycle.

[0020] 2. Enhanced flexibility and customization: The system provides a detailed test case scheduling schedule, allowing each test item to be customized based on multiple factors such as startup time, preconditions, dependencies, etc. This high degree of configurability allows testers to flexibly adjust the test process according to specific needs, ensuring that the test process is more in line with actual application scenarios.

[0021] 3. Simplified test management: Default configuration management and parameterized execution functions simplify the test plan preparation process; testers can quickly call preset test plans and run visual test scripts through simple configuration, reducing the complexity of manual operations and improving work efficiency.

[0022] 4. Ensure execution order and expected results: By identifying and processing the dependencies between test cases, the system can automatically determine whether it is necessary to terminate the entire test process early or skip only related subsequent test items when a test item fails; this ensures that the test proceeds smoothly according to the predetermined logic and avoids result deviations caused by incorrect sequences.

[0023] 5. Support multiple programming languages: Based on a software framework that can be implemented using any object-oriented language (such as Python, Ruby, Java, C++, C#, Rust, Go, etc.), the present invention has good versatility and extensibility and is suitable for projects with different technology stacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the software framework of the present invention; Figure 2 is an example diagram of the configuration management interface of the present invention; Figure 3 is a schematic diagram of distributed parallel execution of the present invention; Figure 4 is a logic diagram of the present invention; Figure 5 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments; it can be understood that the specific embodiments described herein are only used to explain the present invention rather than to limit the present invention; it should also be noted that, for ease of description, only the parts related to the present invention rather than all are shown in the accompanying drawings.

[0026] Combination Figure 1-Figure 5 As shown, the present invention provides a batch programmable execution automated testing system, the system comprising: Test case scheduling table, used to customize the start time, preconditions, dependencies, whether to terminate execution upon failure, group execution, result archiving, product dependencies and user-defined tags of each test item; The default configuration management module is used for parameterized execution and supports simple configuration to visually run test scripts; Software framework that supports object-oriented languages ​​and provides multi-process processing to improve execution efficiency; Distributed structure, used to achieve concurrent parallel execution across multiple processors or machines, supporting synchronous import and concurrent asynchronous execution of multiple test items.

[0027] The default configuration management module also allows users to save commonly used configuration templates so as to quickly apply them to new test plans.

[0028] The software framework includes: Test file loading module, used to read the test batch plan file and convert it into a form that can be processed by the program; Test plan preparation module, including test case retrieval and filtering, precondition sorting, condition verification checking, and execution command generation; Test plan parsing module, used to parse the test plan and schedule test cases; Batch execution module, used to coordinate the execution of concurrent test cases; Report generation module collects test results and generates reports.

[0029] The software framework also includes: Logging module, used to record log information during the test process; The result merging module aggregates the results of each test case into a complete report and converts it into the final format for viewing and analysis.

[0030] The present invention also provides a batch programmable automated testing method, based on the above automated testing system, the method comprises the steps of: S1, create a test batch plan file, which is generated by the test case arrangement plan table, and contains specific test cases and their execution order information; S2, reading the above plan file through the test file loading module; S3, schedules the execution of test cases according to the plan file, and executes test cases concurrently in a multi-process manner; S4, generates logs and records the test status during execution; S5, collect test results and generate reports.

[0031] In step S1, Adjust the test execution strategy according to different test environments to ensure efficient execution under different hardware conditions; Provides a user-friendly interface for testers to manage and monitor the testing process.

[0032] Also includes, in step S1, Allow users to customize test case labels for easy classification management and query.

[0033] Also includes, in step S3, For each test item, test cases can be automatically generated and executed according to the preset schedule; Dynamically handle dependencies between test cases to ensure the correct execution order; When a test item fails and is set to terminate execution early, the execution of subsequent related test items is stopped.

[0034] Also includes, in step S3, Distributed deployment is adopted to realize the synchronous execution of test cases on multiple machines, thereby improving the overall testing efficiency; Supports multiple execution modes, including synchronous, asynchronous, sequential, and batch execution.

[0035] The specific implementation of distributed deployment includes the following: 1. System architecture design, Master Node: responsible for task allocation, resource management, status monitoring, etc.

[0036] Worker Nodes: perform specific test tasks and return the results to the master node.

[0037] 2. Technology selection Choose the right technology stack to support distributed deployment: Message queues: such as RabbitMQ and Kafka, used for task distribution and result collection.

[0038] Distributed coordination services: such as Zookeeper and etcd, used for coordination and service discovery between nodes.

[0039] Containerization technologies: such as Docker and Kubernetes, facilitate environment consistency and resource management.

[0040] Remote procedure call framework: such as gRPC and Thrift, used for cross-node communication.

[0041] 3. Test plan document preparation Create a test plan file containing all the test cases and their dependencies.

[0042] Upload the test plan file to the master node or shared storage.

[0043] 4. Master node function Task parsing and scheduling: Parse the test plan file and schedule tasks based on the dependencies and priorities between test cases.

[0044] Load balancing: Distribute tasks evenly to each working node.

[0045] Status monitoring: monitor the status of each working node in real time to ensure normal execution of tasks.

[0046] Result summary: Collect the test results from each working node and generate a final report.

[0047] 5. Worker Node Function Receive tasks: Receive assigned tasks from the master node.

[0048] Execute test: Run the specified test case and record the execution log.

[0049] Report results: Send the test results back to the master node through the message queue or directly.

[0050] Exception handling: Capture and handle exceptions that may occur during execution.

[0051] 6. Configuration Management Environment configuration: Ensure that the test environment on each working node is consistent. This step can be simplified by using containerization technology.

[0052] Parameter configuration: allows global parameters and test case-specific parameters to be set on the master node and passed to the worker nodes.

[0053] 7. Security and permissions Network isolation: Ensure that only authorized worker nodes can access the master node.

[0054] Authentication: Verify the identity of the working node to prevent illegal access.

[0055] Data encryption: Encrypt transmitted data to protect sensitive information.

[0056] 8. Logging and Monitoring Logging: Detailed execution logs are recorded on each worker node and stored centrally for analysis.

[0057] Performance monitoring: Monitor system performance indicators such as CPU usage, memory usage, network latency, etc. to optimize resource allocation.

[0058] 9. Elastic scaling Automatic scaling: Dynamically adjust the number of worker nodes based on the current workload, such as using Kubernetes’ automatic scaling feature.

[0059] Fault recovery: When a working node fails, the master node can automatically detect and reallocate tasks to other available nodes.

[0060] 10. Implementation steps Requirements analysis: clarify test requirements and goals.

[0061] Architecture Design: Design the architecture of distributed systems.

[0062] Environment construction: Build the infrastructure of the master node and working nodes.

[0063] Code development: Write the logic code for the master node and the worker node.

[0064] Integration testing: Integration testing is performed in a small-scale environment to ensure that the components work together.

[0065] Deployment and launch: Deploy the system in a production environment and conduct large-scale testing.

[0066] Continuous optimization: Continuously optimize system performance and stability based on actual operating conditions.

[0067] Through the above steps and technical means, an efficient distributed automated testing system can be effectively implemented, thereby greatly improving testing efficiency and quality.

[0068] In step S5, After the test is completed, the results of all test cases are automatically merged and the final test report is generated; Automatically archive individual test results for easy subsequent review and analysis; Save log information to a log file for further review and troubleshooting.

[0069] Example: The field of intelligent software testing automation involves the automatic generation and execution of test cases in synchronization, asynchronous, sequence, batch, or according to a preset schedule, as well as dynamic processing.

[0070] During the software testing process, testers need to arrange and execute a large number of written test cases according to a certain logic; there are many relationships between test cases, such as dependency, cause-and-effect, before-and-after, and synchronization. During execution, existing tests can only rely on manual calls to corresponding test cases, which is inefficient and cannot guarantee that the execution order is as expected.

[0071] Moreover, different test environments have poor universality and low efficiency due to differences in execution time caused by hardware, and require excessive manpower.

[0072] It can be seen that when testing software performance, there are currently defects such as poor versatility, inability to guarantee the test case execution experience, or low efficiency and excessive manpower consumption.

[0073] 1. Software use case arrangement, providing a test case arrangement schedule, so that each test item can be individually customized according to the following options: a) Start-up time; b) Preconditions; c) Dependent pre-test items or test groups; d) Whether the failed step terminates execution early; e) Group execution; f) Archiving of single test results; g) Are there any other WABI product dependencies and optional parameters? h) User-defined label chart is as follows:

[0074] 2. Provide default configuration management and parameterized execution. Testers can directly call the test plan and visualize the test script running with simple configuration.

[0075] An example of the configuration management interface is as follows Figure 2 As shown: 3. Software testing software framework, the language of which can be any object-oriented language, especially Python, Ruby, Java, C++, C#, RUST, Go, among which the relevant technical points of this article will be explained by taking Python language as an example; The code is as follows: class Case_Plan: def __init__(self, record): self.case_id = record['case_id_string'] self.start_time = record['start_time_string'] self.preceding_case_id = record['preceding_case_id_string'] self.stop_execution_if_fail = record['stop_execution_if_fail_string'] self.group = record['group_string'] self.group_in_report = record['group_in_report_string'] self.protocol = record['protocol_string'] self.status = Case_Status.NOYETSTART self.preceding_group = [] run_group.add_case(self.group, self) report_group.add_case(self.group_in_report, self) 4. WABI-Batch replaces the original single-process multi-threaded processing method with multi-process, fully utilizes the multiple processors of the test machine, supports the simultaneous import of multiple test items, and concurrent parallel asynchronous execution, thus improving execution efficiency.

[0076] The code is as follows: while(True): for idx, case in enumerate(cases_list): if(case_status_dic[case.case_id] == Case_Status.ALL_COMPLETE andcase.stop_execution_if_fail == "Y" and case_result_dic[case.case_id]): stop_and_exit = True break if(case_status_dic[case.case_id] == Case_Status.NOYETSTART andcase.condition_satisfy(case_status_dic, case_result_dic)): logger.info("Step %s started" % case.case_id) pool.apply_async(case.run_case, args=(case_status_dic, case_result_dic, env_config,)) case_status_dic[case.case_id] = Case_Status.STARTED current_running_case_amount += 1 # time.sleep(1) if current_running_case_amount>= total_case or stop_and_exit == True: break 5. Distributed structure: after orchestration and deployment, distributed parallel execution can be achieved.

[0077] Distributed parallel execution diagram, such as Figure 3 shown.

[0078] 6. If Figure 1 As shown, the software framework includes: test file loading module, test plan preparation module (including test case retrieval and filtering, precondition sorting, condition verification check, execution command generation), test plan parsing module, batch execution module, report generation module, etc.

[0079] Config: represents configuration information, which may be used to store some global parameters or settings.

[0080] Test Batch Plan File: It represents the test batch plan file, which is a file created by the tester and contains information such as specific test cases and their execution order.

[0081] file_loader: Used to read the above test batch plan file and convert it into a form that the program can handle.

[0082] batch_execute: Batch execution module, used to schedule and execute test cases.

[0083] multi_process: Multi-process module, indicating that this system supports multi-process concurrent execution of test cases to improve test efficiency.

[0084] case_execution: The execution module of a single test case, used to perform specific test tasks.

[0085] logger: Log module, which records the log information during the test process.

[0086] generate report: Generate report module, collects information about test execution and generates a report.

[0087] combine report: The merge report module summarizes the execution results of each test case into a complete report.

[0088] report_generator: Report generator, which converts the merged report into the final test result report.

[0089] Test Case Execution: Test case execution module, which shows its relationship with multi_process and WABI-Batch 2, indicating that the execution of test cases is completed through multi-process and the test results in Json format are generated.

[0090] Test Result Reports: The final test result report is generated for testers to review and analyze.

[0091] Log File: Log file, which saves all log information during the test process.

[0092] Tester: Tester, that is, the person who uses this system.

[0093] The whole process is as follows: the tester first creates a test batch plan file, and then reads the file through file_loader; then, the batch_execute module schedules the execution of test cases according to the plan, which involves multi-process concurrent execution (multi_process); during the execution process, the case_execution module is used to execute specific test cases, and generate logs (logger) and reports (generate report); finally, report_generator integrates all reports to form the final test result report, which is stored in Test Result Reports. The log will be written to Log File.

[0094] def prepare_preceding_case(self, cases_list): if self.preceding_case_id == "Nil": return if self.preceding_case_id == "Previous": idx = cases_list.index(self) if idx>0: self.preceding_group.append(cases_list[idx - 1]) elif re.match("Group:(.*)", self.preceding_case_id): group_name = re.match("Group:(.*)", self.preceding_case_id).groups()[0] if group_name in run_group.group_map.keys(): self.preceding_group = run_group.group_map[group_name] Technical advantages: According to the test arrangement plan of the present invention, the software under test is controlled to start running through the test control unit, and multiple concurrent and distributed parallel execution is performed, which shortens the running time of the test. This solves the current problems of low execution efficiency, the need for manual supervision of the execution of related test items, or low efficiency and excessive manpower consumption when testing software performance, and achieves the beneficial effects of strong versatility, separation of logical modules, high concurrency and parallelism, multiple execution orders, and higher efficiency.

[0095] Working principle: Test case arrangement: The system provides a test case scheduling table, which allows users to customize the start time, preconditions, dependencies and other attributes for each test item. These attributes ensure that the test cases can be executed according to the predetermined logic and sequence.

[0096] Users can further refine the test process by setting group execution, failed step processing rules (such as early termination or skipping subsequent test items), result archiving methods, and custom tags.

[0097] Default configuration management and parameterized execution: The system has built-in default configuration management function and supports parameterized execution, which means that testers can directly call the preset test plan and run the visual test script through simple configuration.

[0098] This configuration management simplifies the test preparation process, improves productivity, and also reduces the possibility of human error.

[0099] Object-oriented language support: The system is based on a software framework that supports any object-oriented language (such as Python, Ruby, Java, C++, C#, Rust, Go, etc.), which makes the system versatile and cross-platform.

[0100] Python is used as an example language to explain the key technical points and how to use the advantages of object-oriented programming to implement various functions of the system.

[0101] Multi-process concurrent execution: The new version uses a multi-process processing method to replace the original single-process multi-threading method. This change fully utilizes the multiple processor resources of the test machine and realizes synchronous import and concurrent asynchronous execution of multiple test items.

[0102] Multi-process concurrent execution significantly improves test efficiency, especially when dealing with a large number of test cases.

[0103] Distributed structure: The system supports a distributed structure, which can achieve distributed parallel execution across network nodes after proper orchestration and deployment.

[0104] Distributed execution not only enhances the system's scalability, but also improves its adaptability and execution speed in different environments.

[0105] Modular Architecture: The system architecture includes multiple functional modules, such as test file loading module, test plan preparation module (including test case retrieval and filtering, precondition sorting, condition verification check and execution command generation), test plan parsing module, batch execution module and report generation module.

[0106] The modular architecture design makes the system more flexible and easier to maintain.

[0107] Test control unit and automated execution: The test control unit is used to trigger the software under test to start running and coordinate the execution of each test item according to the test scheduling plan.

[0108] By formulating a detailed test orchestration plan, the system can automatically identify and handle the dependencies between test cases to ensure that the tests are executed in the expected order.

[0109] Detailed test report generation: The system provides a detailed test report generation function that can record the results of each round of testing in a structured form and support multiple output formats (such as HTML, PDF or XML).

[0110] The report includes performance indicator analysis to facilitate later analysis and problem location. It can also compare historical data to evaluate system performance change trends.

[0111] In summary, the present invention provides a highly configurable test case orchestration system, combined with multi-process concurrent execution, distributed structure and modular design, to achieve an efficient, flexible and reliable automated testing solution.

[0112] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the protection scope of the present invention.

Claims

1. A batch programmable automated testing system, characterized in that: The system includes: Test case scheduling table, used to customize the start time, preconditions, dependencies, whether to terminate execution upon failure, group execution, result archiving, product dependencies and user-defined tags of each test item; The default configuration management module is used for parameterized execution and supports simple configuration to visually run test scripts; Software framework that supports object-oriented languages ​​and provides multi-process processing to improve execution efficiency; Distributed structure, used to achieve concurrent parallel execution across multiple processors or machines, supporting synchronous import and concurrent asynchronous execution of multiple test items.

2. The batch programmable execution automated testing system according to claim 1, characterized in that: The default configuration management module also allows users to save commonly used configuration templates so as to quickly apply them to new test plans.

3. The batch programmable execution automated testing system according to claim 1, characterized in that: The software framework includes: Test file loading module, used to read the test batch plan file and convert it into a form that can be processed by the program; Test plan preparation module, including test case retrieval and filtering, precondition sorting, condition verification checking, and execution command generation; Test plan parsing module, used to parse the test plan and schedule test cases; Batch execution module, used to coordinate the execution of concurrent test cases; Report generation module collects test results and generates reports.

4. The batch programmable execution automated testing system according to claim 3, characterized in that: The software framework also includes: Logging module, used to record log information during the test process; The result merging module aggregates the results of each test case into a complete report and converts it into the final format for viewing and analysis.

5. A batch programmable automated testing method based on the automated testing system according to any one of claims 1 to 3, characterized in that: The method comprises the steps of: S1, create a test batch plan file, which contains specific test cases and their execution order information; S2, reading the above plan file through the test file loading module; S3, schedules the execution of test cases according to the plan file, and executes test cases concurrently in a multi-process manner; S4, generates logs and records the test status during execution; S5, collect test results and generate reports.

6. The batch programmable automated testing method according to claim 5, characterized in that: In step S1, Adjust the test execution strategy according to different test environments to ensure efficient execution under different hardware conditions; Provides a user-friendly interface for testers to manage and monitor the testing process.

7. The batch programmable automated testing method according to claim 5, characterized in that: Also includes: In step S1, Allow users to customize test case labels for easy classification management and query.

8. The batch programmable execution automated testing method according to claim 5, characterized in that: Also includes: In step S3, For each test item, test cases can be automatically generated and executed according to the preset schedule; Dynamically handle dependencies between test cases to ensure the correct execution order; When a test item fails and is set to terminate execution early, the execution of subsequent related test items is stopped.

9. The batch programmable automated testing method according to claim 5, characterized in that: Also includes: In step S3, Distributed deployment is adopted to realize the synchronous execution of test cases on multiple machines, thereby improving the overall testing efficiency; Supports multiple execution modes, including synchronous, asynchronous, sequential, and batch execution.

10. The batch programmable automated testing method according to claim 5, characterized in that: Also includes: In step S5, After the test is completed, the results of all test cases are automatically merged and the final test report is generated; Automatically archive individual test results for easy subsequent review and analysis; Save log information to a log file for further review and troubleshooting.

Images