Implementation method of variation test framework for distributed system

Abstract

The invention discloses an implementation method of a variation test framework for a distributed system, which comprises the following steps of: selecting a variation operator required by variation test from a variation operator library after a tester selects a system to be tested; then, the variation test framework generates variants on the basis of a source code of a system to be tested on the basis of the selected variation operator, and some invalid variants or equivalent variants are screened out through a method of combining static analysis and dynamic execution; and then entering a variant execution stage, sequentially loading a certain variant by a variant test framework according to coverage rate information of unit test of the system to be tested and a certain priority, running a test suite of the system to be tested on the system to be tested after variation, and collecting a result. And finally, after all variants are loaded and run, the variation test framework collects and arranges data to form a readable test report, so that the quality of the test suite of the to-be-tested system is reflected to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and is particularly applicable to the field of distributed system mutation testing in software testing. Specifically, it relates to a method for implementing a mutation testing framework for distributed systems, and in particular, designs and implements a set of general mutation operators with distributed characteristics for evaluating the robustness, coverage, and reliability of the unit test set of distributed systems. The present invention is applicable to the quality assurance of various distributed systems, covering various distributed architecture scenarios such as storage systems, stream processing platforms, and coordination services. Background Art

[0002] In today's digital age, distributed systems have become a key component of modern society and software development, and are widely used in fields such as large-scale e-commerce platforms, cloud computing services, and infrastructure construction. By distributing data and computing tasks to multiple locations in the network, distributed systems significantly improve processing efficiency, system stability, and scalability. However, with the continuous progress of technology and the growth of application requirements, the quality assurance of distributed systems has gradually become a hot issue in software engineering.

[0003] During the long-term development of distributed systems, the continuous changes in code have led to the need to adjust test cases, which is particularly obvious in open-source projects. These changes may bring potential test insecurity, especially in distributed systems, where the complexity and interdependence of components significantly increase the likelihood of hidden defects. Although traditional testing methods (such as unit testing) are widely used to maintain the security and stability of distributed systems, their coverage and effectiveness often fail to meet the requirements of complex systems.

[0004] As an efficient software testing technology, mutation testing detects defects in the test set by introducing small changes in specific parts of the program, providing a more powerful testing method than traditional testing. However, the current mutation testing tools on the market are mainly designed for single-machine systems and do not fully consider the unique challenges in distributed systems. For example:

[0005] (1) Existing mutation operators lack effective simulation of network anomalies, thread concurrency, and file consistency issues.

[0006] (2) The mutation operators designed for distributed architectures are not yet systematic. Most mutation operators still involve simple modifications to specific operations or operators, and these mutation operators do not have distributed characteristics.

[0007] (3) Considering the code scale of distributed systems, applying traditional mutation operators in distributed systems will generate a large number of mutants, which means a large amount of time cost and computing cost.

[0008] This results in low efficiency of mutation testing when using existing mutation testing tools and existing mutation operators in the scenario of compiling and testing for distributed systems. It is difficult to discover hidden defects in the code while wasting computing resources. Summary of the Invention

[0009] Objective of the Invention: Aiming at the problems and deficiencies in the prior art, the present invention provides an implementation method for a mutation testing framework (abbreviated as DisMuTe) for distributed systems. By designing general mutation operators for the characteristics of distributed systems, the efficiency and effectiveness of mutation testing for distributed systems are comprehensively improved.

[0010] Technical Solution: An implementation method for a mutation testing framework (abbreviated as DisMuTe) for distributed systems. After testers select the system to be tested, they select the mutation operators required for mutation testing from the mutation operator library. Then, based on the selected mutation operators, the mutation testing framework generates mutants on the basis of the source code of the system to be tested, and filters out some invalid mutants or equivalent mutants through a combination of static analysis and dynamic execution. Then, it enters the mutant execution stage. The mutation testing framework will load a certain mutant in sequence according to a certain priority based on the coverage information of the unit tests of the system to be tested, run the test suite of the system to be tested on the mutated system to be tested and collect the results. Finally, after all mutants are loaded and run, the mutation testing framework will collect and organize the data to form a readable test report, thereby reflecting the quality of the test suite of the system to be tested to a certain extent. Specifically, the method includes the following steps:

[0011] 1) Mutation Operator Library and Operator Selection. According to the three characteristics of network interaction, high concurrency, and consistency guarantee within the distributed system cluster, based on the literature review and error report analysis of common failure modes and vulnerabilities in distributed systems, three types of mutation operators, namely network, concurrency, and consistency, are designed and implemented.

[0012] Each mutation operator is implemented based on JavaParser, which identifies specific patterns in the source code of the system to be tested and modifies them according to the constraints of the operator. In addition, by adopting a modular design concept and design patterns such as static factory and template method, users can simply and efficiently implement their own customized mutation operators according to their needs, horizontally expanding the capabilities of the testing framework.

[0013] When a user uses this mutation testing framework to perform mutation testing on a target system, the following need to be defined: (1) a list of mutation operators for generating mutants; (2) the path of the system under test; (3) the construction tool for the system under test to support system compilation and test suite execution. Currently, maven, ant, and gradle are supported; (4) the source code wildcards of the system under test to identify the list of source code files to be scanned, and mutants will only be generated in these source code files; (5) the construction output path of the system under test to identify and delete equivalent mutants at the bytecode level; (6) the result output path: for outputting test results; (7) whether to use docker (optional). For test suites that do not support parallel testing, docker is used for parallel execution to fully utilize computing resources. (8) Whether to enable code coverage analysis (optional). If code coverage analysis is enabled, in the subsequent test suite running phase, the framework will only execute test cases that cover the mutated positions, rather than executing all test cases in full volume.

[0014] 2) Mutant generation. After the user defines the path of the system under test, the mutation testing framework will scan and construct a list of source code files of the system under test according to the specified source code wildcards by the user, and generate mutants based on the specified list of mutation operators on the basis of these source code files of the system under test. This mutation testing framework only mutates one piece of code at a time, that is, only generates first-order mutants.

[0015] During the mutant generation process, this framework will filter out some invalid or equivalent mutants through static analysis methods. This mutation testing framework will filter out similar equivalent or invalid mutants through static analysis means to reduce the total number of mutants, save the running overhead of the test suite, and increase the proportion of high-quality mutants.

[0016] In addition to static analysis means, this framework will also eliminate mutants with syntax or semantic errors and equivalent mutants through actual compilation. Specifically, after all mutants are generated, this framework will load each mutant and compile the system under test after loading the mutant. If the compilation fails, it means that the mutant has syntax or semantic errors and is an invalid mutant. If the compilation is successful, the framework will collect the mutation results of the mutants and compare the compilation results of all mutants pairwise. If the compilation results of two mutants are completely equivalent, it means that these two mutants are still equivalent mutants after compilation optimization. For a group of equivalent mutants, this framework will only retain a randomly selected mutant in the group to save the test running overhead in the subsequent steps.

[0017] 3) Mutant execution. In this step, the test suite of the system under test is executed for all mutants generated in step 2), and the test results are recorded. Specifically, for each mutant, the test execution engine runs the test suite according to the set frequency and relevance.

[0018] 4) Result statistics and analysis. For the output results after the test suite is executed for each mutant, the framework determines whether the mutant survives based on the analysis results of the set keywords and the success or failure of the test cases, and collects statistical information such as running time and covered test cases. Finally, the test results of all mutants are summarized, the test results are evaluated, metrics such as mutation coverage rate and mutation kill rate are calculated, a readable test report is generated, and the deficiencies of the test set are identified.

[0019] Advantageous effects: Compared with the prior art, the present invention has the following advantages:

[0020] (1) Significantly improved test coverage: The mutant coverage rate generated by DisMuTe reaches 2.2 times that of existing tools.

[0021] (2) Higher-quality mutants: The generated mutants are difficult to be killed by test cases ("difficult to kill"), thus effectively evaluating the deficiencies of the test set.

[0022] (3) Strong tool versatility: The proposed mutation operators are applicable to a variety of distributed systems, filling the gap in the design of general mutation operators in the field.

[0023] (4) Improved test efficiency: Through the optimized mutant screening mechanism, the generation of invalid mutants is reduced, and the test calculation overhead and test running time overhead are lowered. Description of the Drawings

[0024] Figure 1 is the overall architecture diagram of the DisMuTe framework according to an embodiment of the present invention;

[0025] Figure 2 is the flowchart of mutant generation and screening according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of operator application cases. (a) shows a mutant of the Learner.java file in the apache-zookeeper-3.5.8 version, and (b) shows the unit test running situation of apache-zookeeper-3.5.8 after loading the above mutant. Detailed Embodiments

[0027] The present invention will be further illustrated below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, those skilled in the art's various equivalent forms of modification of the present invention all fall within the scope defined by the appended claims of this application.

[0028] A mutation testing framework for distributed systems, the core of which is a set of carefully designed mutation operators with distributed characteristics. Aiming at the inherent complexity and high concurrency of distributed systems, it can generate high-quality mutants, and by running the test suite of the system under test on these mutants, evaluate the coverage and effectiveness of the test set of the distributed system. The mutation testing framework includes a mutation operator library, a mutant generator, a test runner, and a result analysis module, and covers a large number of typical fault scenarios in distributed systems by implementing operators for the three major fields of network, concurrency, and consistency. Experimental results show that compared with existing tools (such as PIT), the present invention can generate higher-quality mutants, improve test coverage, and significantly enhance the challenge and effectiveness of the test set. The present invention is applicable to various quality assurance scenarios of distributed systems and has good generality and scalability.

[0029] (1) Design of the mutation operator library: Starting from the three characteristics of distributed systems depending on the network, high concurrency, and the need to maintain consistency among nodes in the cluster, 13 mutation operators in the three major fields of network, concurrency, and consistency are designed and implemented. The list of mutation operators is shown in the following table.

[0030]

[0031]

[0032] The network mutation operators simulate common network problems in distributed systems, including timeouts, resource leaks, and insufficient exception handling. For example: modifying network timeout parameters (MNT), removing network resource shutdown operations (RRC), etc.

[0033] The concurrency mutation operators target thread synchronization problems and try to trigger race conditions or deadlock scenarios by modifying the thread synchronization situation in the code. For example: modifying thread waiting time (MCT), removing synchronization locks or synchronization statements (RTS), etc.

[0034] The consistency mutation operators evaluate the effectiveness of the test suite in verifying and handling file-related consistency problems by deleting or modifying checks and exception handling in critical file operations. For example: removing file existence checks (RCF), modifying or removing exception handling related to file operations (RFE).

[0035] (2) Mutant Generation and Screening: Based on the mutant operator library, by combining static analysis and dynamic execution, high-quality mutants are screened out to avoid generating redundant or invalid mutants (such as equivalent mutants or non-compilable mutants). Ensure that only those mutants that can be successfully compiled and executed are retained for the subsequent test phase. By carefully filtering out equivalent and non-compilable mutants, the mutant testing process is ensured to be efficient, which helps to reduce the consumption of time resources and computing resources during operation.

[0036] (3) Framework Implementation and Tool Support: The running engine is implemented, which is responsible for generating mutants according to operators, loading mutants, running test suites, and evaluating the results of test suites. This process starts with systematically replacing the original code with each mutant generated in the mutation phase. For each execution of the test suite, only one mutant is used, and they are tested one by one to avoid conflicts or masking between mutants. If the system supports it, tools such as Docker can be used for parallel testing in this phase, but parallel testing is not run in the same environment to avoid false positives or unstable tests caused by port conflicts. The results of the completed runs are uniformly collected and sorted by the result analysis module to generate statistical data and test reports.

[0037] Detailed Description of the Technical Solution

[0038] After the tester selects the system to be tested, the mutant operators required for mutant testing are selected from the mutant operator library; then, based on the selected mutant operators, the mutant testing framework generates mutants on the source code of the system to be tested and filters out some invalid mutants or equivalent mutants through a combination of static analysis and dynamic execution; then enters the mutant execution phase, and the mutant testing framework will load a certain mutant in sequence according to the coverage information of the unit tests of the system to be tested, run the test suite of the system to be tested on the mutated system to be tested and collect the results. Finally, after all mutants have been loaded and run, the mutant testing framework will collect and sort out the data to form a readable test report, thus reflecting to a certain extent the quality of the test suite of the system to be tested. Specifically, the method includes the following steps.

[0039] 1) Mutant Operator Library and Operator Selection. Based on the three characteristics of network interaction, high concurrency, and consistency guarantee within the distributed system cluster, we designed and implemented 13 mutant operators in three categories: network, concurrency, and consistency. Before designing these operators, an in-depth literature review and error report analysis of common failure modes and vulnerabilities in distributed systems were conducted, which ensured the comprehensiveness and effectiveness of the operator library and enabled it to identify and address potential failures in distributed systems. By focusing on these areas, the mutant operators introduced realistic and powerful changes, simulating common failure scenarios.

[0040] Each mutation operator is implemented based on JavaParser, which can comprehensively and efficiently identify specific patterns in the source code of the system under test and modify them according to the constraints of the operator. In addition, we adopt a modular design concept and design patterns such as static factories and template methods, enabling users to simply and efficiently implement their customized mutation operators according to their needs and horizontally expand the capabilities of the testing framework.

[0041] When using this mutation testing framework to perform mutation testing on the target system, users need to define: (1) a list of mutation operators for generating mutants; (2) the path of the system under test; (3) the construction tool for the system under test, which is used to support system compilation and test suite execution. Currently, maven, ant, and gradle are supported; (4) the source code wildcards of the system under test, which are used to identify the list of source code files to be scanned, and mutants will only be generated in these source code files; (5) the output path of the system under test construction, which is used to identify and delete equivalent mutants at the bytecode level; (6) the result output path: used to output test results; (7) whether to use docker (optional). For test suites that do not support parallel testing, docker is used for parallel execution to make full use of computing resources. (8) Whether to enable coverage analysis (optional). If coverage analysis is enabled, in the subsequent test suite running process, the framework will only execute test cases that cover the mutated positions, rather than all test cases in full.

[0042] 2) Mutant generation. After the user defines the source code scanning path of the system under test, the mutation testing framework will scan all the source code files specified by the user according to the file wildcards and generate mutants based on the list of mutation operators specified by the user on the basis of these source code files of the system under test. This mutation testing framework only mutates one piece of code at a time, that is, only first-order mutants are generated.

[0043] During the mutant generation process, this framework will filter out some invalid or equivalent mutants through certain static analysis methods. For example, the MNT mutation operator will shorten the network timeout time in a specific call to 1 / 10 of the original, but in line 1188 of the Zookeeper source code file QuorumCnxManager.java, sock.setsoTimeout(0) will be changed to sock.setsoTimeout(0 / 10). After mutation, the system under test is syntactically equivalent to the original system under test, so this mutant is an equivalent mutant. This mutation testing framework will filter out similar equivalent or invalid mutants through certain static analysis means to reduce the total number of mutants, save the running overhead of the test suite, and increase the proportion of high-quality mutants.

[0044] In addition to static analysis, this framework will also eliminate mutants with syntax or semantic errors and eliminate equivalent mutants by actual compilation. Specifically, after all mutants are generated, this framework will load each mutant and compile the system under test after loading the mutant. If the compilation fails, it means that the mutant has syntax or semantic errors and is an invalid mutant. If the compilation is successful, the framework will collect the mutation results of the mutant and compare the compilation results of all mutants pairwise. If the compilation results of two mutants are exactly equivalent, it means that these two mutants are still equivalent mutants after compilation optimization. For a group of equivalent mutants, this framework will only retain a randomly selected mutant in the group to save test running overhead in subsequent steps.

[0045] 3) Mutant execution. This step will sequentially execute the test suite of the system under test for all mutants generated in step 2) and record the test results. Specifically, for each mutant, the test execution engine will run the test suite according to a certain frequency and relevance. For example, if the code coverage analysis function is enabled, for each mutant, the framework will only execute the test cases that cover the mutant location, rather than executing all test cases in full. In particular, we have implemented a mechanism to terminate tests that greatly exceed the normal execution time. Tests that take much longer than expected will be terminated to prevent unnecessary time waste and ensure that the test process remains efficient. This approach helps to balance comprehensive testing and actual time management. In addition, if the test suite of the system under test does not support concurrent testing, this framework provides a docker mode to maximize the use of computing resources and save mutation testing time by running multiple mutants in multiple docker containers simultaneously.

[0046] 4) Result statistics and analysis. For the output results after each mutant executes the test suite, the framework will judge whether the mutant survives based on the analysis results of specific keywords and whether the test cases succeed or fail, and collect statistical information such as running time and covered test cases. Finally, summarize the test results of all mutants, evaluate the test results, calculate metrics such as mutation coverage and mutation kill rate, generate a readable test report, and identify the deficiencies of the test set.

[0047] Furthermore, the specific steps of step 1) are as follows:

[0048] Step 1)-1: Starting state;

[0049] Step 1)-2: Input the list of mutation operators;

[0050] Step 1)-3: Input the path of the system under test;

[0051] Step 1)-4: Input the construction tool of the system under test;

[0052] Step 1)-5: Input the source code path wildcard of the system under test;

[0053] Step 1)-6: Input the build output path of the system under test;

[0054] Step 1)-7: Input the output path of the mutation test result;

[0055] Step 1)-8: Input whether to use docker;

[0056] Step 1)-9: Input whether to enable the code coverage analysis function;

[0057] Step 1)-10: End state;

[0058] Furthermore, the specific steps of the above step 2) are as follows:

[0059] Step 2)-1: Starting state;

[0060] Step 2)-2: Scan the source code path of the system under test and list the list of source code files to be mutated;

[0061] Step 2)-3: Mutate each source code file in the list sequentially using the mutation operator in step 1);

[0062] Step 2)-4: Perform static analysis on the mutated files;

[0063] Step 2)-5: Load the mutated files into the system under test and compile the mutants;

[0064] Step 2)-6: Collect the compilation results;

[0065] Step 2)-7: Compare the mutation results of all mutants;

[0066] Step 2)-8: Eliminate invalid mutants and equivalent mutants;

[0067] Step 2)-9: Update the mutant list;

[0068] Step 2)-10: End state;

[0069] Furthermore, the specific steps of the above step 3) are as follows:

[0070] Step 3)-1: Starting state;

[0071] Step 3)-2: If the code coverage analysis function is enabled, first read the code coverage file passed in by the user and perform unit test selection according to the unit test coverage of mutants;

[0072] Step 3)-3: If the docker mode is enabled, start the container according to the configuration to complete the task distribution;

[0073] Step 3)-4: Load the mutants and run the test suite;

[0074] Step 3)-5: Collect the test reports;

[0075] Step 3)-6: Load the mutants until the test suite has been run on all mutants;

[0076] Step 3)-7: End state;

[0077] Furthermore, the specific steps of the above step 4) are as follows:

[0078] Step 4)-1: Starting state;

[0079] Step 4)-2: Perform keyword analysis on the test reports of each mutant;

[0080] Step 4)-3: Collect and summarize the statistical results to form a mutation test report;

[0081] Step 4)-4: End state;

[0082] Taking ZooKeeper as an example, it shows how to generate a valid mutant by modifying the network timeout (MNT) and verify the effect of the test set.

[0083] Among them, Figure 3 (a) shows a mutant in the Learner.java file of the apache-zookeeper-3.5.8 version. This mutant modifies the call of the network-related API sock.connect(addr,timeout) on line 234 of the file. Among them, the mutant (right) changes the timeout of the function parameter to 1 / 10 of the original.

[0084] Appendix Figure 3 (b) shows the unit test running situation of apache-zookeeper-3.5.8 after loading the above mutant. After loading the mutant, all unit tests still pass, indicating that the mutant generated by using the distributed mutation operator library has not been killed, which to a certain extent verifies the effectiveness of the distributed mutation operator library of the present invention.

Claims

1. An implementation method for a mutation testing framework for distributed systems, characterized in that, After selecting the system under test, the tester selects the mutation operators required for mutation testing from the mutation operator library. Then, based on the selected mutation operators, the mutation testing framework generates mutants on the source code of the system under test and filters out some invalid or equivalent mutants through a combination of static analysis and dynamic execution. After that, it enters the mutant execution phase. The mutation testing framework will load a certain mutant in sequence according to the set priority based on the code coverage information of the unit tests of the system under test, run the test suite of the system under test on the mutated system under test and collect the results. Finally, after all mutants have been loaded and run, the mutation testing framework will collect and organize the data to form a readable test report.

2. The implementation method of the mutation testing framework for a distributed system according to claim 1, wherein Based on the three characteristics of network interaction, high concurrency, and consistency guarantee within the distributed system cluster, and based on the literature review and error report analysis of fault modes and vulnerabilities in the distributed system, three types of mutation operators for network, concurrency, and consistency are designed and implemented. Each mutation operator is implemented based on JavaParser, which identifies patterns in the source code of the system under test and modifies them according to the constraints of the operator. The modular design concept and the design patterns of static factory and template method are adopted, enabling users to implement their own customized mutation operators according to their needs.

3. The implementation method of the mutation testing framework for a distributed system according to claim 1, wherein When using this mutation testing framework to perform mutation testing on the target system, the user needs to define: (1) a list of mutation operators for generating mutants; (2) the path of the system under test; (3) the construction tool for the system under test to support system compilation and test suite execution; (4) the source code wildcards of the system under test to identify the list of source code files to be scanned, and mutants will only be generated in these source code files; (5) the output path for building the system under test to identify and delete equivalent mutants at the bytecode level; (6) the result output path: for outputting test results; (7) whether to use Docker. For test suites that do not support parallel testing, Docker is used for parallel execution to make full use of computing resources; (8) whether to enable coverage analysis. If coverage analysis is enabled, in the subsequent test suite running session, the framework will only execute the test cases covering the mutated positions according to the unit test coverage file passed in by the user, rather than executing all test cases in full.

4. The implementation method of the mutation testing framework for a distributed system according to claim 1, characterized in that During mutant generation, after the user defines the source code scanning path of the system under test, the mutation testing framework scans all the source code files specified by the user according to the file wildcards and generates mutants based on the list of mutation operators specified by the user on the basis of these source code files of the system under test. This mutation testing framework only mutates one piece of code at a time, that is, only generates first-order mutants.

5. The implementation method of the mutation testing framework for a distributed system according to claim 1 or 4, characterized in that, During the mutant generation process, this framework will filter out some invalid or equivalent mutants through static analysis methods.

6. The implementation method of the mutation testing framework for a distributed system according to claim 1 or 4, characterized in that, Eliminate mutants with syntax or semantic errors and eliminate equivalent mutants through actual compilation; specifically, after all mutants are generated, the mutation testing framework will load each mutant and compile the system under test after loading the mutant; if the compilation fails, it means that the mutant has syntax or semantic errors and is an invalid mutant; if the compilation is successful, the framework will collect the mutation results of the mutants and compare the compilation results of all mutants pairwise; if the compilation results of two mutants are exactly equivalent, it means that these two mutants are still equivalent mutants after compilation optimization. For a group of equivalent mutants, the mutation testing framework will only retain a randomly selected mutant in the group.

7. The implementation method of the mutation testing framework for a distributed system according to claim 1, wherein During mutant execution, execute the test suite of the system under test for all generated mutants in sequence and record the test results; specifically, for each mutant, the test execution engine will run the test suite according to the set frequency and relevance.

8. The implementation method of the mutation testing framework for a distributed system according to claim 7, characterized in that, If the code coverage analysis function is enabled, for each mutant, the mutation testing framework will only execute the test cases that cover the mutated positions, rather than executing all test cases in full; tests that exceed the expected set value in running time will be terminated; if the test suite of the system under test does not support concurrent testing, the mutation testing framework provides a docker mode to run multiple mutants simultaneously in multiple docker containers.

9. The implementation method of the mutation testing framework for a distributed system according to claim 1, wherein For the output result after each mutant executes the test suite, the mutation testing framework judges whether the mutant survives based on the analysis result of the set keywords and whether the test case is successful or failed, and collects the statistical information of the running time and the covered test cases; summarize the test results of all mutants, evaluate the test results, calculate the mutation coverage rate and mutation kill rate indicators, and generate a readable test report.

Images