Variation operator evaluation method and system for variation test

Abstract

The invention provides a mutation operator evaluation method and system for a mutation test, and the method comprises the following steps: selecting a corresponding mutation operator subset according to the language type of a tested system, injecting mutation operators to generate variants, and designing a test case set for killing all variants in a variant set; executing an original program and a variant program according to the test case set, obtaining execution conditions and type information of variants, and placing the variants into a stubborn variant set, a crash variant set and an equivalent variant set respectively; calculating evaluation index values according to the proportions of stubborn variants, collapse variants and equivalent variants in the three types of sets of different mutation operators, and designing mutation operator evaluation rules based on different types of variants; and evaluating each mutation operator in the mutation operator set according to an evaluation rule and the index values of different mutation operators so as to select a reasonable mutation operator to implement mutation. By using the mutation operator selected by the method, the number of generated variants can be reduced while the effectiveness of the mutation test is maintained, so that the cost of the mutation test is reduced, and the test efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems related to specific computing models, and more specifically, to a mutation operator evaluation method and system for mutation testing, which can be used in the field of mutation testing. By evaluating the effectiveness of mutation operators, reasonable mutation operators are selected to generate higher quality mutants, thereby reducing the number of generated mutants without reducing the effectiveness of the test, thereby improving the efficiency of mutation testing. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Mutation testing is a testing technique for discovering software errors. It measures the quality of a test case set by using killed mutants. The key to mutation testing research is to use targeted theories and methods to generate a set of mutants that can simulate real faults, and then generate test cases that meet certain coverage criteria based on the set or perform other testing work.

[0004] The inventors found that the existing mutation testing methods have various problems: ① When using the random method to select a subset of mutants, the mutation score decreases to a certain extent; ② When randomly selecting a subset of mutation operators, it is necessary to repeatedly select and screen to find an excellent subset of mutation operators, and the process is relatively complicated; ③ After the mutation, a certain runtime crash mutant generated can usually be detected by any test case. ④ When evaluating the mutation operator, usually only the effect of a certain mutant is considered, without considering the joint effects of stubborn mutants, equivalent mutants and crash mutants at the same time. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a mutation operator evaluation method and system for mutation testing, which evaluates the quality of different mutants and evaluates the mutation operator. According to the evaluation results, using a reasonable mutation operator can generate more valid mutants, while maintaining the effectiveness of the test and reducing the number of invalid mutants generated, thereby improving the efficiency of mutation testing.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] One or more embodiments provide a mutation operator evaluation method for mutation testing, including the following process:

[0008] According to the defined mutation operator set, all mutation operators are injected into the system under test to generate a mutant set;

[0009] Generate a test case set that can kill all mutants based on the execution status of the mutants, and generate a new test case set by adding or using evolutionary methods, search-based methods, etc. on the existing test case set;

[0010] Execute the mutant set based on the test case set, and add the mutants to the stubborn mutant set, the crash mutant set, and the equivalent mutant set respectively by analyzing the execution and output of the mutant program and the original program;

[0011] According to the mutation operator evaluation method, the mutation operator is evaluated in combination with the evaluation index value of the mutation operator.

[0012] One or more embodiments provide a mutation operator evaluation system for mutation testing, including a collection device, a communication network, and a test terminal, wherein the collection device and the test terminal are connected via the communication network;

[0013] A collection device: configured to obtain specific mutation rules of the mutation operator to be evaluated and obtain information on the type of the analyzed mutant;

[0014] Test terminal: configured to execute the steps of the above-mentioned mutation operator evaluation method for mutation testing.

[0015] Compared with the prior art, the invention has the following beneficial effects.

[0016] The present invention evaluates mutation operators by analyzing the probability of mutation operators generating different mutants, and only uses mutation operators with higher effectiveness in the results. This can not only generate as many effective mutants as possible, but also reduce the number of invalid mutants generated from the perspective of mutant generation. While ensuring the effectiveness of the test, the cost of mutation testing is reduced, and the efficiency of mutation testing is greatly improved.

[0017] The advantages of the present invention and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute limitations of the present invention.

[0019] Figure 1 This is a diagram of a method for evaluating a mutation operator according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0022] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments of the present invention and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0023] Technical term explanation:

[0024] Mutation operator: The rules based on which program statements are mutated are called mutation operators.

[0025] Mutant: The new program obtained by injecting the mutation operator into the original program is called a mutant.

[0026] Test case suite: A collection of test cases used to execute the system under test.

[0027] Stubborn mutants: mutants that survive and whose intermediate states of the program remain unchanged after executing them with all test cases in the test case set.

[0028] Equivalent mutant: After being executed using the test case set, the mutant is not killed and the semantics are equivalent to the original program.

[0029] Crash variant: A variant in which a runtime error such as crash or deadlock occurs during the execution of any test case.

[0030] Example 1

[0031] In the technical solutions disclosed in one or more embodiments, Figure 1 As shown, a mutation operator evaluation method for mutation testing includes the following steps:

[0032] Step 1: Select the mutation implementation object according to the mutation operator set, inject all mutation operators into the system set under test, generate a set of mutants, and generate a test case set;

[0033] Step 2: According to the different results of executing the variant and the original program in the test case set, the variant is added to the stubborn variant set, the crash variant set and the equivalent variant set;

[0034] Step 3: According to the quality of different mutants in the set, design the evaluation rules of the mutation operator and calculate the index value to evaluate the mutation operator.

[0035] This embodiment evaluates the importance of mutation operators based on the probability of different mutation operators generating stubborn mutants, crash mutants and equivalent mutants, and selects mutation operators with higher importance to perform mutation, which can generate a higher quality mutant set and improve the efficiency of mutation testing.

[0036] In step 1, select a suitable mutation operator set to inject, generate a mutant set, and generate a test case set that can kill all mutants, including the following steps:

[0037] Step 1.1 Variant generation:

[0038] Mutation testing simulates potential errors by introducing small changes to the program source code and then checking whether the existing test suite can detect these changes. Mutation operators are rules or patterns used to generate mutants, and their design should be able to cover various situations where errors may occur in the program in order to comprehensively evaluate the effectiveness of the test suite.

[0039] Select a mutation operator set. Specifically, select the corresponding mutation operator set according to the language type of the system under test. If there is a lack of suitable mutation operators, it is necessary to design mutation operators based on the characteristics of the program and the common errors of this type of program. Inject appropriate mutation operators according to the type of mutation statement to generate a mutant set. Inject the selected mutation operator manually or use a suitable automated tool.

[0040] Step 1.2 Test case set generation:

[0041] Generate a test case set that can kill all mutants. If there is a historical test case set, you can use it to add a new test case set to meet the requirements; if there is no existing test case set, use a specific method to generate a test case set, such as random method, search-based method, and evolutionary method.

[0042] In step 2, according to the different results of executing the variant and the original program in the test case set, the variant is added to the stubborn variant set, the crash variant set and the equivalent variant set, including the following steps:

[0043] Step 2.1 Variant execution:

[0044] Use the test case set generated in step 1 to execute the original program and each variant respectively, and record the program execution information, including whether errors occur during runtime, the output of the original program, the output of the variant program, the intermediate state of the original program, and the intermediate state of the variant program, for the next step of analysis.

[0045] Step 2.2 Variant classification:

[0046] Based on the acquired execution information and certain manual analysis, the mutants are classified into stubborn mutants, crash mutants, and equivalent mutants. Mutants that have errors during execution are called crash mutants. The output results of stubborn mutants and equivalent mutants are the same as the original program, and further manual analysis is required. Mutants with the same output and changes in the intermediate state of the program, and such changes can be propagated to the output are stubborn mutants; variants with the same semantics as the original program are called equivalent mutants.

[0047] Based on the above classification method, the generated variants are added to the corresponding stubborn variant set, crash variant set and equivalent variant set respectively.

[0048] In step 3, according to the quality of different mutants in the set, the evaluation rules of the mutation operator are designed, and the index value is calculated to evaluate the mutation operator, including the following steps:

[0049] Step 3.1 Evaluation rule definition:

[0050] Different types of mutants have different qualities. Stubborn mutants can usually find potential defects and have the highest quality; crash mutants can simulate runtime errors, but are easily detected and have lower quality; equivalent mutants have the same semantics as the original program and cannot be detected, which is one of the important factors affecting the efficiency of mutation testing. Therefore, in this embodiment, the mutation operator evaluation rules defined are specifically:

[0051] (1) The higher the probability of generating high-quality mutants, the higher the effectiveness of the mutation operator.

[0052] (2) The higher the probability of generating low-quality mutants, the lower the effectiveness of the mutation operator.

[0053] Among them, stubborn variants are high-quality variants, crash variants and equivalent variants are both low-quality variants, and the quality of crash variants is higher than that of equivalent variants.

[0054] Step 3.2 Evaluation index design:

[0055] Design mutation operator evaluation indicators based on the evaluation method. The specific indicators are:

[0056] (1) Mutation operator stubbornness probability. The ratio of the number of stubborn mutants generated by a mutation operator to the total number of mutants generated by it.

[0057] (2) Mutation operator crash probability. The ratio of the number of crashed mutants generated by a mutation operator to the total number of mutants generated by it.

[0058] (3) Mutation operator equivalence probability. The ratio of the number of equivalent mutants generated by a mutation operator to the number of all mutants generated by it.

[0059] Step 3.3 Mutation operator evaluation:

[0060] The number of stubborn mutants, crashed mutants, and equivalent mutants in the three types of mutant sets of each mutation operator is counted, and the evaluation index values, namely, the stubborn probability, crash probability, and equivalent probability of the mutation operator, are calculated. According to the defined evaluation rules, the index values ​​of different mutation operators are compared to evaluate the mutation operators. According to the evaluation results, the mutation operator subset with the highest importance is the mutation operator subset used for subsequent mutation.

[0061] Example 2

[0062] Based on Example 1, this embodiment provides a mutation operator evaluation system for mutation testing, including a collection device, a communication network and a test terminal, and the collection device and the test terminal are connected through the communication network:

[0063] Collection device: configured to obtain the specific mutation rules of the mutation operator to be evaluated and obtain information on the type of analyzed variants:

[0064] Test terminal: configured to be used for the steps of a mutation operator evaluation method for mutation testing described in Example 1.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0066] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A mutation operator evaluation method for mutation testing, characterized in that: The process includes the following: Select the corresponding mutation operator set according to the language type of the system under test, inject the mutation operator to generate mutants, and design a test case set to kill all mutants in the mutant set; Execute the original program and the variant program according to the test case set, obtain the execution status and type information of the variant, and place the variant into the stubborn variant set, the crash variant set, and the equivalent variant set respectively; The evaluation index value is calculated based on the proportion of stubborn mutants, crash mutants and equivalent mutants in the three types of sets of different mutation operators, and the mutation operator evaluation rules based on different types of mutants are designed; Each mutation operator in the mutation operator set is evaluated according to the evaluation rules and the index values ​​of different mutation operators to select a reasonable mutation operator to implement mutation.

2. A mutation operator evaluation method for mutation testing as claimed in claim 1, wherein a corresponding mutation operator set is selected according to the language type of the system under test, mutation operators are injected to generate mutants, and a test case set is designed to kill all mutants in the mutant set, specifically referring to: By analyzing program characteristics, such as traditional serial programs, distributed storage parallel programs, shared storage parallel programs, etc., we select mutation operators that can better simulate program failures according to the program type, inject mutation operators to generate mutants, and use certain test generation strategies to generate a set of test cases that can kill all mutants.

3. A mutation operator evaluation method for mutation testing as claimed in claim 1, characterized in that: Execute the original program and the variant program according to the test case set, obtain the variant's execution status and type information, and place the variant into the stubborn variant set, the crash variant set, and the equivalent variant set, respectively. Specifically, it means: Use the test case set to execute the original program and each variant program to obtain execution information, including whether errors occur during runtime, the output of the original program, the output of the variant, the intermediate state of the original program, and the intermediate state of the variant program. Use the obtained information to classify the mutants into stubborn mutants, crash mutants, and equivalent mutants. Set three types of mutant sets for each mutation operator, and place the different mutants obtained into the corresponding sets.

4. A mutation operator evaluation method for mutation testing as claimed in claim 1, characterized in that: The evaluation index value is calculated according to the proportion of stubborn mutants, crash mutants and equivalent mutants in the three types of sets of different mutation operators, and the mutation operator evaluation rules based on different types of mutants are designed, specifically: According to the number of three types of mutants in the set of stubborn mutants, crashed mutants and equivalent mutants, the stubborn probability, crash probability and equivalent probability of the mutation operator are calculated respectively. The specific evaluation rules are as follows: the mutation operator with the highest effectiveness is the mutation operator that can generate stubborn mutants; the mutation operator whose crash probability is more than twice the equivalent probability is second in effectiveness; most of the mutants generated by other mutation operators are equivalent mutants, which have the lowest effectiveness.

5. A mutation operator evaluation method for mutation testing as claimed in claim 1, characterized in that: According to the evaluation rules and the index values ​​of different mutation operators, each mutation operator in the mutation operator set is evaluated to select a reasonable mutation operator to implement mutation, specifically: According to the defined evaluation rules, the stubbornness probability, crash probability and equivalent probability of different mutation operators are compared. The mutation operator that generates more stubborn mutants is more important, followed by crash mutants, and the mutation operator that generates a large number of equivalent mutants is the least important. Based on this, the mutation operator subset with the highest effectiveness is selected to perform mutation in order to generate high-quality mutants.

6. A mutation operator evaluation system for mutation testing, characterized by: It includes a collection device, a communication network and a test terminal, and the collection device and the test terminal are connected via the communication network; A collection device: configured to obtain specific mutation rules of the mutation operator to be evaluated and obtain information on the type of the analyzed mutant; Test terminal: configured to execute the steps of a mutation operator evaluation method for mutation testing as described in any one of claims 1-5.

Images