An adaptive random software testing method based on region division

Abstract

The present invention relates to an adaptive random testing method based on partitioning and adaptive components, comprising the following steps: selecting a public data set; constructing an adaptive random testing model M; selecting a tested program s as required, and forming a program set D, and determining the input domain partition ratio parameter α of M; judging whether the program has defects by selecting the result of inputting a test case into the program; wherein the test case selection is selected by using a calculation method matching the two constructed functions as required. The method of the present invention can quickly and effectively improve the efficiency and accuracy of FSCS‑ART in finding defects.

Description

Technical Field

[0001] The invention relates to the technical field of test automation, and in particular to an adaptive random software testing method based on region division. Background Art

[0002] Testing is a common activity in the manufacturing process, and its results play a vital role in product quality assurance. Software testing is a process of finding defects in a software system, and its goal is to find defects in the software system at the lowest cost. With the emergence of more and more software systems, the reliability of software systems has received more and more attention, so a large number of software testing methods have been continuously proposed by researchers.

[0003] Since the random testing (hereinafter referred to as "RT") method is very simple, RT has been widely used since it was proposed. The main idea of ​​RT is to randomly find the next test case to be executed in the set of all possible inputs of the program (hereinafter referred to as the "input domain"), and then execute the test case in the software system under test and analyze the program results to find possible defects in the program. RT uses very little information, that is, it only considers the input domain of the program, so the time to generate test cases is very fast, achieving the goal of finding defects in the program at the lowest cost. In reality, for many software system programs, RT has the characteristics of low cost and high efficiency, and it has even become one of the basic ideas of software testing strategies.

[0004] However, on the one hand, the complexity and intelligence of the system put forward higher requirements for testers, and the test efficiency of RT cannot meet the test requirements of complex systems. On the other hand, with the deepening of research, RT has also caused a lot of controversy, such as Myers said that "probably the worst [test case design] method is random input testing.", because researchers found that RT has problems such as blindness, redundancy and inefficiency in finding defects.

[0005] Previous studies have found that inputs that cause errors usually tend to cluster together to form continuous defect areas, and inputs that do not cause errors also cluster together, which seems to mean that the flatter the distribution of generated test cases, the better its defect detection ability. Therefore, adaptive random testing was proposed. Its idea is to hope that if no errors are found in the previous input, the test cases generated in the future will be evenly distributed in the input domain. There are many methods for adaptive random testing, but generally speaking, the most cost-effective is adaptive random testing based on fixed-size candidate sets (hereinafter referred to as "FSCS-ART"). In order to achieve a uniform distribution of test cases in the input domain, FSCS-ART selects candidate test cases with the largest nearest neighbor value as the test cases to be executed next time. Therefore, FSCS-ART has a potential boundary effect problem, that is, it tends to select test cases close to the boundary of the input domain, resulting in fewer test cases close to the center of the input domain. In order to alleviate the boundary effect of FSCS-ART and further improve the effect of FSCS-ART, researchers have proposed many related FSCS-ART algorithms in recent years, but overall, the improvement of the effect of FSCS-ART is very limited. Summary of the invention

[0006] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to improve the efficiency of finding defects in FSCS-ART.

[0007] To solve the above technical problems, the present invention adopts the following technical solution: an adaptive random software testing method based on region division, comprising the following steps:

[0008] S100: selecting a public data set, the public data set including a number of real system programs, each of which has a one-to-one corresponding input domain;

[0009] S200: constructing an adaptive random test model M, wherein the adaptive random test model includes component one, component two and component three;

[0010] The first component is used to divide the input domain of the system program, and the second component is used to calculate the adaptive relative comparison of the test case. The component three is used to calculate the adaptive distance of the test case The test case is a value selected from the input domain;

[0011] S300: Select the program s to be tested, determine the input domain dimension and input domain range of s using the existing technology, select P real system programs with the same input domain dimension as s from the public data set to form a program set D, and use component one to determine the input domain partition ratio parameter α of M;

[0012] S400: define a threshold value of the number of test cases used, select the first candidate test case t from the input domain range of s, and set the number of used test cases T=1;

[0013] S500: t is used as the input of s. When the output result of s is correct, it is considered that no defect of s is found, and then the next step is executed;

[0014] When the output result of s is an error, it is considered that the defect of s is found, and then the number of test cases currently input F-measure is output, and the test ends;

[0015] S510: randomly select k candidate test cases from the input domain range of s to form a candidate test case set C;

[0016] S520: Select the i-th candidate test case c from C i , using component 2 to calculate c i Adaptive relative contrast Then use component three to calculate c i Adaptive distance Where C = {c1,…,c i ,…c k}, i = 1, ..., k;

[0017] when When , execute S530;

[0018] when When , execute S540;

[0019] exter indicates the number of times the function Exter_func is called, and inter indicates the number of times the function Inter_func is called;

[0020] S530: Use the function Exter_func to calculate the nearest neighbor value using Manhattan distance Calculate using formula (1)

[0021]

[0022] in, represents the adaptive distance, E represents the set of all test cases that have been executed but did not find defects in s, represents adaptive relative contrast, i=1,...,k;

[0023] k Arrange in descending order, select the test case corresponding to the maximum adaptive distance value as the best candidate test case, assign the best candidate test case to t, set T=T+1, and execute S600;

[0024] S540: Use the function Inter_func to calculate the nearest neighbor value using the Euclidean distance Calculate using formula (2)

[0025]

[0026] Arrange the k in descending order, select the test case corresponding to the largest adaptive distance value as the best candidate test case, assign the best candidate test case to t, let T = T + 1, and execute S600;

[0027] S600: When T is greater than the test case usage quantity threshold, end the test, otherwise return to S500.

[0028] Preferably, the system program in S100 is a numerical program.

[0029] Preferably, the specific steps for using Component 1 in S300 to determine the input domain partition ratio parameter α of M are as follows:

[0030] S310: Let α = 0.1;

[0031] S320: Divide the input domain corresponding to each real system program in D into an external region and an internal region according to α. The internal region is represented as α d , and the external region is represented as 1 - α d , and α ∈ [0.1, 0.9], where d represents the dimension of the real system program;

[0032] S330: Randomly select a real program W from D, and let W = 1;

[0033] S340: Select the first candidate test case w from the input domain of W;

[0034] S350: Use w as the input of W. When the output result is correct, it is considered that no defect of W is found, and then execute the next step;

[0035] When the output result is incorrect, it is considered that a defect of W is found, and then output the number of test cases F - measure that have been input currently; and let W = W + 1. If W < P, return to S340, otherwise execute S380;

[0036] S360: Randomly select k candidate test cases from the input domain of W to form a candidate test case set B;

[0037] S370: Calculate the adaptive distances of all test cases in B and arrange them in descending order, select the test case corresponding to the maximum adaptive distance value and assign it to w as the next input of W, and return to S350;

[0038] S380: For the current α value, sum and output the F-measures corresponding to all real system programs, and set α=α+0.1. If α<=0.9, return to S320, otherwise execute S390;

[0039] S390: Arrange all F-measures in ascending order, and select α corresponding to the minimum value as the partition ratio parameter of M.

[0040] Preferably, the adaptive relative ratio is calculated in S520 The details are as follows:

[0041] The i-th candidate test case c in C i The dimension of is d;

[0042] Calculate c i Adaptive relative contrast in dimension j The expression is as follows:

[0043]

[0044] Where j represents the jth dimension and j=1,…,d, Represents candidate test case c i The maximum value in the jth dimension, Represents candidate test case c i Minimum value in the jth dimension;

[0045] Calculate c using formula (3) i The adaptive relative contrast of all dimensions, the minimum value of the adaptive relative contrast is selected as c i Adaptive relative contrast

[0046] Compared with the prior art, the present invention has at least the following advantages:

[0047] 1. Compared with the prior art, the present invention greatly alleviates the boundary effect problem of the FSCS-ART algorithm by partitioning and using different algorithms to find defects in different partitions, greatly improves the efficiency of finding defects, makes software testing more efficient, and is simpler to implement.

[0048] 2. First, the area is divided so that test cases can be generated in different areas. Then, the adaptive relative contrast and adaptive distance of each candidate test case are used to achieve the purpose of evenly distributing the test cases in the input domain, further alleviating the boundary effect problem of FSCS-ART, and finally improving the effect of adaptive random testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the overall framework of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in further detail below.

[0051] The present invention describes an adaptive random testing method based on partitioning and adaptive components. The core idea of ​​the invention is to partition the input domain into two regions, and use adaptive relative comparison and adaptive distance to select the most satisfactory test case from the candidate test case set as the next test case, until the test case finds a defect in the program.

[0052] First, find a real numerical program with the same dimension as the system program under test to determine the area division ratio parameter; then calculate the adaptive relative ratio of the candidate test cases; finally, calculate the adaptive distance based on the adaptive ratio and nearest neighbor value of the candidate test cases, and select the next test case to be executed with the maximum adaptive distance.

[0053] The present invention comprises three components. Component 1 determines the area division ratio parameter; Component 2 calculates each candidate test case c i Adaptive relative contrast Used to represent candidate test cases c i Relative position in the input domain; Component 3 calculates each candidate test case c i Adaptive distance Then, the next test case to be executed is selected from the candidate test case set with the maximum adaptive distance; the purpose is to alleviate the boundary effect caused by the maximum nearest neighbor value with the maximum adaptive distance, thereby improving the defect detection capability of FSCS-ART.

[0054] See also Figure 1 , an adaptive random software testing method based on region division, comprising the following steps:

[0055] S100: selecting a public data set, the public data set including a number of real system programs, each of which has a one-to-one corresponding input domain;

[0056] The system program in S100 is a numerical program.

[0057] S200: Construct an adaptive random testing model M, where the adaptive random testing model includes Component One, Component Two, and Component Three;

[0058] Component One is used to divide the input domain of the system program, and Component Two is used to calculate the adaptive relative ratio of test cases Component Three is used to calculate the adaptive distance of test cases Component Three is used to select test cases that can be used for M, and the test cases are values randomly selected from the input domain;

[0059] S300: Select the program under test s, use the existing technology to determine the input domain dimension and input domain range of s, and select P real system programs with the same input domain dimension as s from the public dataset to form the program set D. Use Component One to determine the input domain division ratio parameter α of M;

[0060] The specific steps for using Component One to determine the input domain division ratio parameter α in S300 are as follows:

[0061] S310: Let α = 0.1;

[0062] S320: Divide the input domain corresponding to each real system program in D into an external region and an internal region according to α. The internal region is denoted as α d and the external region is denoted as 1 - α d and α ∈ [0.1, 0.9], the initial value of α is taken as 0.1, and it is incremented by 0.1 each time; where d represents the dimension of the real system program, and the dimension value of the real system program is the same as the number of parameters of the input real system program;

[0063] S330: Randomly select a real program W from D, and let W = 1;

[0064] S340: Select the first candidate test case w from the input domain of W;

[0065] S350: Take w as the input of W. When the output result is correct, it is considered that no defect of W is found, and then the next step is executed;

[0066] When the output result is incorrect, it is considered that a defect of W is found, and then output the number of test cases F - measure that have been input currently; and let W = W + 1. If W < P, return to S340, otherwise execute S380; After all real programs in D have been executed, it ends, that is, when W = |D|, all tests end.

[0067] S360: Randomly select k candidate test cases from the input domain of W to form the candidate test case set B;

[0068] S370: Calculate the adaptive distances of all test cases in B and arrange them in descending order, select the test case corresponding to the maximum adaptive distance value and assign it to w as the next input of W, and return to S350;

[0069] S380: For the current α value, sum and output the F-measures corresponding to all real system programs, and set α=α+0.1. If α<=0.9, return to S320, otherwise execute S390;

[0070] S390: Arrange all F-measures in ascending order, and select α corresponding to the minimum value as the partition ratio parameter of M; since α∈[0.1,0.9], 9 cumulative F-measure values ​​will be obtained.

[0071] S400: define a threshold value of the number of test cases used, select the first candidate test case t from the input domain range of s, and set the number of used test cases T=1;

[0072] S500: t is used as the input of s. When the output result of s is correct, it is considered that no defect of s is found, and then the next step is executed;

[0073] When the output result of s is an error, it is considered that the defect of s is found, and then the number of test cases currently input F-measure is output, and the test ends; the number of test cases input here is 1; at this time, the test cost of this test can be measured according to the value of F-measure;

[0074] S510: randomly select k candidate test cases from the input domain range of s to form a candidate test case set C;

[0075] S520: Select the i-th candidate test case c from C i , using component 2 to calculate c i Adaptive relative contrast Then use component three to calculate c i Adaptive distance Where C = {c1,…,c i ,…c k}, i = 1, ..., k;

[0076] when When , execute S530;

[0077] when When , execute S540;

[0078] exter indicates the number of times the function Exter_func is called, inter indicates the number of times the function Inter_func is called, and the initial values ​​of exter and inter are both 1;

[0079] The adaptive relative contrast is calculated in S520 The details are as follows:

[0080] The i-th candidate test case c in C i The dimension of is d;

[0081] Calculate c i Adaptive relative contrast in dimension j The expression is as follows:

[0082]

[0083] Where j represents the jth dimension and j=1,…,d, Represents candidate test case c i The maximum value in the jth dimension, Represents candidate test case c i Minimum value in the jth dimension;

[0084] Calculate c using formula (3) i The adaptive relative contrast of all dimensions, the minimum value of the adaptive relative contrast is selected as c i Adaptive relative contrast

[0085] S530: Use the function Exter_func to calculate the nearest neighbor value using Manhattan distance Manhattan distance is an existing technology and is calculated using formula (1):

[0086]

[0087] in, represents the adaptive distance, E represents the set of all test cases that have been executed but did not find defects in s, represents adaptive relative contrast, i=1,...,k;

[0088] k Arrange in descending order, select the test case corresponding to the maximum adaptive distance value as the best candidate test case, assign the best candidate test case to t, set T=T+1, and execute S600;

[0089] S540: Use the function Inter_func to calculate the nearest neighbor value using the Euclidean distance The Euclidean distance is an existing technology and is calculated using formula (2):

[0090]

[0091] k Arrange in descending order, select the test case corresponding to the maximum adaptive distance value as the best candidate test case, assign the best candidate test case to t, set T=T+1, and execute S600;

[0092] S600: When T is greater than the test case usage quantity threshold, the test is terminated; otherwise, the process returns to S500.

[0093] Data Experiment

[0094] The present invention uses 12 real C++ numerical programs for experiments. These programs are from ACM programs and Numerical Recipes, which are also the most widely used in ART related research. These real programs are mainly generated by six mutant seeds. Including arithmetic operator replacement (AOR), constant replacement (CR), relational operator replacement (ROR), return statement replacement (RSR), statement deletion (SD) and scalar variable replacement (SVR). We run these test cases in the original program and variant programs. Run the test cases in the original program and variant programs, and collect the results of each test case. By comparing their execution results, we can determine whether the generated test cases detect faults.

[0095] The present invention uses F-ratio (referred to as "F ratio" in Table 1) as an evaluation index, and F-ratio represents F ART With F RT The ratio of where F RT F represents the F-measure of RT. ARTrepresents the F-measure of method ART, and F-measure represents the number of test cases that have been used when the first defect in the program of the system to be tested is found. We use the F-ratio case to measure the degree of improvement of method ART over method RT. The smaller the value of F-ratio, the greater the improvement. The larger the value of F-ratio, the smaller the improvement. At the same time, we use P-value and effect size for statistical analysis. We hope to determine whether the two algorithms are significantly different through Wilcoxon's P-value rank sum test; when the P value is less than 0.05, it means that AP-FSCS-ART is significantly different from the comparison algorithm. We use the non-parametric Vargha and Delaney effect size to determine the possibility that AP-FSCS-ART has an advantage in fault detection ability. The effect size is greater than 0.5, which means that AP-FSCS-ART is likely to be better than the comparison algorithm.

[0096] In the current data set, in order to achieve the best performance of the present invention, the best value of the region partition parameter α is α=0.7, and the value of k is the default k=10. In order to further verify the effectiveness of the present invention, it is compared with the mainstream and recent algorithms, which are as follows:

[0097] FSCS-ART: Fixed-Size Candidate Set Adaptive Random Testing Algorithm is the first ART algorithm proposed and implemented.

[0098] FSCS-ctsr: Candidate test set reduction for adaptive random testing, a technique to reduce overhead.

[0099] NNDC-ART: A Nearest Neighbor Divide-and-Conquer Adaptive Randomized Testing Approach.

[0100] The adaptive random testing method based on region division and adaptive components proposed in the present invention is called AP-ART. The experimental results of the method of the present invention and the comparative method are shown in Table 1. In Table 1, red indicates the best F-ratio value for the same real numerical program (bold indicates the best current running effect, and the improvement degree calculation formula is 1-x / y). It can be seen that the method AP-FSCS-ART of the present invention performs better than FSCS-ART and FSCS-ctsr; in addition, AP-FSCS-ART outperforms NNDC-ART in most cases, except for Golden and Plgndr programs. By accumulating the F-ratio values ​​of all 12 real programs. The results show that the AP-FSCS-ART algorithm can provide 8.8%-11.4% performance improvement.

[0101] Table 1. Comparison of the effects of the present invention and other methods

[0102]

[0103]

[0104] We conducted a static analysis of the experimental results. From the results of the P value, compared with FSCS-ART, the P values ​​of AP-FSCS-ART are all less than 0.05, except for Bessj and Golden; compared with FSCS-ctsr, the P values ​​of AP-FSCS-ART are less than 0.05 in almost all cases, except for Golden; and compared with NNDC-ART, the P values ​​of AP-FSCS-ART are less than 0.05 in most cases, except for Sncndn and Golden programs. It can be seen from these P value results that in most cases, the algorithm proposed by the present invention is significantly different from the comparison algorithm. From the results of effect size, compared with FSCS-ART and FSCS-ctsr, the effect size values ​​of AP-FSCS-ART are all greater than 0.5, which indicates that AP-FSCS-ART is better than FSCS-ART and FSCS-ctsr in fault detection; however, compared with NNDC-ART, the effect size values ​​of AP-FSCS-ART are greater than 0.5 in most cases, and even the effect size of Cel reaches 0.7096; except for Golden and Plgndr. Therefore, it can be concluded that AP-FSCS-ART is better than the comparison algorithm in almost all cases, but may be worse than NNDC-ART on Golden, and worse than NNDC-ART on Golden and Plgndr programs.

[0105] In short, the experimental results prove that the adaptive random testing method based on partitioning and adaptive components proposed in the present invention can effectively improve the defect detection capability of the ART method on these 12 real numerical programs; at the same time, the present invention can be applied to the process of finding defects in other real numerical programs.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. An adaptive random software testing method based on region division, characterized in that: It includes the following steps: S100: Select a public dataset, where the public dataset includes several real system programs, and each real system program has a corresponding input domain; S200: Construct an adaptive random testing model M, and the adaptive random testing model includes Component One, Component Two, and Component Three; The first component is used to divide the input domain of the system program, and the second component is used to calculate the adaptive relative comparison of the test case. The component three is used to calculate the adaptive distance of the test case The test case is a value selected from the input domain; S300: Select the program under test s, use the existing technology to determine the input domain dimension and input domain range of s, and select P real system programs with the same input domain dimension as s from the public dataset to form a program set D. Use Component One to determine the input domain division ratio parameter α of M; S400: Define the threshold for the number of test cases used, select the first candidate test case t from the input domain range of s, and let the number of used test cases T = 1; S500: Use t as the input of s. When the output result of s is correct, it is considered that no defect of s is found, and then the next step is executed; When the output result of s is incorrect, it is considered that a defect of s is found, and then output the number of test cases F-measure that have been input currently, and end the test; S510: Randomly select k candidate test cases from the input domain range of s to form a candidate test case set C; S520: Select the i-th candidate test case c from C i , using component 2 to calculate c i Adaptive relative contrast Then use component three to calculate c i Adaptive distance Where C = {c1,…,c i ,…c k }, i = 1, ..., k; when When , execute S530; when When , execute S540; exter represents the number of times the function Exter_func is called, and inter represents the number of times the function Inter_func is called; S530: Use the function Exter_func to calculate the nearest neighbor value using Manhattan distance Calculate using formula (1) in, represents the adaptive distance, E represents the set of all test cases that have been executed but did not find defects in s, represents adaptive relative contrast, i=1,...,k; k Arrange in descending order, select the test case corresponding to the maximum adaptive distance value as the best candidate test case, assign the best candidate test case to t, set T=T+1, and execute S600; S540: Use the function Inter_func to calculate the nearest neighbor value using the Euclidean distance Calculate using formula (2) k Arrange in descending order, select the test case corresponding to the maximum adaptive distance value as the best candidate test case, assign the best candidate test case to t, set T=T+1, and execute S600; S600: When T is greater than the threshold for the number of test cases used, end the test, otherwise return to S500.

2. The method for adaptive random software testing based on region division according to claim 1, characterized in that: The system program in S100 is a numerical program.

3. The method for adaptive random software testing based on region division as claimed in claim 2, characterized in that: The specific steps of using Component One in S300 to determine the input domain division ratio parameter α of M are as follows: S310: Let α = 0.1; S320: Divide the input domain corresponding to each real system program in D into an external area and an internal area according to α, where the internal area is represented by α d , the external area is represented by 1-α d , and α∈[0.1,0.9], where d represents the dimension of the real system program; S330: Randomly select a real program W from D, and let W = 1; S340: Select the first candidate test case w from the input domain of W; S350: Use w as the input of W. When the output result is correct, it is considered that no defect of W is found, and then the next step is executed; When the output result is incorrect, it is considered that a defect of W is found, and then output the number of test cases F-measure that have been input currently; and let W = W + 1. If W < P, return to S340, otherwise execute S380; S360: Randomly select k candidate test cases from the input domain of W to form a candidate test case set B; S370: Calculate the adaptive distances of all test cases in B, sort them in descending order, select the test case corresponding to the maximum adaptive distance value and assign it to w as the next input of W, and return to S350; S380: For the current α value, sum and output the F-measures corresponding to all real system programs, and at the same time let α = α + 0.

1. If α <= 0.9, return to S320, otherwise execute S390; S390: Sort all the F-measures in ascending order, and select the α corresponding to the minimum value as the division ratio parameter of M.

4. The method for adaptive random software testing based on region division as claimed in claim 3, characterized in that: The adaptive relative contrast is calculated in S520 The details are as follows: The i-th candidate test case c in C i The dimension of is d; Calculate c i Adaptive relative contrast in dimension j The expression is as follows: Where j represents the jth dimension and j=1,…,d, Represents candidate test case c i The maximum value in the jth dimension, Represents candidate test case c i Minimum value in the jth dimension; Calculate c using formula (3) i The adaptive relative contrast of all dimensions, the minimum value of the adaptive relative contrast is selected as c i Adaptive relative contrast

Images