Pairwise covering array optimization method and testing method based on dual-mode combined search
By optimizing the paired covering array through dual-mode combination search, the problem of high testing overhead in the existing technology is solved, and more efficient software testing is achieved.
Patent Information
- Application Number
- CN202311581581.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-24
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-11-24
AI Technical Summary
Existing pairwise covering array generation algorithms have not been effectively optimized, resulting in excessive testing overhead and making it difficult to efficiently test all possible configurations of highly configurable software.
A dual-mode combinatorial search method is adopted to optimize the pairwise covering array by alternating between greedy mode and exploration mode. The size of the pairwise covering array is optimized by combining loss function, mild modification technique, forced covering technique and allocation-level taboo strategy.
It effectively compresses the size of the pairwise coverage array, improves the test efficiency of highly configurable software, and reduces test overhead.
Smart Images
Figure CN117591409B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of combination testing, and more particularly to a pairwise covering array optimization method and a testing method based on dual-mode combination search. Background Art
[0002] As the industry's demand for customization and diversification continues to grow, the importance of highly configurable software is growing. Highly configurable software provides users with a vast array of options to personalize their software, but this also presents new challenges for software testing. As the number of software options grows, the number of possible configurations also increases exponentially, making testing all possible configurations impractical.
[0003] Combinatorial testing is a feasible method for testing highly configurable software. Existing combinatorial testing methods construct a covering array (i.e., a set of test cases) to test the interaction triggering problems between t options. In pairwise testing, the most commonly used combinatorial testing method aims to generate a pairwise covering array, i.e., a set of test cases or a set of assignments. Currently, many pairwise covering array generation algorithms have been proposed, but the current advanced pairwise covering array generation algorithms only focus on the generation of pairwise covering arrays, and do not take any optimization measures for the pairwise covering arrays, leaving room for further optimization.
[0004] Therefore, how to optimize the paired covering array to obtain a smaller paired covering array so as to reduce the test overhead is a problem that those skilled in the art need to solve urgently. Summary of the Invention
[0005] In view of this, the present invention provides a pairwise covering array optimization method and a testing method based on dual-mode combined search to solve some of the technical problems mentioned in the background technology.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] The pairwise covering array optimization method based on dual-mode combined search includes the following steps:
[0008] S1. Input the pairwise covering array into the loss function to calculate the loss of each allocation in the pairwise covering array. Select an allocation from the pairwise covering array based on the calculated loss and remove it to obtain a partial covering array.
[0009] S2. Modify the remaining allocations in the partial covering array and perform pairwise tuple coverage using the greedy mode and the exploration mode according to the dual-mode combined search framework to obtain a modified pairwise covering array or a partial covering array. The greedy mode and the exploration mode are run alternately until the partial covering array becomes a pairwise covering array.
[0010] S3. Repeat steps S1 and S2, determine the time to stop the optimization according to the search budget, and obtain the optimized pairwise covering array.
[0011] Preferably, the loss for each assignment in the pairwise covering array is calculated as:
[0012] loss(α)=|U(A-α)|-|U(A)|
[0013] Wherein, |U(A-α)| is the set size of uncovered pair tuples after removing the assignment α from the pairwise covering array A, and |U(A)| is the set size of uncovered pair tuples of the pairwise covering array A.
[0014] Preferably, the method for selecting the allocation to be removed from the pairwise covering array is:
[0015] α=argmax(loss(α i ))
[0016] Among them, loss(α i ) is the loss of the i-th assignment in the pairwise covering array A.
[0017] Preferably, the specific content of the greedy mode operation is:
[0018] Select an uncovered pair of tuples τ from the uncovered pair of tuples in the partially covered array, select a valid allocation α that does not violate the allocation-level taboo policy, and use the mild modification technique and the forced coverage technique to modify the uncovered pair of tuples τ into another valid allocation α′ to cover the uncovered pair of tuples τ. Denote a modification (α, α′) as a candidate operation (α, α′), calculate the benefit of each candidate operation (α, α′), and select a candidate operation to execute based on the benefit of the candidate operation.
[0019] The specific contents of the exploration mode are:
[0020] Randomly traverse each variable e of the partially covered array. For each variable e of the partially covered array, select an allocation α, modify the variable e in α to obtain a new allocation α′. If α′ is a legal allocation, perform the operation (α, α′) and terminate the exploration mode.
[0021] Preferably, the specific content of the allocation-level taboo strategy is:
[0022] For the allocation α in A, calculate the number of search steps age(α) since the last modification. If age(α) < δ, then it is forbidden to modify α, δ to the specified number of search steps.
[0023] Preferably, the specific content of the light modification technology is:
[0024] For the specified uncovered pair tuple τ={(o i ,q i ), (o j ,q j )}, modify the option o in the allocation α i , o j The value of q i ,q j , and obtain the modified distribution α′;
[0025] The forced coverage technology modification process is:
[0026] For the specified uncovered paired tuple τ and the allocation α to be modified, call the SAT solver to generate an allocation α′ that is similar to α and covers the paired tuple;
[0027] During the search process, the forced coverage technique is called with a certain probability to obtain the modified allocation α′.
[0028] Preferably, the profit(α,α′) of the candidate operation (α,α′) is specifically:
[0029] profit(α,α′)=|U(B)|-|U(B-a+α′)|
[0030] Among them, |U(B-α+α′)| represents the set size of uncovered pair tuples after the allocation α is modified to the allocation α′ in the partially covered array B, and |U(B)| represents the set size of uncovered pair tuples in the partially covered array B.
[0031] Preferably, the content of selecting a candidate operation to be executed according to the benefits of the candidate operation is:
[0032] (α, α′)=argmax(profit(α i ,α′ i )) Among them, (α i ,α i ′) represents the i-th candidate operation.
[0033] Preferably, the specific content of determining the time to stop optimization is:
[0034] Initialize the search budget γ to γ0, and update the search budget γ to γ-1 after each search step. When A becomes a pairwise covering array, reset the search budget γ to γ0. When the search budget γ is 0, the optimization process stops.
[0035] A testing method, based on the pairwise covering array optimization method based on dual-mode combined search, comprises:
[0036] S1. Generate pairwise covering arrays;
[0037] S2. Optimizing the paired covering array using a pairwise covering array optimization method based on a dual-mode combinatorial search to obtain an optimized paired covering array;
[0038] S3. Software testing using the optimized pairwise covering array.
[0039] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a pairwise covering array optimization method and testing method based on dual-mode combinatorial search. By removing allocations from the pairwise covering array and modifying the remaining allocations, the scale of the pairwise covering array can be further compressed, thereby achieving optimization of the pairwise covering array, which helps to improve the efficiency of highly configurable software testing and reduce testing overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0041] Figure 1 A schematic flow chart of the paired covering array optimization method based on dual-mode combined search provided by the present invention. DETAILED DESCRIPTION
[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0043] The embodiment of the present invention discloses a pairwise covering array optimization method based on dual-mode combined search, comprising the following steps:
[0044] S1. Input the pairwise covering array into the loss function to calculate the loss of each allocation in the pairwise covering array. Select an allocation from the pairwise covering array based on the calculated loss and remove it to obtain a partial covering array.
[0045] S2. Modify the remaining allocations in the partial covering array and perform pairwise tuple coverage using the greedy mode and the exploration mode according to the dual-mode combined search framework to obtain a modified pairwise covering array or a partial covering array. The greedy mode and the exploration mode are run alternately until the partial covering array becomes a pairwise covering array.
[0046] S3. Repeat steps S1 and S2, determine the time to stop the optimization according to the search budget, and obtain the optimized pairwise covering array.
[0047] To further implement the above technical solution, the method for calculating the loss of each assignment in the pairwise covering array is:
[0048] loss(α)=|U(A-α)|-|U(A)|
[0049] Wherein, |U(A-α)| is the set size of uncovered pair tuples after removing the assignment α from the pairwise covering array A, and |U(A)| is the set size of uncovered pair tuples of the pairwise covering array A.
[0050] To further implement the above technical solution, the method for selecting the allocation to be removed from the pairwise covering array is:
[0051] α=argmax(loss(αi))
[0052] Among them, loss(α i ) is the loss of the i-th assignment in the pairwise covering array A.
[0053] In order to further implement the above technical solution, the specific content of the greedy mode operation is as follows:
[0054] Select an uncovered pair of tuples τ from the set of uncovered pair of tuples in the partially covered array, select a valid allocation α that does not violate the allocation-level taboo strategy, and use the mild modification technique and the forced covering technique to modify the uncovered pair of tuples τ into another valid allocation α′ to cover the uncovered pair of tuples τ. Denote a modification (α, α′) as a candidate operation (α, α′), calculate the profit (α, α′) of each candidate operation (α, α′), and select a candidate operation to execute based on the profit of the candidate operation.
[0055] In this embodiment, since the local search algorithm usually faces a loop problem during the search process and the search stays in a small part of the search space, and considering that the unit-level prohibition strategy is weak and insufficient to handle the loop problem, an allocation-level taboo strategy is designed.
[0056] In order to further implement the above technical solution, the specific content of the allocation-level taboo strategy is as follows:
[0057] For an allocation α in a partially covered array, calculate the number of search steps age(α) since the last modification. If age(α) < δ, then modifying α and δ to the specified number of search steps is prohibited.
[0058] In order to further implement the above technical solution, the specific content of the slightly modified technology is as follows:
[0059] For the specified uncovered pair tuple τ={(o i ,q i ), (o j ,q j )}, modify the option o in the allocation α i , o j The value of q i ,q j , and obtain the modified distribution α′;
[0060] In this embodiment, since highly configurable software has a large number of complex constraints, the light modification strategy may not be able to cover the specified pair of tuples. Considering that the SAT solver can find an effective allocation that covers the specified pair of tuples, a forced coverage technique is designed;
[0061] The forced coverage technology modification process is:
[0062] For the specified uncovered paired tuple τ and the allocation α to be modified, call the SAT solver to generate an allocation α′ that is similar to α and covers the paired tuple;
[0063] Considering that calling the SAT solver for solving requires a certain amount of computing time, in order to ensure the efficiency of the solution, during the search process, the forced covering technique is called with a certain probability to obtain the modified allocation α′.
[0064] In order to further implement the above technical solution, the profit(α,α′) of the candidate operation (α,α′) is specifically:
[0065] profit(α,α′)=|U(B)|-|U(B-α+α′)|
[0066] Among them, |U(B-α+α′)| represents the set size of uncovered pair tuples after the allocation α is modified to the allocation α′ in the partially covered array B, and |U(B)| represents the set size of uncovered pair tuples in the partially covered array B.
[0067] In order to further implement the above technical solution, a candidate operation is selected based on the benefits of the candidate operation and the execution content is as follows:
[0068] (α, α′)=argmax(profit(α i ,α′ i ))
[0069] Among them, (α i ,α i ′) represents the i-th candidate operation.
[0070] In this embodiment, considering that only a greedy search strategy is used in the search process, the search process may fall into a local optimal situation and the search may stagnate in a small part of the search space, an exploration mode is designed.
[0071] In order to further implement the above technical solutions, the specific contents of the exploration model are as follows:
[0072] Randomly traverse each variable e of the partially covered array. For each variable e of the partially covered array, select an allocation α, modify the variable e in α to obtain a new allocation α′. If α′ is a legal allocation, perform the operation (α, α′) and terminate the exploration mode.
[0073] Exploring patterns using random search strategies can effectively explore promising search spaces and further optimize pairwise covering arrays.
[0074] In order to further implement the above technical solution, the specific content of determining the time to stop optimization is as follows:
[0075] Initialize the search budget γ to γ0, and update the search budget γ to γ-1 after each search step. When the partial covering array becomes a pairwise covering array, reset the search budget γ to γ0. When the search budget γ is 0, the optimization process stops.
[0076] A testing method based on a pairwise covering array optimization method based on a dual-pattern combinatorial search, comprising:
[0077] S1. Generate pairwise covering arrays;
[0078] S2. Optimizing the paired covering array using a pairwise covering array optimization method based on a dual-mode combinatorial search to obtain an optimized paired covering array;
[0079] S3. Software testing using the optimized pairwise covering array.
[0080] In another embodiment, a dataset of 124 instances of pairwise covering array generation problems is collected from real-world highly configurable systems. Each instance is encoded into a conjunction normal form, as shown in Table 1 below:
[0081] Table 1 shows the dataset statistics for the pairwise covering array generation problem:
[0082] Number of variables Constraint Quantity Minimum 94 190 Maximum 11254 62183
[0083] For the dataset, pairwise covering arrays were generated using different pairwise covering array generation methods, including SamplingCA, AutoCCAG, FastCA, and TCA. These were run under all instances of the pairwise covering array generation problem. The SamplingCA method was used to generate the pairwise covering arrays required for the input of the present invention. The results are shown in Table 2 below:
[0084] Table 2 compares the running results for all instances of the pairwise covering array generation problem:
[0085]
[0086]
[0087] Among them, SamplingCA is a pairwise covering array generation method that adopts context-aware sampling strategy, AutoCCAG is a pairwise covering array generation method based on automatic algorithm optimization, FastCA is a meta-heuristic pairwise covering array generation method, and TCA is a high-performance meta-heuristic pairwise covering array generation method; from the offline experimental results, the size of the pairwise covering array generated by the pairwise covering array optimization algorithm based on dual-mode combined search proposed in the present invention is smaller than that of other advanced pairwise covering array generation methods, which shows the effectiveness of the pairwise covering array optimization method based on dual-mode combined search proposed in the present invention for optimizing pairwise covering arrays.
[0088] The comparison of the ideal experimental results using different methods as the initialization method of the present invention is shown in Table 3 below:
[0089] Table 3 compares the effects of using different methods as initialization methods:
[0090]
[0091] It can be seen from Table 3 that by using different algorithms as the initialization method of the present invention, the method of the present invention has excellent performance in optimizing the size of the pairwise covering array.
[0092] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0093] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A pairwise covering array optimization method based on dual-mode combined search, characterized in that: The following steps are involved: S1. Input the pairwise covering array into the loss function to calculate the loss of each allocation in the pairwise covering array. Select an allocation from the pairwise covering array based on the calculated loss and remove it to obtain a partial covering array. S2. Modify the remaining allocations in the partial covering array and perform pairwise tuple coverage using the greedy mode and the exploration mode according to the dual-mode combined search framework to obtain a modified pairwise covering array or a partial covering array. The greedy mode and the exploration mode are run alternately until the partial covering array becomes a pairwise covering array. S3 repeats steps S1 and S2, determines the time to stop optimization according to the search budget and obtains the optimized pairwise covering array, and uses the optimized pairwise covering array for software testing; A coverage array is a set of test cases; The specific contents of the greedy mode operation are: Select an uncovered pair of tuples τ from the uncovered pair of tuples in the partially covered array, select a valid allocation α that does not violate the allocation-level taboo policy, and use the mild modification technique and the forced coverage technique to modify the uncovered pair of tuples τ into another valid allocation α′ to cover the uncovered pair of tuples τ. Denote a modification (α, α′) as a candidate operation (α, α′), calculate the benefit of each candidate operation (α, α′), and select a candidate operation to execute based on the benefit of the candidate operation. The specific contents of the exploration mode are: Randomly traverse each variable e of the partially covered array. For each variable e of the partially covered array, select an allocation α, modify the variable e in α to obtain a new allocation α′. If α′ is a legal allocation, perform the operation (α, α′) and terminate the exploration mode. The specific content of the light modification technology is: For the specified uncovered pair tuple τ={(o i ,q i ), (o j ,q j )}, modify the option o in the allocation α i , o j The value of q i ,q j , and obtain the modified distribution α′; The forced coverage technology modification process is: For the specified uncovered paired tuple τ and the allocation α to be modified, call the SAT solver to generate an allocation α′ that is similar to α and covers the paired tuple; During the search process, the forced coverage technique is invoked probabilistically to obtain the modified allocation α′.
2. The pairwise covering array optimization method based on dual-mode combined search according to claim 1, characterized in that: The loss for each assignment in the pairwise covering array is calculated as: loss(α)=|U(A-α)|-|U(A)| Wherein, |U(A-α)| is the set size of uncovered pair tuples after removing the assignment α from the pairwise covering array A, and |U(A)| is the set size of uncovered pair tuples of the pairwise covering array A.
3. The pairwise covering array optimization method based on dual-mode combined search according to claim 1, characterized in that: The method for selecting the removed allocation from the pairwise covering array is: α=argmax(loss(α i )) Among them, loss(α i ) is the loss of the i-th assignment in the pairwise covering array A.
4. The pairwise covering array optimization method based on dual-mode combined search according to claim 1, characterized in that: The specific content of the allocation-level taboo strategy is: For an allocation α in a partially covered array, calculate the number of search steps age(α) since the last modification. If age(α) < δ, then modifying α and δ to the specified number of search steps is prohibited.
5. The pairwise covering array optimization method based on dual-mode combined search according to claim 1, characterized in that: The profit(α,α′) of the candidate operation (α,α′) is specifically: profit(α,α′)=|U(B)|-|U(B-α+α′)| Among them, |U(B-α+α′)| represents the set size of uncovered pair tuples after the allocation α is modified to the allocation α′ in the partially covered array B, and |U(B)| represents the set size of uncovered pair tuples in the partially covered array B.
6. The pairwise covering array optimization method based on dual-mode combined search according to claim 5, characterized in that: The content of selecting a candidate operation based on its benefits is as follows: (α, α′)=argmax(profit(α i ,a' i )) Among them, (α i ,α i ′) represents the i-th candidate operation.
7. The pairwise covering array optimization method based on dual-mode combined search according to claim 1, characterized in that: The specific contents of determining the time to stop optimization are: Initialize the search budget γ to γ0, and update the search budget γ to γ-1 after each search step. When the partial covering array becomes a pairwise covering array, reset the search budget γ to γ0. When the search budget γ is 0, the optimization process stops.
8. A testing method, characterized in that: The pairwise covering array optimization method based on dual-mode combined search according to any one of claims 1 to 7 comprises: S1. Generate pairwise covering arrays; S2. Optimizing the paired covering array using a pairwise covering array optimization method based on a dual-mode combinatorial search to obtain an optimized paired covering array; S3. Software testing using the optimized pairwise covering array.