High-test-strength test set acquisition method applied to configurable software system

Abstract

The invention discloses a high-test-strength test set acquisition method applied to a configurable software system, which comprises the following steps of: acquiring a corresponding Boolean formula based on a configuration constraint relationship among configuration items in a to-be-tested configurable software system; constructing an initial test set with the test strength of 2 through a combined test solver; identifying each to-be-tested triple in the to-be-tested configurable software system by adopting a rapid ineffectiveness detection technology, and generating a triple set which is not covered by the initial test set and meets a Boolean formula; constructing a plurality of test cases based on the triple set, and screening out the optimal test case of the current round through a test case evaluation function; adding the optimal test case into the initial test set, and updating the triple set based on the optimal test case; and repeating the process until a final test set with the coverage test strength of 3 is successfully constructed. According to the method, the execution efficiency of the algorithm is greatly improved, and the required high-strength test set can be generated more quickly.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and more particularly to a method for acquiring a high-test intensity test set applied to a configurable software system. Background Art

[0002] Software has become a key driver of my country's economic development and industrial innovation, especially in the fields of aerospace, urban transportation, etc., where software quality assurance is crucial. Design defects can lead to catastrophic consequences, endangering the safety of life and property, and causing huge economic losses. As the demand for customized software grows, highly configurable software plays an important role in meeting changing needs and has rich options for users to customize. However, despite the strong customization capabilities of this type of software, testing faces huge challenges, especially when dealing with the exponential growth in the number of configurations.

[0003] Combinatorial testing is widely used to detect option interaction faults in configurable software. By generating coverage arrays, various option combinations are systematically tested to reduce the number of test cases and improve test efficiency. A coverage array is a set of test cases that ensures that all possible combinations of any t options in the software are covered by at least one test case. Here, t represents the test intensity. Although coverage arrays with a test intensity of 2 are commonly used in industry, empirical studies have shown that a test set with a test intensity of 2 can only detect about 77% of defects, while a test set with a test intensity of 3 can identify more than 95% of faults. Therefore, it is particularly urgent to study how to generate a coverage array with high test intensity, that is, a test intensity of 3.

[0004] The size of the covering array directly affects the test efficiency. Therefore, generating a smaller covering array is crucial to reducing the test time. However, generating a covering array that satisfies complex constraints is an NP-hard combinatorial optimization problem. Especially when facing highly configurable software, existing methods often encounter problems such as memory exhaustion and program failure when dealing with large-scale covering arrays. For example, when dealing with a system with thousands of options, generating a covering array with a test intensity of 2 is very time-consuming and produces an array that is too large, which seriously affects the test efficiency and effect.

[0005] Although recent algorithms have solved the scalability problem of generating covering arrays with a test strength of 2, covering arrays with a test strength of 3 still face great challenges. In particular, as the number of combinations required to be covered increases significantly when t=3, the efficiency of existing algorithms is difficult to meet the requirements. In summary, solving the scalability problem of generating covering arrays with a test strength of 3 has become an important research topic in the field of combinatorial testing, and has important practical significance for the effective testing of highly configurable software. Summary of the invention

[0006] In view of this, the present invention provides a method for obtaining a high-test-intensity test set for a configurable software system. By gradually adding test cases on the basis of an initial test set with a test intensity of 2, a test set with a test intensity of 3 is successfully generated, thereby effectively addressing the scalability challenges of high-test-intensity combinatorial testing, and efficiently generating a test set with a test intensity of 3 for large-scale instances in practical applications, so as to at least solve some of the technical problems mentioned in the above background technology.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] The present invention provides a method for obtaining a high-test intensity test set for a configurable software system, comprising the following steps:

[0009] Step S1: based on the configuration constraint relationship between configuration items in the configurable software system to be tested, obtain a corresponding Boolean formula;

[0010] Step S2: construct an initial test set with a test intensity of 2 by combining test solvers;

[0011] Step S3: using a fast invalidity detection technology to identify each triple to be tested in the configurable software system to be tested, and generating a set of triples that are not covered by the initial test set and satisfy the Boolean formula;

[0012] Step S4: constructing a test case evaluation function based on the triple set;

[0013] Step S5: construct multiple test cases based on the triple set, and select the best test case of the current round through the test case evaluation function;

[0014] Step S6: adding the optimal test case to the initial test set to form an intermediate test set; updating the triple set according to the intermediate test set to obtain an intermediate triple set;

[0015] Step S7: Based on the intermediate triple set, repeat steps S5 to S6 until a final test set with a test strength of 3 is generated.

[0016] Preferably, it also includes:

[0017] Step S8: Optimize the scale of the final test set.

[0018] Preferably, the step S3 specifically includes:

[0019] Step S31: for any triplet to be tested consisting of configuration items in the configurable software system to be tested, determine whether it is covered by the initial test set;

[0020] Step S32: for each triplet to be tested that is not covered by the initial test set, decompose it into a plurality of 2-tuples consisting of subsets of the triplet to be tested;

[0021] Step S33: Using a fast invalidity detection technique, detect multiple bigrams corresponding to each triplet to be tested:

[0022] (1) if any one of the multiple bigrams is not covered by the initial test set, it is determined that the corresponding triple to be tested does not satisfy the Boolean formula;

[0023] (2) If multiple tuples are covered by the initial test set, a SAT solver is used to determine whether the corresponding triples to be tested conform to the Boolean formula;

[0024] Step S33: taking the triples to be tested that are not covered by the initial test set and satisfy the Boolean formula as a triple set.

[0025] Preferably, the step S5 specifically includes:

[0026] Step S51: for any configuration item x, count the 0 / 1 distribution ratio of x in the triples to be tested involving the configuration item in the triple set, and set a corresponding sampling probability according to the 0 / 1 distribution ratio; sample each configuration item according to the sampling probability to generate a test case;

[0027] Step S52: Repeat step S51 to generate multiple test cases, thereby constructing a test case candidate set;

[0028] Step S53: Using the test case evaluation function, filter out the test case that can cover the largest number of triples in the triple set in the current round from the test case candidate set as the optimal test case.

[0029] Preferably, in step S6, updating the triple set according to the intermediate test set includes: removing the triples that have been covered by the intermediate test set from the triple set of the current round.

[0030] Preferably, the step S7 specifically includes:

[0031] S71, based on the intermediate triple set, repeatedly executing step S5-step S6 until the constructed test cases cannot cover the remaining triples in the intermediate triple set of the current round, and stopping the repetitive process;

[0032] S72. For each remaining triple in the intermediate triple set of the current round, construct a corresponding test case that can be covered, recorded as the remaining test case; add the remaining test case to the intermediate test set of the current round to obtain a final test set with a test intensity of 3.

[0033] Preferably, the step S8 specifically includes:

[0034] Step S81: for the final test set, keep the initial test set generated in step S2 unchanged, and pre-delete one of the test cases added in steps S5 to S7;

[0035] Step S82: determine whether the final test set after the pre-deletion process can cover all triples in the triple set constructed in step S3; if so, completely delete the pre-deleted test cases; if not, completely delete the pre-deleted test cases and modify the final test set after the complete deletion process so that it can cover all triples in the triple set;

[0036] Step S83: Repeat steps S81 and S82 to achieve compression optimization of the final test set size.

[0037] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a method for obtaining a high-test intensity test set for a configurable software system, which has the following beneficial effects:

[0038] 1. The present invention constructs a test set with a test intensity of 3 based on a test set with a test intensity of 2, which can effectively reduce the number of triples that need to be covered. Compared with the traditional method of building a test set with a test intensity of 3 from scratch, the present invention significantly reduces the problem space, thereby greatly improving the execution efficiency of the algorithm, especially when dealing with large-scale, highly configurable software systems, and can generate the required high-intensity test sets more quickly.

[0039] 2. This invention introduces a fast invalidity detection technology to address the computational challenges in the triple verification process. The fast invalidity detection technology can significantly reduce the number of calls to the SAT solver without reducing the accuracy of the solution, thereby greatly improving the computational efficiency. This improvement makes the testing of complex software systems more feasible and efficient.

[0040] 3. The present invention adopts a local compression strategy instead of comprehensively compressing the entire test set with a test intensity of 3. In the traditional method, when compressing the entire test set with a test intensity of 3, all valid triples must be considered, which will lead to a sharp increase in the problem space and increase the computational complexity. To this end, the compression strategy of the present invention only optimizes the newly added test case set. This strategy effectively reduces the scope to be processed, greatly improves the compression efficiency, ensures that the generated test set is more streamlined without affecting the coverage quality, and reduces the time and resource consumption in the actual testing process.

[0041] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention.

[0042] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A schematic flow chart of a method for acquiring a high-test intensity test set for a configurable software system provided in an embodiment of the present invention.

[0045] Figure 2 A schematic diagram of a process for generating a triplet set provided in an embodiment of the present invention.

[0046] Figure 3 A schematic diagram of the process of screening the optimal test cases provided by an embodiment of the present invention.

[0047] Figure 4 A schematic diagram of a process for performing scale optimization on a final test set provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment 1:

[0050] See also Figure 1 As shown, the embodiment of the present invention discloses a method for obtaining a high-test intensity test set for a configurable software system, comprising the following steps:

[0051] Step S1: based on the configuration constraint relationship between configuration items in the configurable software system to be tested, obtain a corresponding Boolean formula;

[0052] Step S2: construct an initial test set with a test intensity of 2 by combining test solvers;

[0053] Step S3: using a fast invalidity detection technique to identify each triple to be tested in the configurable software system to be tested, and generating a set of triples that are not covered by the initial test set and satisfy the Boolean formula;

[0054] Step S4: constructing a test case evaluation function based on the triple set;

[0055] Step S5: construct multiple test cases based on the triple set, and select the best test case of the current round through the test case evaluation function;

[0056] Step S6: adding the optimal test case to the initial test set to form an intermediate test set; updating the triple set according to the intermediate test set to obtain an intermediate triple set;

[0057] Step S7: Based on the intermediate triple set, repeat steps S5 to S6 until a final test set with a test strength of 3 is generated;

[0058] Step S8: Optimize the scale of the final test set.

[0059] This method adds new test cases to the test set with test intensity 2 and gradually builds a test set with test intensity 3, which effectively alleviates the scalability problem in high test intensity combinatorial testing. This method can efficiently handle large-scale instances and generate test sets that meet test intensity requirements.

[0060] Next, each of the above steps and related features will be described in further detail.

[0061] In the above step S1, based on the configuration constraint relationship between the configuration items in the configurable software system to be tested, the corresponding Boolean formula F is obtained to accurately describe the logical dependency relationship between the configuration items.

[0062] In the above step S2, an initial test set A with a test intensity of 2 is constructed by combining test solvers; specifically:

[0063] The combinatorial test solver is generated by the existing combinatorial test set. Under the premise of satisfying the Boolean formula F, an initial test set A with a test strength of 2 is generated in an efficient and fast way. The initial test set A is designed to cover the vast majority of triples in the configurable software system. The next step will be to expand the initial test set A to a test set with a higher test strength of 3 by adding test cases on this basis.

[0064] In the above step S3, a fast invalidity detection technique is used to identify each triple to be tested in the configurable software system to be tested, and a triple set U that is not covered by the initial test set and satisfies the Boolean formula is generated; see Figure 2 As shown, the following steps are included:

[0065] Step S31: for any triplet to be tested consisting of configuration items in the configurable software system to be tested, determine whether it is covered by the initial test set;

[0066] Step S32: for each triplet to be tested that is not covered by the initial test set, decompose it into a plurality of 2-tuples consisting of subsets of the triplet to be tested; for example, for a triplet (a, b, c), it can be decomposed into 3 2-tuples: (a, b), (a, c) and (b, c);

[0067] Step S33: Using a fast invalidity detection technique, detect multiple bigrams corresponding to each triplet to be tested:

[0068] (1) If any of the two-tuples among the multiple two-tuples is not covered by the initial test set, then the two-tuple does not satisfy the Boolean formula, so it can be determined that the corresponding three-tuple to be tested does not satisfy the Boolean formula;

[0069] (2) If multiple tuples are covered by the initial test set, a SAT solver is used to determine whether the corresponding triples to be tested conform to the Boolean formula;

[0070] Step S34: taking the triples to be tested that are not covered by the initial test set and satisfy the Boolean formula as the triple set U;

[0071] The task of this stage is to construct a set U of triples that have not been covered by the initial test set A and satisfy the Boolean formula F. In order to obtain the set U of triples, an intuitive method is to enumerate all possible triples and check whether each tuple satisfies the Boolean formula and is not covered. However, since verifying whether the uncovered triples satisfy the Boolean formula requires a call to the SAT solver, and the SAT problem is NP-complete, each call requires a long computation time, resulting in the naive enumeration method being very time-consuming.

[0072] Considering the existence of hard constraints, there are actually a large number of illegal tuples. In order to speed up the enumeration process and reduce the number of calls to the SAT solver, the embodiment of the present invention adopts a fast invalidity detection technology; the principle of the fast invalidity detection technology is as follows:

[0073] The technology is based on the following property: for a given triple, if any of the three binary tuples derived from it does not satisfy the Boolean formula, then the triple also does not satisfy the Boolean formula.

[0074] The proof is as follows: Suppose a binary tuple derived from a triple is not covered by the initial test set A, which means that there is no legal test case that can cover the binary tuple, so there is no legal test case that can cover the triple, so the triple does not satisfy the Boolean formula.

[0075] Based on this, during the enumeration process, when testing triples, first screen out the triples to be tested that are not covered by the initial test set A, and then use the fast invalidity detection technology to determine whether the three derived bigrams satisfy the Boolean formula; if any bigram does not satisfy the Boolean formula, it is possible to directly determine that the triple does not satisfy the Boolean formula without calling the SAT solver; if all bigrams are covered by the initial test set A, the SAT solver can be called at this time to determine whether the triple satisfies the Boolean formula, ensuring that the screened triples satisfy the Boolean formula. In addition, since the test set A with a known test strength of 2 is known, determining whether a bigram satisfies the Boolean formula is equivalent to determining whether it is covered by A, and there is no need to call the SAT solver, which ensures the efficiency of the fast invalidity detection technology.

[0076] In the above step S4, a test case evaluation function is constructed based on the triple set; specifically, the test case evaluation function can be defined as benefit(tc), which is used to represent the number of triples in the triple set U that can be newly covered after a certain test case tc is added to the initial test set A; in the subsequent test case generation process, benefit will be used to evaluate the coverage and effectiveness of the test case.

[0077] In the above step S5, multiple test cases are constructed based on the triple set, and the optimal test case of the current round is screened out through the test case evaluation function; for details, see Figure 3 As shown, including:

[0078] Step S51: for any configuration item x, count the 0 / 1 distribution ratio of x in the triples to be tested involving the configuration item in the triple set, and set the corresponding sampling probability according to the 0 / 1 distribution ratio; sample each configuration item according to the sampling probability to generate a test case;

[0079] Step S52: Repeat step S51 to generate multiple test cases, thereby constructing a test case candidate set;

[0080] Step S53: From the test case candidate set, filter out the test case that can cover the largest number of triples in the triple set in the current round through the test case evaluation function, and use it as the optimal test case;

[0081] In this stage, the embodiment of the present invention adopts a greedy strategy to select test cases. Specifically, in each iteration, first construct a candidate set containing d test cases, and then select the test case with the largest contribution and add it to the initial test set A. Therefore, the quality of the candidate set constructed in each iteration directly affects the effect of the present invention, so a method for constructing a high-quality candidate set needs to be designed; step S51 is the key step for constructing the candidate set one by one.

[0082] After constructing the test case candidate set, it is necessary to sample and filter out the optimal test case from the test case candidate set through the test case evaluation function; in this process, the concept of sampling probability is: for a given Boolean formula F, the sampling probability prob(x) of any configuration item x represents the probability that x = 1 in the sampling process. Correspondingly, the probability that x = 0 is 1 - prob(x).

[0083] Regarding the calculation of prob(x), it is obviously determined by the triples in the set U. As the set U of uncovered triples is updated in each iteration, the sampling probability prob(x) will also be updated accordingly. For a given configuration item x and triple τ, if τ contains x and the value of x is 0, then τ is called an x-false triple; if the value of x is 1, then τ is called an x-true triple. For this reason, for each configuration item x, define l as the number of x-false triples in U, and e as the number of x-true triples in U. When sampling, in order to make the sampling result cover more triples in U, for each configuration item x, if l < e, the sampling probability prob(x) should be larger; conversely, if l > e, the sampling probability prob(x) should be smaller. Therefore, let the sampling probability prob(x) be the ratio of e to l + e to optimize the sampling process.

[0084] In the above step S6, add the optimal test case to the initial test set to form an intermediate test set; update the triple set according to the intermediate test set to obtain an intermediate triple set;

[0085] Among them, the update of the triple set specifically includes: removing the triples that have been covered by the intermediate test set from the triple set in the current round.

[0086] In the above step S7, based on the intermediate triple set, steps S5 to S6 are repeatedly executed until a final test set with a test strength of 3 is generated; specifically, the process includes:

[0087] S71, based on the intermediate triple set, repeatedly execute step S5-step S6 until the constructed test cases cannot cover the remaining triples in the intermediate triple set of the current round, and stop the repetitive process;

[0088] S72. For each remaining triple in the intermediate triple set of the current round, construct a corresponding test case that can be covered, recorded as the remaining test case; add the remaining test cases to the intermediate test set of the current round to obtain the final test set with a test intensity of 3; ensure that higher-level configuration item interactions are fully tested.

[0089] In the above step S8, the final test set is scaled; see Figure 4 As shown, including:

[0090] Step S81: for the final test set, keep the initial test set generated in step S2 unchanged, and pre-delete one of the test cases added in steps S5 to S7;

[0091] Step S82: determine whether the final test set after the pre-deletion process can cover all triples in the triple set constructed in step S3; if so, completely delete the pre-deleted test cases; if not, completely delete the pre-deleted test cases and modify the final test set after the complete deletion process so that it can cover all triples in the triple set;

[0092] The modification operation can be referred to as follows:

[0093] First, a triple τ is randomly selected from the set of triples constructed in step S3 that is not currently covered; then, a candidate operation set Θ is constructed,

[0094] Θ={(β,β°τ)|, β is the test case added in steps S5 to S7, β°τ is legal}

[0095] Among them, β°τ means that the options in the test case β that are the same as those in the triple τ are modified to the same values ​​as those in the triple τ, and the values ​​of the other options remain unchanged. If the candidate operation set Θ is not empty, the operation in Θ is selected and executed. Otherwise, a test case β is randomly selected and modified into a legal test case that can cover τ using the SAT solver. The above modification operation is repeated until all the triples in the triple set constructed in step S3 are covered, then the modification operation is stopped and the next round of random deletion begins.

[0096] Step S83: Repeat steps S81 and S82 until the final test set after complete deletion cannot cover all triples in the triple set even if it is modified, thereby achieving compression optimization of the size of the final test set.

[0097] At this stage, the embodiment of the present invention adopts a local compression strategy instead of comprehensively compressing the entire test set with a test strength of 3. When compressing a test set with a test strength of 3, the traditional method must consider all triples that are not covered by the initial test set A and satisfy the Boolean formula, which leads to a sharp increase in the problem space and an increase in computational complexity. To this end, the embodiment of the present invention only optimizes the newly added test cases (hereinafter referred to as T) in steps S5-S7, and only needs to consider the triple set U obtained in step S3;

[0098] In each search step, we first check whether the current T covers all triples in U. If so, we randomly remove a test case from T to reduce the size of T; if not, we modify T to try to cover more triples in U.

[0099] The core of this compression strategy is to gradually solve a series of decision problems: given a specific size s, the decision problem is to find a test set of size s so that all triples in U are covered. Once a test set of size s that satisfies the conditions is found, then try to solve the decision problem of size s-1. In this way, the size of the test set can be gradually compressed. When a smaller test set cannot be found after L search steps, the compression process terminates. L is a hyperparameter used to strike a balance between the effectiveness and efficiency of the compression process.

[0100] Embodiment 2:

[0101] The embodiment of the present invention will illustrate a method for obtaining a high-test intensity test set for a configurable software system provided by the present invention through a more specific embodiment.

[0102] This example tests a highly configurable Linux system, with the goal of evaluating the interaction between any three configuration options. The system has 1232 configuration options, and there are 3154 constraints between these options. Theoretically, the configuration space of the system is 2 1232 If a combination test with a test intensity of 3 is used, it will greatly save test resources and time.

[0103] The method to obtain a test set with a test intensity of 3 is as follows:

[0104] Step S1: By analyzing the configuration manual and source code of the Linux system, the constraint relationship between all configuration options is identified and converted into a Boolean expression to form a Boolean formula F.

[0105] Step S2: Using the existing combinatorial test solver, construct an initial test set A with a test strength of 2.

[0106] Step S3: When generating a set of triples that are not covered and satisfy the Boolean formula F, first use the fast invalidity detection technology to determine whether the current triple is invalid by enumerating all possible value combinations between the three options in the system. If it cannot be quickly determined to be invalid, then further call the SAT solver for judgment, and finally obtain a set of triples that are not covered by A and satisfy the Boolean formula F.

[0107] Steps S4 and S5: When generating the candidate set, the sampling probability is calculated according to the current situation of the tuples in U, and multiple test cases are generated based on these probabilities to form the candidate set.

[0108] Then, according to the distribution of tuples in the current U, calculate the benefit of each test case in the candidate set. For example, if a test case can cover 5 tuples in U, its benefit is 5. Select the test case with the highest benefit and add it to the test set. If the benefits of all test cases in the candidate set are 0, then for a specific tuple in U, construct a test case that can cover it.

[0109] Step S:6: As new test cases are added, the covered tuples are gradually removed from U.

[0110] Step S7: By continuously iterating the above process, the initial test set A with a test intensity of 2 is expanded to a test set with a test intensity of 3.

[0111] Step S8: To further optimize the scale of the test set, the present invention adopts a local compression strategy, that is, under the premise of keeping the initial test set A unchanged, the subsequent newly added test case set is compressed, so as to obtain a smaller and more efficient test set with a test intensity of 3; specifically:

[0112] First, a test case that is not in A in the current test set is randomly deleted, and then observe whether the test set still covers all triples that satisfy the Boolean formula after deleting the test case. If the deletion operation adds uncovered triples, the current test set is modified to try to cover all triples that satisfy the Boolean formula. By repeating the above steps, the size of the test set will continue to shrink.

[0113] Through the above steps, this embodiment demonstrates how to efficiently generate and optimize a test set with a test intensity of 3 in a highly configurable Linux system, thereby ensuring sufficient testing of key system configurations under limited resources.

[0114] In addition, this embodiment also collects a dataset consisting of 122 public instances based on a highly configurable system, and each instance is encoded as a conjunction normal form.

[0115] For each test case generation method (i.e., the most advanced SamplingCA and CAmpactor), according to preliminary experimental results, they can construct a test set with a test intensity of 3 for a highly configurable system within 2 CPU days. Therefore, in order for SamplingCA and CAmpector to successfully generate a test set with a test intensity of 3 for all adopted instances, in the experiment, the deadline for each algorithm to run was set to 172800 CPU seconds (i.e., 2 CPU days).

[0116] In addition, since all methods including this scheme are random methods, for each instance (i.e., the configuration encoded as the conjunction normal form in the data set), three methods are used in turn to perform 10 independent runs to try to generate a test set with a test intensity of 3 for it. The size of the minimum coverage array is represented as "min.", the average size is represented as "avg.", and the average time spent is represented as "time". In addition, in order to evaluate the overall performance, for each algorithm, the average size of the coverage array on all instances is represented by "avg.size", and the average running time is represented by "avg.time". All running times are in CPU seconds. After running the test on all instances, the test results obtained are shown in the following table:

[0117] Table 1 Average size and average time of the coverage array of this scheme and its comparison algorithm

[0118] This program SamplingCA CAmpactor avg.size 526.5 903.9 862.1 avg.time(sec) 1778.3 67198.6 167433.9

[0119] Table 2 Array size and average time covered by this solution and its comparison algorithm on 10 representative examples

[0120]

[0121]

[0122] As can be seen from the above table, the scheme provided by the present invention is the best performing algorithm, and compared with its comparison algorithm, it constructs a much smaller test set with a test strength of 3. This shows its effectiveness. On the entire set of instances, the average size of the test set with a test strength of 3 generated by this scheme is 526.5 (containing an average of 526.5 test cases), while SamplingCA and CAmpactor are 903.9 and 862.1 respectively. Therefore, the size of the test set with a test strength of 3 generated by this scheme is 38.9% smaller than the current state-of-the-art algorithm.

[0123] In addition, the average running time of this scheme is 1778.3 seconds, while SamplingCA and CAmpactor are 67198.6 seconds and 167433.9 seconds respectively. On the 10 selected instances, the running time of this scheme is much less than that of SamplingCA and CAmpactor. Therefore, according to the results in the above table, when building a coverage array with a test intensity of 3 for a highly configurable system, the running speed of this scheme is one or even two orders of magnitude faster than its most advanced comparison algorithm, which demonstrates the efficiency of this scheme.

[0124] In summary, the present invention provides a method for obtaining a high-test-intensity test set for a configurable software system, which significantly improves the reliability and stability of the software system through an effective combination test method. In addition, the present invention uses a test case set for combination testing, which helps to deeply explore the complex interaction failures between configuration options, especially in highly configurable complex software systems, and can greatly reduce the risk of system failures caused by configuration errors, ensuring that the software can operate normally according to the expected functions. This is particularly important for key areas such as aerospace, urban transportation, and smart medical care that have extremely high requirements for software quality. In addition, the present invention increases the test intensity of the combination test from 2 to 3, significantly improving the fault detection capability. According to empirical research data, after the test intensity is increased from 2 to 3, the detection rate of software faults increases from 77% to more than 95%. This improvement greatly enhances the reliability of the software system, reduces the probability of potential failures, and thus improves the user's satisfaction and trust in the software system.

[0125] This specification describes each embodiment in a progressive manner, and each embodiment focuses on the key points that are different from other embodiments. For the same or similar parts in each embodiment, reference can be made to each other for better understanding and application. As for the system corresponding to each embodiment, since its design and implementation are closely related to the method part, less ink is given in the description, and the relevant details can be directly referred to the description of the method part.

[0126] Through the detailed elaboration of this specification, those skilled in the art can clearly understand and implement the present invention. For these embodiments, professionals can make various modifications and adjustments according to demand, and these changes are obvious to professionals in this field. At the same time, the basic principles and novelty of the present invention can be flexibly applied to other embodiments without departing from its core spirit and scope. Therefore, the present invention should not be limited to the specific embodiments shown in this specification, but should cover all wide applications that meet its principles and innovative features.

Claims

1. A method for acquiring a high-test-intensity test set for a configurable software system, characterized in that: The steps include: Step S1: based on the configuration constraint relationship between configuration items in the configurable software system to be tested, obtain a corresponding Boolean formula; Step S2: construct an initial test set with a test intensity of 2 by combining test solvers; Step S3: using a fast invalidity detection technology to identify each triple to be tested in the configurable software system to be tested, and generating a set of triples that are not covered by the initial test set and satisfy the Boolean formula; Step S4: constructing a test case evaluation function based on the triple set; Step S5: construct multiple test cases based on the triple set, and select the best test case of the current round through the test case evaluation function; Step S6: adding the optimal test case to the initial test set to form an intermediate test set; The triple set is updated according to the intermediate test set to obtain an intermediate triple set; Step S7: Based on the intermediate triple set, repeat steps S5 to S6 until a final test set with a test strength of 3 is generated.

2. A method for acquiring a high-test intensity test set for a configurable software system according to claim 1, characterized in that: Also includes: Step S8: Optimize the scale of the final test set.

3. The method for obtaining a high-test-intensity test set for a configurable software system according to claim 1, characterized in that: The step S3 specifically includes: Step S31: for any triplet to be tested consisting of configuration items in the configurable software system to be tested, determine whether it is covered by the initial test set; Step S32: for each triplet to be tested that is not covered by the initial test set, decompose it into a plurality of 2-tuples consisting of subsets of the triplet to be tested; Step S33: Using a fast invalidity detection technique, detect multiple bigrams corresponding to each triplet to be tested: (1) if any one of the multiple bigrams is not covered by the initial test set, it is determined that the corresponding triple to be tested does not satisfy the Boolean formula; (2) If multiple tuples are covered by the initial test set, a SAT solver is used to determine whether the corresponding triples to be tested conform to the Boolean formula; Step S34: taking the triples to be tested that are not covered by the initial test set and satisfy the Boolean formula as a triple set.

4. The method for obtaining a high-test-intensity test set for a configurable software system according to claim 1, characterized in that: The step S5 specifically includes: Step S51: for any configuration item x, count the 0 / 1 distribution ratio of x in the triples to be tested involving the configuration item in the triple set, and set a corresponding sampling probability according to the 0 / 1 distribution ratio; sample each configuration item according to the sampling probability to generate a test case; Step S52: Repeat step S51 to generate multiple test cases, thereby constructing a test case candidate set; Step S53: Using the test case evaluation function, filter out the test case that can cover the largest number of triples in the triple set in the current round from the test case candidate set as the optimal test case.

5. The method for acquiring a high-test intensity test set for a configurable software system according to claim 1, characterized in that: In step S6, updating the triple set according to the intermediate test set includes: removing the triples that have been covered by the intermediate test set from the triple set of the current round.

6. The method for acquiring a high-test-intensity test set for a configurable software system according to claim 1, characterized in that: The step S7 specifically includes: S71, based on the intermediate triple set, repeatedly executing step S5-step S6 until the constructed test cases cannot cover the remaining triples in the intermediate triple set of the current round, and stopping the repetitive process; S72. For each remaining triple in the intermediate triple set of the current round, construct a corresponding test case that can be covered, recorded as the remaining test case; add the remaining test case to the intermediate test set of the current round to obtain a final test set with a test intensity of 3.

7. The method for acquiring a high-test intensity test set for a configurable software system according to claim 2, characterized in that: The step S8 specifically includes: Step S81: for the final test set, keep the initial test set generated in step S2 unchanged, and pre-delete one of the test cases added in steps S5 to S7; Step S82: determine whether the final test set after the pre-deletion process can cover all triples in the triple set constructed in step S3; if so, completely delete the pre-deleted test cases; if not, completely delete the pre-deleted test cases and modify the final test set after the complete deletion process so that it can cover all triples in the triple set; Step S83: Repeat steps S81 and S82 to achieve compression optimization of the final test set size.

Images