Executing tests in a deterministic order

Abstract

A method for detecting and confining defects in a system under test (SUT) includes generating an initial set of test vectors that provide full n-wise coverage of a reduced test space. The method further includes generating and executing the initial set of test cases to obtain a first set of execution results. The method further includes determining that one or more test cases failed to execute based at least in part on the first set of execution results. The method further includes generating a new set of test cases from the selected failing test cases. The method further includes executing the new set of test cases to obtain a second set of execution results. The method further includes detecting and confining defects based at least in part on the second set of execution results.

Description

[Technical field]

[0001] The present invention relates to testing of computer programs, and more particularly to reducing test case infrastructure. [Background technology]

[0002] Computerized devices control almost every aspect of our lives, from writing documents to controlling traffic lights. However, newly designed computerized devices are often prone to errors and therefore require a testing phase to find and correct the errors. The testing phase is considered one of the most difficult tasks in designing a computerized device. The cost of not finding an error can be huge depending on how the computerized device is used, so thorough testing is important. Coverage tools for checking software provide a measure of how well the software being evaluated performed during testing, thereby giving a level of assurance that the software is of high quality. Summary of the Invention

[0003] According to one or more embodiments of the present invention, a method for detecting and localizing defects in a system under test (SUT) includes modeling inputs to the SUT as a set of attribute-value pairs and assigning a probability value to each attribute-value pair from the set of attribute-value pairs. The method further includes reducing a Cartesian product space containing all possible combinations of the attribute-value pairs to a reduced test space by eliminating one or more combinations of the attribute-value pairs based at least in part on the corresponding probability values. The method further includes generating an initial set of test vectors that provide full n-wise coverage of the reduced test space. The method further includes generating an initial set of test cases from the initial set of test vectors. The method further includes executing the initial set of test cases to obtain a first set of execution results. The method further includes determining that one or more test cases have failed to execute based at least in part on the first set of execution results. The method further includes generating a new set of test cases from the selected failing test cases. The method further includes executing the new set of test cases to obtain a second set of execution results. The method further includes detecting and localizing defects based at least in part on the second set of execution results.

[0004] In one or more embodiments of the present invention, the Cartesian product space is further reduced based on architectural constraints of the SUT. Further, in one or more embodiments of the present invention, the architectural constraints include at least one of a requirement that the first attribute cannot take one or more candidate attribute values ​​if the second attribute has a particular attribute value, a requirement that the first attribute must take a particular candidate attribute value if the second attribute has a particular attribute value, or a requirement to introduce a third attribute if the second attribute has a particular attribute value.

[0005] In one or more embodiments of the invention, the attribute-value pairs and corresponding probability values ​​are represented as a hash, and a first attribute-value pair being tested is assigned a first probability value that is higher than a second probability value of a second attribute-value pair.

[0006] In one or more embodiments of the invention, detecting and localizing defects based at least in part on the second set of execution results includes determining particular combinations of attribute values ​​that cause the defects. Generating regression buckets of failing test cases based at least in part on the particular combinations of attribute values ​​that cause the defects.

[0007] The technical solution may also be implemented by a system, a computer program product, an apparatus, a machine, a device, or other practical application in one or more embodiments of the present invention.

[0008] Additionally, in one or more embodiments of the invention, a method for detecting and localizing defects in a system under test (SUT) includes modeling an input to the SUT as a set of attribute-value pairs. The method further includes assigning a probability value to each attribute-value pair from the set of attribute-value pairs. The method further includes reducing a Cartesian product space that includes all possible combinations of the attribute-value pairs to a reduced test space by eliminating one or more combinations of the attribute-value pairs based at least in part on the corresponding probability values. The method further includes generating an initial set of test vectors that provide full n-wise coverage of the reduced test space. The method further includes generating an initial set of test cases from the initial set of test vectors. The method further includes executing the initial set of test cases to obtain a first set of execution results. The method further includes determining that one or more test cases have failed to execute based at least in part on the first set of execution results. The method further includes generating a set of new test cases from the selected failing test cases, where generating the set of new test cases includes, associated with each attribute in the selected failing test cases, generating a respective subset of the new test cases by, at least in part, changing a respective attribute value for the attribute in the selected failing test cases to a respective other candidate attribute value for the attribute not present in any of the one or more test cases that failed to execute. The method further includes executing the set of new test cases to obtain a second set of execution results. The method further includes detecting and localizing defects based at least in part on the second set of execution results.

[0009] According to one aspect, a method is provided for detecting and localizing defects in a system under test (SUT), the method including: modeling input to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space containing all possible combinations of attribute-value pairs to a reduced test space by eliminating one or more combinations of attribute-value pairs based at least in part on the corresponding probability values; generating an initial set of test vectors that provides full n-wise coverage of the reduced test space; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining, based at least in part on the first set of execution results, that one or more test cases failed to execute; generating a new set of test cases from the selected failing test cases; executing the new set of test cases to obtain a second set of execution results; and detecting and localizing defects based at least in part on the second set of execution results.

[0010] According to another aspect, a system includes a memory and a processor coupled to the memory, the processor configured to execute a method for detecting and localizing defects in a system under test (SUT), the method including: modeling inputs to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space containing all possible combinations of the attribute-value pairs to a reduced test space by eliminating one or more combinations of the attribute-value pairs based at least in part on the corresponding probability value; and generating test vectors that provide complete n-wise coverage of the reduced test space. generating an initial set of test cases from the initial set of test vectors; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining that one or more test cases failed to execute based at least in part on the first set of execution results; generating a new set of test cases from the selected failing test cases; executing the new set of test cases to obtain a second set of execution results; and detecting and localizing defects based at least in part on the second set of execution results.

[0011] According to another aspect, a computer program product includes a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by the processor, cause a processor to execute a method for detecting and localizing defects in a SUT, the method including: modeling an input to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space that includes all possible combinations of the attribute-value pairs to a reduced test space by eliminating one or more combinations of the attribute-value pairs based at least in part on the corresponding probability value; and performing a test that provides complete n-wise coverage of the reduced test space. In one embodiment, a computer program product is provided that includes generating an initial set of test vectors, generating an initial set of test cases from the initial set of test vectors, executing the initial set of test cases to obtain a first set of execution results, determining that one or more test cases failed to execute based at least in part on the first set of execution results, generating a new set of test cases from the selected failed test cases, executing the new set of test cases to obtain a second set of execution results, and detecting and localizing defects based at least in part on the second set of execution results.

[0012] According to another aspect, a method for detecting and localizing defects in a system under test (SUT) includes modeling inputs to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space containing all possible combinations of the attribute-value pairs to a reduced test space by eliminating one or more combinations of the attribute-value pairs based at least in part on the corresponding probability values; generating an initial set of test vectors that provides full n-wise coverage of the reduced test space; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; and based at least in part on the first set of execution results, A method is provided that includes determining that one or more test cases failed to execute; generating a set of new test cases from the selected failing test cases, where generating the set of new test cases includes, associated with each attribute in the selected failing test cases, generating a respective subset of the new test cases by, at least in part, changing a respective attribute value for the attribute in the selected failing test cases to a respective other candidate attribute value for the attribute that is not present in any of the one or more test cases that failed to execute; executing the new set of test cases to obtain a second set of execution results; and detecting and localizing defects based at least in part on the second set of execution results.

[0013] According to another aspect, a method is provided for detecting and localizing defects in a system under test (SUT), the method including: modeling inputs to the SUT as a set of attribute-value pairs; reducing a test space containing all possible combinations of attribute-value pairs to a reduced test space by eliminating one or more combinations of the attribute-value pairs based at least in part on corresponding probability values; generating an initial set of test vectors that provide full n-wise coverage of the reduced test space associated with the set of attribute-value pairs; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining that one or more test cases failed to execute based at least in part on the first set of execution results; generating a new set of test cases from selected failing test cases, where generating the set of new test cases includes modifying respective attribute values ​​in the selected failing test cases to generate each new test case; running the new set of test cases to obtain a second set of execution results; and detecting and localizing defects based at least in part on the second set of execution results.

[0014] Additional technical features and advantages are realized through the techniques of the present invention. Embodiments and aspects of the present invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, reference is made to the detailed description and drawings.

[0015] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief description of the drawings]

[0016] [Figure 1] 1 illustrates a schematic diagram of a computerized environment in accordance with one or more embodiments of the present invention. [Diagram 2]FIG. 2 is a block diagram illustrating modules for providing a testing infrastructure according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a schematic hybrid data flow / block diagram illustrating defect detection and localization using combinatorial test design (CTD) and generation of regression buckets of failing test cases that expose detected defects, according to one or more example embodiments of the present invention. [Figure 4] 1 illustrates an example set of CTD vectors, according to one or more embodiments of the present invention. [Diagram 5] 1 illustrates an example CTD vector, according to one or more embodiments of the present invention. [Figure 6] FIG. 1 is a process flow diagram of an example method for detecting and localizing n-wise defects using CTD techniques and generating regression buckets of failing test cases that expose the detected n-wise defects, in accordance with one or more example embodiments of the present invention. [Figure 7] 1 illustrates a flowchart for providing a deterministic ordering of tests run for a system under test in a test infrastructure in accordance with one or more embodiments of the present invention. [Figure 8] 1 illustrates a computer system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The figures shown herein are exemplary. There may be many variations to the figures or the operations described therein without departing from the spirit of the invention. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Also, the term "coupled" and variations thereof describe having a communication path between two elements, and do not imply a direct connection between elements with no intervening elements / connections between the elements. All of these variations are considered part of the specification.

[0018] Exemplary embodiments of the present invention relate, inter alia, to devices, systems, methods, computer readable media, techniques, and methodologies for performing defect detection and localization using combinatorial test design (CTD) techniques and for generating regression buckets of test cases that expose the detected defects. In exemplary embodiments of the present invention, the detected and localized defects occur in a system under test (SUT). The SUT may be a hardware or software system. Furthermore, in exemplary embodiments of the present invention, the defect detection and localization may be performed while adhering to architectural constraints on the SUT.

[0019] Additionally, coverage tools for testing the SUT provide a measure of how well the SUT is behaving during testing, thereby providing a level of assurance that the SUT is of high quality. There are several types of coverage known in the art, such as statement coverage, line coverage, condition coverage, path coverage, and method coverage. One further coverage method is functional coverage. Functional coverage is designed to measure the amount, portion, or similar metric of tests that have examined a given functional behavior. Once functional coverage is measured, quality assurance (QA) personnel may design additional tests to examine untested behaviors.

[0020] However, test infrastructure can be very costly, and without constant optimization of the product, tests, and environment, the cost per test cycle increases. As more tests are added, more test suites are worked with, and the complexity of support services increases along with the cost of each test cycle. The test environment, test dependencies, and test suites need to be managed with the same rigor as the product being shipped (i.e., the SUT). Thus, just as managing the test infrastructure is a technical challenge, so is efficiently scaling the test infrastructure. Note that scaling the test infrastructure does not mean a decrease in the quality of testing the SUT. Rather, the technical challenge is that test quality should improve along with scaling the test infrastructure.

[0021] Embodiments of the present invention address such technical challenges. In accordance with one or more embodiments of the present invention, a reduced set of test cases for the SUT is executed on the system. Note that the system is distinct from the SUT, and the SUT is executed with the system. The reduced set of test cases may be generated using combinatorial test design (CTD). Errors may be identified based on a subset of tests that fail from the reduced set. Further, in accordance with one or more embodiments of the present invention, inverse CTD is used to localize defects in the system.

[0022] According to one or more embodiments of the present invention, CTD is performed with respect to interaction requirements, such as pair-wise interactions, n-tuple interactions, or others. The test plan may provide sufficient coverage tasks to ensure that the interaction requirements are met. In some exemplary embodiments, the test plan is based on modeling the test space with a functional coverage model. The test space may represent inputs, scenarios, configurations, internal states of the application, or any other aspect that may need to be tested. The functional coverage model may include a set of functional attributes. In other embodiments of the present invention, the coverage model may include one of several types of coverage known in the art, such as statement coverage, line coverage, condition coverage, path coverage, method coverage, etc. The type of coverage does not affect the reduction of the test infrastructure techniques described herein.

[0023] Additionally, with respect to a test plan, a test space may define potential tests that may be executed on the SUT. A test, as opposed to a coverage task, may be associated with only one potential test. In some exemplary embodiments of the present invention, several different tests may have the same potential test.

[0024] In some exemplary embodiments of the present invention, the motivation for the CTD approach is that most errors depend on interactions between a small number of feature attributes. CTD algorithms may also address scalability issues. CTD algorithms may assume that the combinatorial space is too large to be represented explicitly, and they attempt to generate optimized test plans without explicitly enumerating all possible combinations, using various techniques. An example of such an algorithm may be the reduction to orthogonal arrays or covering arrays problems. Those techniques are limited in the type of restrictions they place on value combinations, or the type of interaction level required, and still run into scalability issues as the number of model variables increases.

[0025] In an example embodiment of the present invention, the input to the SUT is modeled as a set of attribute-value pairs. More specifically, the input to the SUT may be modeled as a set of attributes, each of which may take on one or more corresponding attribute values. In an example embodiment of the present invention, the entire Cartesian product space, which contains all possible combinations of attribute-value pairs, may be reduced to a smaller set of test vectors that provide full n-wise coverage of the entire test space. For example, if four different attributes A, B, C, and D are assumed to be modeled, and if it is further assumed that attribute A can take on four distinct values, attribute B can take on three distinct values, attribute C can take on three distinct values, and attribute D can take on two distinct values, then the total number of possible combinations of attribute-value pairs is 4×3×3×2=72. Thus, in this illustrative example, the entire Cartesian product space contains 72 different combinations of attribute-value pairs. As mentioned earlier, these 72 different combinations of attribute-value pairs can be reduced to a smaller set of combinations that still provide full n-wise coverage of the Cartesian product space. For example, referring to the same example introduced above, if full pairwise coverage is sought, the 72 different combinations can be reduced to 12 distinct combinations that together include every possible pairwise interaction of the attribute values. The reduced number of combinations required to provide n-wise coverage can increase logarithmically as n increases.

[0026] In an exemplary embodiment of the present invention, the entire Cartesian product space including all combinations of attribute-value pairs is reduced to a smaller set of CTD test vectors that provide complete n-wise coverage for a desired n. In complex hardware or software systems, the total number of attributes and corresponding candidate attribute values can be extremely large. In such cases, the total number of possible combinations of attribute values that make up the entire Cartesian product space can be astronomically large, thereby making it virtually impossible to test all possible combinations. By reducing the entire Cartesian product space to a significantly smaller number of CTD test vectors that still provide complete n-wise coverage, it becomes possible to detect any n-wise (or m-wise for m < n) defects without directly testing all possible combinations in the test space.

[0027] In an exemplary embodiment of the present invention, a binary decision diagram or the like can be used to perform the reduction and identify a reduced set of CTD vectors that provide complete n-wise coverage. In an exemplary embodiment of the present invention, each generated CTD test vector includes a unique combination of attribute values, and the set of CTD test vectors together includes all possible n-wise interactions of the attribute values. In particular, each CTD vector can have a dimension corresponding to the number of attributes being modeled. In that case, each component of the CTD vector is the respective attribute value of the corresponding attribute. However, the set of generated CTD vectors may not be unique. That is, there may be multiple different sets of CTD vectors, each of which provides complete n-wise coverage. In an exemplary embodiment of the present invention, the CTD vectors can be randomly selected while still ensuring complete n-wise coverage. In other exemplary embodiments, the initial set of CTD vectors can be selected according to certain criteria, such as considering increasing or decreasing the representation of specific attribute values in the set of CTD vectors.

[0028] In an exemplary embodiment of the present invention, once an initial set of CTD test vectors is generated, they are used to generate a corresponding set of test cases. For example, a set of CTD test vectors can be provided as input to a test case generation tool configured to generate a respective corresponding test case for each CTD vector. Each test case can be designed to test the interaction between a specific combination of attribute values included in the corresponding CTD vector.

[0029] In an exemplary embodiment of the present invention, the test cases are then executed. The execution of each test case results in either a successful execution result indicating that the combination of attribute values associated with the test case does not contain an n-wise (or m-wise if m < n) error, or a failed execution result indicating that the combination of attribute values associated with the test case contains an n-wise (or m-wise if m < n) error. In an exemplary embodiment of the present invention, a specific failed test case is then selected, and inverse combinatorics is applied to the failed test case to create a new set of test cases capable of manifesting bugs. In an exemplary embodiment of the present invention, applying inverse combinatorics to the selected failed test case includes generating a respective new test case for each attribute. Thus, the number of new test cases generated can be equal to the number of attributes. In an exemplary embodiment of the present invention, in each new test case, the attribute value of the corresponding attribute in the selected failed test case is changed to the attribute value for an attribute that does not exist in any of the failed test cases, and the respective attribute values for each of the other attributes are not changed from those that exist in the selected failed test case.

[0030] Those new test cases that result in successful execution results can then be evaluated to detect and localize n-wise (or m-wise where m < n) errors. In particular, the specific attribute value pairs that cause an error can be identified based on the new test cases where the execution is successful. The regression bucket of test cases can then be generated based on the combinations that cause an error in the attribute value pairs. More specifically, all possible combinations of attribute values including the attribute values determined to cause an error may be determined, and the corresponding regression bucket of test cases for testing these combinations can be output for use, for example, by a manual tester. In particular, in an exemplary embodiment of the present invention, all the test cases included in the regression bucket should fail at runtime until the bug is fixed, and after the bug is fixed, all the regression bucket test cases should pass (i.e., result in a successful execution result).

[0031] In an exemplary embodiment of the present invention, architectural restrictions can be applied to the SUT. The architectural restrictions can include any of a variety of restrictions on the input to the SUT. For example, as an example, the restriction may be that when a given attribute has a specific attribute value, one or more other attributes are excluded from having a certain attribute value. As another example, an architectural restriction may be that when a given attribute has a specific attribute value, one or more other attributes must have a certain attribute value. As yet another example, an architectural restriction may be that a new attribute is introduced only when a specific attribute has a specific attribute value. It should be understood that the above examples of architectural restrictions are merely illustrative and not exhaustive.

[0032] In an exemplary embodiment of the present invention, architectural constraints are taken into account before performing a reduction of the entire Cartesian space to an initial set of CTD vectors that provide the desired complete n-wise coverage. That is, in an exemplary embodiment of the present invention, certain combinations of attribute values ​​that violate any architectural constraints are first excluded from the Cartesian product space, and then a reduction to a set of CTD vectors that provide the desired complete n-wise coverage is performed. In this way, it can be ensured that combinations of attribute values ​​that violate architectural constraints are not included in the initial set of CTD vectors. Furthermore, in an exemplary embodiment of the present invention, when a selected test case corresponding to an initial CTD vector fails and is selected for expansion to obtain a new set of test cases designed to detect and localize n-wise errors, the expansion can be performed first without taking into account any architectural constraints. Any new test cases that violate architectural constraints can then be excluded. In this way, it can be ensured that the expansion provides as much coverage of the test space as possible in the neighborhood of the failing test cases. Additionally, in an exemplary embodiment of the present invention, a regression bucket of failing test cases may be first generated without regard to architectural restrictions, and then any test cases that violate architectural restrictions may be excluded from the regression bucket. That is, test cases corresponding to all possible combinations in the entire Cartesian product test space, including certain combinations of attribute values ​​that cause n-wise errors or errors of lower order, may be first generated and then reduced to exclude any test cases that violate the restrictions. Alternatively, the test cases to be included in the regression bucket may be selected from a reduced Cartesian space from which an initial set of CTD vectors is selected, in which case the regression bucket is generated to include only combinations of attribute values ​​that include a subset of attribute values ​​that cause errors and that do not violate any architectural restrictions.

[0033] Exemplary embodiments of the present invention include various technical features that produce technical effects that provide various improvements to computer technology. For example, exemplary embodiments of the present invention include the technical feature of inverse combinatorics to expand a set of test cases from a selected failing test case. This technical feature allows n-wise or lower order faults to be detected and localized in a single iteration of the expansion. This technical effect represents an improvement to debugging and fault detection computer technology because it automates the generation of additional test cases around a selected failing test case in a manner specifically designed to facilitate detection and localization of the fault. Thus, the automated generation of new test cases according to exemplary embodiments of the present invention allows for more efficient fault exposure and the use of fundamentally different methodologies than manual test case generation. Exemplary embodiments of the present invention also include the technical feature of generating a regression bucket of test cases that tests all possible combinations of attribute values ​​including a particular subset of attribute values ​​that cause n-wise faults. Thus, the regression bucket contains only test cases that all fail before the correction of the fault and all pass after the correction of the fault. Thus, the automatic generation of regression buckets of failing test cases in accordance with example embodiments of the present invention provides an improvement to computer technology by providing a tester or automated testing algorithm with a set of test cases, each of which is sure to fail, so that the tester or automated testing algorithm can use the regression buckets of test cases to verify that defects have been corrected when all test cases ultimately pass after a debug run.

[0034] A technical challenge with using tests generated using CTD is when generating test programs that require a specific sequence of action actions to be executed on the SUT. In this case, the technical challenge is to reduce the test space using the CTD technique. For example, consider that the following actions (A, B, and C) can occur on the SUT in any order, and that the test requires that each action occur before the next action occurs, with a set of breakpoints: Action A <run to completion, stop at breakpoint 1, stop at breakpoint 2, stop at breakpoint 3> Action B <run to completion, stop at breakpoint 4, stop at breakpoint 5> Action C <run to completion, stop at breakpoint 6>

[0035] Here, each of the actions A, B, and C may be an operation that may be performed by / on the SUT. Each action may be associated with at least one attribute, and a specific value may be assigned to the attribute for each test. Based on the assigned value, a desired outcome is expected from the test. If the desired outcome is obtained, the test is considered to pass, otherwise the test is considered to fail. For example, action A may be associated with attribute Aa, which is assigned value a1 in the first test, value a2 in the second test, etc. The value assigned to Aa may be different in each test. Values ​​a1, a2, etc. may be specified, for example, a1=run to completion, a2=stop at breakpoint 1, a3=stop at breakpoint 2, and a4=stop at breakpoint 3.

[0036] Now, with reference to actions A, B, and C above, consider that any of the following operational sequences of actions are required to test the SUT: ABC, ACB, BAC, BCA, CAB, and CBA. If, in any of these operational sequences, any of the actions do not run to completion, but stop at a breakpoint and then run to completion only after another action is taken, then the operational sequence could look like the following four examples: 1. A<stop at breakpoint 1>-B<run to completion>-A<run to completion>-C<run to completion> 2. A<stop at breakpoint 1>-B<run to completion>-C<run to completion>-A<run to completion> 3. A<stop at breakpoint 1>-B<run to completion>-A<stop at breakpoint 3>-C<run to completion>-A<run to completion> 4. A<stop at breakpoint 2> - B<stop at breakpoint 4> - A<run to completion> - C<stop at breakpoint 6> - B<run to completion> - C<run to completion>

[0037] Given our CTD technique for identifying attribute / value pairs to be used in combinatorial reduction, our CTD-based testing infrastructure creates multiple models that cover each case of a particular sequence of actions. Here, one model must be generated for the sequence of actions themselves, and then a separate second model must represent the execution state of each action (i.e., the values ​​assigned to the attributes). Furthermore, the models must be duplicated to identify the test space that should be used. This is not only time consuming, but also error prone.

[0038] Embodiments of the present invention address such technical problems. According to one or more embodiments of the present invention, the technical problem is addressed by identifying attributes that require tests with a specific order of operations, and further identifying constraints (e.g., action A must occur before action B). Further, a total Cartesian product of unique values ​​of the attributes is determined. Combinatorial reduction and test case reduction are then applied to the set of tests with the Cartesian product to remove any redundant tests. In one or more embodiments of the present invention, identifying the attributes and constraints, and generating the Cartesian product may be performed in a single step, i.e., with a single model.

[0039] In one or more embodiments of the invention, information about the ordering may be stored using a Markov chain that is used as a payload to a system performing CTD reduction and expansion to generate a test program for the SUT. A Markov chain is a probabilistic model that describes a sequence of possible events where the probability of each event depends only on the state reached in the previous event.

[0040] Reference is now made to Figure 1, which illustrates a schematic diagram of a computerized environment in accordance with an embodiment of the present invention. The computerized environment 100 may include one or more computerized tools. It should be noted that the diagram is one possible example of a block diagram and that some components may not be shown for clarity.

[0041] In one or more embodiments of the invention, a user 110, such as a developer, a QA staff member, a tester, a designer, a verification engineer, etc., may interact with the computerized environment 100. The user 110 may utilize a man-machine interface (MMI) 112, such as a terminal, a display, a keyboard, an input device, etc.

[0042] In some exemplary embodiments of the present invention, the coverage model definition tool 102 may be utilized to define a test coverage model, such as a functional coverage model. In some exemplary embodiments of the present invention, a user 110 may define attributes to be tested, e.g., functional attributes if functional coverage is being tested. In some exemplary embodiments of the present invention, a similar tool may be utilized to define a test space. In some exemplary embodiments, a coverage model may be adapted to be utilized as a test model.

[0043] In some exemplary embodiments of the present invention, the test execution engine 108 may be utilized to test the SUT. It should be noted that the SUT may be hardware, firmware, software, a combination thereof, or any other type of computerized device. The test execution engine 108 may be a simulation-based verification tool, a test generation platform, or the like. The test execution engine 108 may be operatively coupled to the test planning tool 106 and configured to execute tests according to a test plan. In some exemplary embodiments, the test planning tool 106 may provide tests for the test execution engine 108 to execute. It should be noted that dynamic verification is a broader concept than testing of the SUT and further includes test plans, coverage analysis, and the like. The test execution engine 108 provides only one aspect of the overall behavioral scope that may be executed during dynamic verification and should not be used to interpret the term "dynamic verification" more narrowly.

[0044] In some example embodiments of the invention, the coverage analysis tool 104 is configured to measure coverage of a test space for the SUT based on the dynamic verification performed by the test execution engine 108. For example, the coverage analysis tool 104 may be a functional coverage analysis tool. The coverage analysis tool 104 provides coverage measurements, such as a coverage test space or a portion of a defined test plan, that indicate coverage tasks that were covered during the dynamic verification performed by the test execution engine 108. The user 110 may review the coverage measurements and / or the list of covered tasks.

[0045] In some exemplary embodiments of the present invention, the test planning tool 106 may define a test plan to be covered. In some exemplary embodiments of the present invention, the test plan may be a set of coverage tasks to be covered. In some exemplary embodiments of the present invention, the test planning tool 106 may provide tests that are deemed to cover the test plan based on test benchmarks, etc., stored in a data store that includes tests known / presumed to cover certain aspects of the SUT. As another example, the test planning tool 106 may be configured to generate tests that cover the coverage tasks. The user 110 may review the test plan, selected tests, etc. In some exemplary embodiments of the present invention, the user 110 may provide parameters for the test planning tool 106 to use in determining the objectives of the test plan, such as the desired interaction level. An embodiment of the present invention allows for the removal of any redundancies in the generated tests.

[0046] Although the illustration of FIG. 1 is described with particular components including a coverage model definition tool 102, a coverage analysis tool 104, a test planning tool 106, and a test execution engine 108, embodiments of the invention are not limited to these components or system configurations and may be practiced with other system configurations employing fewer or additional components.

[0047] 2 is a block diagram illustrating modules that provide a test infrastructure according to an embodiment of the present invention. More specifically, the infrastructure includes a test generator 208. The test generator 208 accesses a test infrastructure, such as a repository of test cases 202, which stores a suite of test cases that can be used to verify the correctness of the SUT 214. Each test case specifies an input to be applied to the SUT 214 and an expected response to be returned in response to this input (to indicate its correct behavior). Typically, the test cases are organized into sets (test suites), e.g., one test suite for each different component of the SUT 214.

[0048] The test generator 208 creates an execution bucket for each execution of a test on the SUT 214. A bucket specifies the actions to be taken to execute the desired test cases in a machine-readable language (e.g., XML-based). In particular, in case of exhaustive testing, all of the available test cases are executed on each component of the SUT 214, whereas in case of regression testing, the execution is limited to a selected subset of test cases. The buckets so obtained can be saved to a file.

[0049] Test generator 208 receives event ordering information 220 in accordance with one or more embodiments of the present invention. Event ordering information 220 indicates a particular order for a set of attributes / actions described herein. For example, event ordering information 220 indicates a predetermined order in which attributes are tested. Event ordering information 220 may be provided in the form of a Markov chain in one or more embodiments of the present invention. Test generator 208 orders the tests in buckets 212 based on event ordering information 220.

[0050] The test execution engine 108 controls the execution of the buckets read from the file. For each test case of the bucket, this involves the application of the corresponding input to the SUT 214. In response, the SUT 214 returns a corresponding output to the test execution engine 108. The test execution engine 108 determines the outcome of the test case by comparing the output with the corresponding expected response (e.g., extracted from the file). The result of the test case (i.e., positive if the two values ​​match, negative otherwise) is saved in a log. For example, this can be realized using the Standard Test Tracking Tool (TTT). The (current) execution results of the tests are available in the log for their analysis.

[0051] For large, complex SUTs 214, the test case infrastructure 202 may contain a large number of duplicate test actions / test cases. According to an embodiment of the present invention, one technique developed to address these challenges combines the use of a fingerprint repository 206 with a store of information that includes a set of fingerprints corresponding to multiple regression tests stored in the test repository 202. In one embodiment, the fingerprints are generated by a fingerprint generator 204. For example, the fingerprint repository 206 may include fingerprints for most, if not all, of the tests stored in the test repository 202, as well as references to where copies of the tests are located throughout the test infrastructure, similar to an index in a book. Each fingerprint uniquely identifies a particular code path covered by the corresponding test case. In this manner, the system may identify fingerprints (and corresponding tests) that are redundant and possibly duplicate.

[0052] According to embodiments of the invention, processing of the regression test results 216 generated by the test generator 208 and executed by the test execution engine 108 may include determining the code paths traversed during execution of each generated regression test, and generating a fingerprint for each executed regression test based at least in part on the code paths. In some embodiments of the invention, these steps may be performed by the fingerprint generator 204.

[0053] The redundancy analyzer 218 compares the fingerprints corresponding to all test cases generated by the test generator 208 with the fingerprints stored in the fingerprint repository 206. The redundancy analyzer 218 provides a list of regression test cases generated by the test generator 208 that have fingerprints that match one or more fingerprints stored in the fingerprint repository 206. This information is used to select and discard duplicate test cases from the test repository 202 (as described below).

[0054] Figure 3 is a schematic hybrid data flow / block diagram illustrating defect detection and localization using CTD techniques and generating regression buckets of failing test cases that expose the detected defects, according to one or more example embodiments of the present invention. Figure 6 is a process flow diagram of an example method 600 for detecting and localizing n-wise defects using CTD techniques and generating regression buckets of failing test cases that expose the detected n-wise defects, according to one or more example embodiments of the present invention. Figure 6 is described in conjunction with Figures 1, 2, and 3.

[0055] Exemplary methods according to example embodiments of the present invention, as well as corresponding data structures (e.g., modules, units, and other such components) for performing the methods, are described herein. It should be noted that each operation of the method 600 shown in FIG. 6 and / or the method 700 shown in FIG. 7 may be performed by one or more of the modules, etc. described herein. These modules may be implemented in any combination of hardware, software, or firmware, or combinations thereof, described herein. In certain example embodiments, one or more of these modules may be implemented, at least in part, as software and / or firmware modules that include computer-executable instructions that, when executed by a processing circuit, cause one or more operations to be performed. A system or device described herein as configured to perform an example embodiment may include one or more processing circuits, each of which may include one or more processing units or nodes. The computer-executable instructions may include computer-executable program code that, when executed by a processing unit, may be accessed and processed to provide output data, such that input data included in or referenced by the computer-executable program code is accessed and processed to provide output data.

[0056] 6 in conjunction with FIG. 3, in an example embodiment of the invention, at block 602 of a method 600, input to the SUT 214 is modeled as a set of attribute-value pairs 302. Any number of attributes may be used to model the SUT input, and each attribute may have any number of candidate attribute values. At block 604, computer-executable instructions of one or more n-wise coverage CTD vector generation modules 304 are executed to generate an initial set 306 of CTD vectors that provide n-wise coverage of the entire Cartesian product space associated with the set of attribute-value pairs 302.

[0057] In particular, the entire Cartesian product space, which contains all possible combinations of attribute-value pairs 302, is reduced to a smaller set of CTD test vectors 306 that provide full n-wise coverage of the entire test space. In an exemplary embodiment of the present invention, the full n-wise coverage provided by the set of CTD vectors 306 may be full pairwise coverage. For example, if three attributes are assumed to be modeled, namely, the "name" attribute, the "color" attribute, and the "shape" attribute shown in FIG. 4, and if it is further assumed that the "name" attribute can take on four distinct attributes (Dale, Rachel, Andrew, and Ryan), the "color" attribute can take on two distinct attributes (green, blue), and the "shape" attribute can take on three distinct attributes (circle, square, triangle), then the total number of possible combinations of attribute-value pairs is 4×3×2=24. Thus, in this illustrative example, the entire Cartesian product space would contain 24 different combinations of attribute-value pairs.

[0058] The 24 different combinations of attribute-value pairs can be reduced to a smaller set of combinations (i.e., set of CTD vectors 306) that still provide complete n-wise coverage of the Cartesian product space. For example, if complete pairwise coverage is desired, the 24 different combinations can be reduced to 12 distinct combinations that together include all possible pairwise interactions of the attribute values. An example set 400 of CTD vectors is shown in FIG. 4. The example set 400 of CTD vectors includes all pairwise interactions between attribute values ​​of the attributes "name", "color", and "shape".

[0059] A binary decision diagram or the like may be used to perform the reduction and identify a reduced set 306 of CTD vectors that provides full n-wise coverage. While each CTD vector in the set 306 of CTD vectors includes a unique combination of attribute values, the set 306 of CTD vectors itself may not be unique. That is, there may be multiple different sets of CTD vectors, each of which provides full n-wise coverage. For example, while FIG. 4 illustrates an example set 400 of CTD vectors for the example attributes and attribute values ​​described, it should be understood that alternative sets of CTD vectors including different combinations of attribute values ​​may also independently provide full n-wise coverage. It should be further understood that while the example set 400 of CTD vectors provides full pairwise coverage and partial 3-wise coverage, a greater number of CTD vectors are required to provide full 3-wise coverage. More generally, as n increases, the number of CTD vectors required to provide full n-wise coverage increases logarithmically with n.

[0060] 5 illustrates an example CTD vector 500 of a type that may be included in the set of CTD vectors 306. The example CTD vector 500 includes a number of attributes 502. As discussed above, the attributes 502 may be used to model input to the SUT 214. The attributes 502 may be associated with an attribute value 504. In particular, each attribute 502 may have a corresponding attribute value 504, which may be one of one or more candidate attribute values ​​that the attribute may take.

[0061] Further, according to one or more embodiments of the present invention, the n-wise coverage CTD vector generation module 304 reduces the entire Cartesian product space associated with the set of attribute-value pairs 302 by identifying and excluding combinations that do not adhere to the event ordering information 220. In one or more embodiments of the present invention, the event ordering information 220 is used to generate architectural constraints associated with the SUT 214. After the entire Cartesian product space has been reduced to exclude combinations that do not satisfy the architectural constraints for the SUT 214, the n-wise coverage CTD vector generation module 304 generates a further reduced initial set 306 of CTD vectors. In an example embodiment of the present invention, each CTD vector 306 is selected only from the reduced test space that excludes invalid combinations of attribute values ​​that violate the architectural constraints for the SUT 214. Here, the invalid combinations include sequences of attributes that do not match the event ordering information 220.

[0062] As a result, an architectural constraint is generated for the Cartesian product space that a particular first attribute must be used before a particular second attribute according to the event ordering information 220. For example, action A must be executed before action B in all cases. The architectural constraint may additionally include any of a variety of constraints on the inputs to the SUT 214. For example, an example constraint may be that if a given attribute has a particular attribute value, then one or more other attributes are excluded from having a certain attribute value. Another example architectural constraint may be that if a given attribute has a particular attribute value, then one or more other attributes must have a certain attribute value. Yet another example architectural constraint may be that a new attribute is introduced only if a particular attribute has a particular attribute value. It should be understood that the above examples of architectural constraints are merely illustrative and not exhaustive.

[0063] In an exemplary embodiment of the invention, the architectural constraints are taken into account in the method 600 before performing a reduction of the entire Cartesian space to an initial set of CTD vectors 306 that provides the desired complete n-wise coverage. That is, certain combinations of attribute values ​​that violate any architectural constraints are first excluded from the Cartesian product space, and then a reduction is performed to a set of CTD vectors 306 that provides the desired complete n-wise coverage. In this way, it can be ensured that combinations of attribute values ​​that violate architectural constraints are not included in the initial set of CTD vectors 306.

[0064] Referring again to FIG. 6, at block 606 of method 600, in an example embodiment of the present invention, the test case generation module 208 may be executed to generate a corresponding set of test cases 202 from the initial set of CTD test vectors 306, which are then executed by the test case execution module 108 to produce an execution result (pass or fail) for each test case.

[0065] For example, the set of CTD test vectors 306 may be provided as input to a test case generation tool 208 that generates a respective corresponding test case for each CTD vector. Each test case in the set of test cases 202 tests interactions between a particular combination of attribute values ​​included in a corresponding CTD vector of the set of CTD vectors 306. It should be understood that the sets of CTD vectors and their corresponding test cases may be described and / or illustrated interchangeably at this point. For example, the example set 400 of CTD vectors shown in FIG. 4 may be considered interchangeably as a corresponding set of test cases that test the particular combination of attribute values ​​represented by the set 400 of CTD vectors.

[0066] In addition, test generator 208 receives event sequencing information 220 in accordance with one or more embodiments of the present invention. Event sequencing information 220 indicates a specific order for a set of attributes / operations described herein. Event sequencing information 220 may be provided in the form of a Markov chain in one or more embodiments of the present invention. Test generator 208 orders the tests within bucket 212 based on event sequencing information 220.

[0067] In block 608 of method 600, computer-executable instructions of test case execution module 108 are executed to determine whether any test case within the set of test cases 202 has failed. In an exemplary embodiment of the present invention, the execution of each test case 202 results in either a successful execution result indicating that the combination of attribute values included in the corresponding CTD vector 306 does not include an n-wise (or m-wise where m < n) error, or a failed execution result indicating that the combination of attribute values in the corresponding CTD vector 306 includes an n-wise (or m-wise where m < n) error.

[0068] With reference to the example shown in FIG. 4, the test cases corresponding to the example set of CTD vectors 400 are executed in block 606 to yield respective execution results for each test case. In particular, two test cases 402 and 404 are illustratively shown in FIG. 4 as yielding failures. Failing test case 402 tests the following combination of attribute values: Dale, Blue, Triangle, which correspond to the name attribute, color attribute, and shape attribute, respectively. Failing test case 404 tests the following combination of attribute values: Dale, Blue, Circle, which correspond to the name attribute, color attribute, and shape attribute, respectively. Although "Dale" and "Blue" are present in both the CTD vector corresponding to failing test case 402 and the CTD vector corresponding to failing test case 404, it is unclear at this stage of the process whether "Dale" and "Blue" are generating pairwise errors, whether "Dale" and ("Triangle" or "Circle") are generating pairwise errors, or whether "Blue" and ("Triangle" or "Circle") are generating pairwise errors. Subsequent operations of method 600 may utilize inverse combinatorics to expand test cases around the selected failing test case to detect and localize the particular combination of attribute values ​​that is causing the n-wise error (the pairwise error in the example shown in FIG. 4).

[0069] In response to a negative determination at block 608, the method 600 may end because none of the initial set of test cases 202 failed. Because the initial set of CTD vectors 306 (based on which the set of test cases 202 is generated) provided full n-wise coverage, it may be guaranteed that there are no n-wise errors or errors of lower order. However, successful execution of all test cases in the initial set of test cases 202 does not guarantee that there are no higher order errors (k-wise, where k>n).

[0070] On the other hand, in response to a positive determination at block 608 indicating that execution of the set of test cases 202 results in one or more failing test cases 314, the method 600 includes executing and selecting, using the inverse CTD test case generation module 316, particular failing test cases 314 and applying inverse combinatorics to the selected failing test cases 314 to create a new set of test cases 318 that can be used to detect and localize combinations of attribute values ​​that are causing n-wise errors or lower order errors.

[0071] In an exemplary embodiment of the invention, applying the inverse combinatorial logic to the selected failing test cases 314 includes generating a respective new test case 318 for each attribute. Thus, the number of new test cases generated may be equal to the number of attributes. In an exemplary embodiment, in each new test case 318, the attribute value in the selected failing test case 314 for the corresponding attribute is changed to the attribute value for the attribute that is not present in any other failing test case, and the respective attribute value for each other attribute is left unchanged from that present in the selected failing test case 314.

[0072] Referring again to the example shown in FIG. 4, assuming that a failing test case 402 is selected in block 610, an example set of new test cases 406 may be generated. In an example embodiment of the present invention, each example new test case 406 may be generated by changing the attribute value of the corresponding attribute in the selected failing test case 402 to a different value not present in any other failing test case. For example, a first new test case corresponding to the CTD vector <Andrew, Blue, Triangle> is obtained by changing the attribute value of the “Name” attribute in the failing test case 402 from “Dale” to “Andrew” while keeping the attribute values ​​for the other attributes the same as in the failing test case 402. Similarly, a second new test case corresponding to the CTD vector <Dale, Green, Triangle> is obtained by changing the attribute value of the “Color” attribute in the failing test case 402 from “Blue” to “Green” while keeping the attribute values ​​for the other attributes the same as in the failing test case 402. Finally, a third new test case corresponding to the CTD vector <Dale, Blue, Square> is obtained by changing the attribute value of the “shape” attribute in the failing test case 402 from “Triangle” to “Square”, while keeping the attribute values ​​for the other attributes the same as in the failing test case 402.

[0073] It should be appreciated that each attribute value that is modified to obtain each new test case 406 is not explicitly present in the selected failing test case 402. Moreover, each modified attribute value is not present in any other failing test case (e.g., failing test case 404). Specifically, the modified attribute value "Andrew" for the first new test case is not present in any failing test case, the modified attribute value "Green" for the second new test case is not present in any failing test case, and the modified attribute value "Square" for the third new test case is not present in any failing test case.

[0074] After generating a new set of test cases 318 by running the inverse combination theory on the selected failed test case 314, the test case execution module 108 is used to execute the new test cases 318 in block 612 of method 600. Further, in block 614, one or more n-wise error localization modules 322 detect and localize n-wise errors or errors of lower order based on the attributes within the selected failed test case 314 and their corresponding original failure attribute values that result in new test cases that pass when changed. In particular, the computer-executable instructions of the n-wise error localization module 322 are executed to detect and localize n-wise errors or errors of lower order based on the execution results 320 for the new set of test cases 318 and based on the new test cases that result in successful execution results. As used herein, an error of lower order refers to an m-wise error where m < n, assuming complete n-wise coverage by the set of CTD vectors 306.

[0075] Referring again to the example shown in FIG. 4 assuming pairwise error detection and localization, the execution of the new set of test cases 406 results in the first new test case corresponding to the CTD vector <Andrew, blue, triangle> passing, the second new test case corresponding to the CTD vector <Dale, green, triangle> passing, and the third new test case corresponding to the CTD vector <Dale, blue, square> failing. In the exemplary embodiment, since changing the attribute value from "triangle" in the failed test 402 to "square" in the third new test case results in a continued failure result, the n-wise error localization module 322 determines that the "shape" attribute is not a contributing factor to the pairwise error based on the failure execution result for the third new test case.

[0076] On the other hand, the n-wise error localization module 322 may determine that an attribute whose attribute value was changed to obtain a new test case whose execution passed contributes to the pairwise error. Specifically, in an example embodiment of the present invention, based on the successful execution result of the first new test case corresponding to the CTD vector <Andrew, blue, triangle>, the n-wise error localization module 322 determines that the "name" attribute and the original failing attribute value "Dale" contribute to the pairwise error. Similarly, in an example embodiment, based on the successful execution result of the second new test case corresponding to the CTD vector <Dale, green, triangle>, the n-wise error localization module 322 determines that the "color" attribute and the original failing attribute value "blue" also contribute to the pairwise error.

[0077] Thus, based on the execution results 320 for the set of new test cases 318, the n-wise error localization module 322 may determine that the attributes "name" and "color," having attribute values ​​"dale" and "blue," respectively, in the example shown in Figure 4, are the source of the pair-wise error. More generally, based on evaluation of the execution results 320 for the set of new cases 318, the n-wise error localization module 322 may determine the particular attribute-value pairs that cause the n-wise error or a lower degree error, or more specifically, the original attribute values ​​that were changed in the failing test cases that were selected to obtain the new test cases that pass.

[0078] The example shown in FIG. 4 assumes an initial set of CTD vectors 400 that provides full pairwise coverage, in which case pairwise errors or lower order errors (e.g., a single attribute value causing an error) may be detected and localized in just one pass of applying reverse combinatorics to the selected failing test cases to obtain a new set of test cases 318. In general, if the initial set of CTD vectors 306 provides n-wise coverage, just one pass of applying reverse combinatorics to the selected failing test cases will reveal n-wise errors or lower order errors. Although k-wise errors for k>n may be detectable using an initial set of CTD vectors that provides full n-wise coverage, this cannot be guaranteed because the initial set of CTD vectors 306 does not include all k-wise interactions. However, an initial set of CTD vectors 306 that provides complete n-wise coverage (e.g., complete pairwise coverage) may also provide some k-wise coverage (e.g., 3-wise coverage) where k>n, and thus, based on the particular CTD vectors selected, method 600 may reveal k-wise errors in just one pass or after multiple passes that provide reverse combinatorial logic for the selected failing test case.

[0079] 6, at block 616 of method 600, computer-executable instructions of one or more regression bucket generation modules 326 are executed to generate and output a regression bucket 212 of failing test cases that includes all possible combinations in the Cartesian product space that include the original failing attribute-value pairs. In an example embodiment of the invention, the regression bucket generation module 326 receives as input an indication of the error-generating subset 324 of attribute-value pairs and determines and outputs a regression bucket 212 that includes only test cases that correspond to all possible combinations in the Cartesian product space that include the particular attribute value that is causing the detected error.

[0080] Referring again to the example of FIG. 4, the regression bucket generation module 326 receives as input the attributes “name” and “color” and the corresponding attribute values ​​“Dale” and “Blue” that are causing the pair-wise error, determines all attribute value combinations in the Cartesian product space that contain “Dale” and “Blue”, and populates the regression bucket 212 with test cases that correspond to all of these combinations. In this example, the regression bucket 212 includes the following test cases: (Dale, Blue, Triangle), (Dale, Blue, Circle), and (Dale, Blue, Square). Each of the test cases in the regression bucket 212 includes the attribute values ​​“Dale” and “Blue” for the attributes “name” and “Color”, respectively, that together are causing the detected pair-wise error, and each test case in the regression bucket 212 is guaranteed to fail. In an example embodiment of the invention, the regression bucket 212 may be output for use by a manual tester or another automated debugging algorithm. The regression bucket 212 of failing test cases may be used to verify whether a detected error has been corrected because all tests contained in the regression bucket 212 will pass only when the error is corrected. Thus, if any test cases in the regression bucket 212 continue to fail despite modifications made to the SUT 214 in an attempt to correct the error, this indicates that the error has not been fully resolved.

[0081] Another technical challenge in a test infrastructure is that when validating the quality of the SUT214, diverse and robust test sequences must be used to adequately cover as many scenarios as possible to which the SUT214 may be subjected. Some scenarios require a specific order of events to occur to ensure that a given service or path is performed as described herein. Some scenarios may not require a specific order of events to occur. When these scenarios are tested, the technical challenge is to introduce additional randomness into the test system to represent the production environment in which the SUT214 is likely to operate.

[0082] One or more embodiments of the present invention address such technical challenges through the use of Markov chains, a mathematical construct used to model state changes in the SUT 214 based on the random probability of certain events occurring while the SUT 214 is in a given state. The probability of an event occurring in any given state may be adjusted to control the degree or randomness to which the SUT 214 is subjected by the test cases executed by the test infrastructure.

[0083] 7 illustrates a flowchart for providing a deterministic ordering of tests to execute for a SUT 214 in a test infrastructure, in accordance with one or more embodiments of the present invention. The illustrated method 700 includes modeling attribute-value pairs for testing, at block 702. The modeling is similar to that performed in method 600. The method further includes assigning probability factors to the attribute-value pairs, at block 704. In one or more embodiments of the present invention, each attribute-value pair is assigned a respective probability value.

[0084] The probability value indicates the probability of an attribute value being selected to generate a test case. Alternatively, or in addition, the probability value indicates the probability of a test case associated with an attribute value being selected to be executed in the test bucket 212. The probability value may be manually configured by a test engineer to facilitate a higher (or lower) chance of a particular attribute value being tested. In an example, setting the probability value to "1" ensures that the attribute-value pair will be tested. On the other hand, setting the probability value to "0" ensures that the attribute-value pair will not be tested.

[0085] In one or more embodiments of the invention, at block 706, the selected attribute-value pairs and their respective assigned probability values ​​are stored in the form of a hash value. In other words, the hash represents the attribute-value pairs and their assigned probability values. In one or more embodiments of the invention, a unique hash is generated for each of the attribute-value pairs and their respective probability values. In one or more embodiments of the invention, operations such as prime coefficients, multiplication, etc. may be used to ensure unique hashes. The attribute-value pairs and corresponding probability values ​​are then stored as a hash table, facilitating the processor to perform reduction to identify a reduced set 306 of CTD vectors that provide complete n-wise coverage using techniques such as binary decision diagrams. Binary decision diagrams may be applied as described elsewhere herein.

[0086] At block 708 of the method 700, the n-wise coverage CTD vector generation module 304 reduces the overall Cartesian product space associated with the set of attribute-value pairs 302 by identifying and eliminating combinations that do not respect architectural constraints associated with the SUT 214. For example, event ordering information 220, constraints on specific values ​​of attributes, and other types of architectural constraints may be associated with one or more of the attribute-value pairs.

[0087] After the entire Cartesian product space has been reduced to exclude combinations that do not satisfy the architectural constraints for the SUT 214, the n-wise coverage CTD vector generation module 304 generates a further reduced initial set 306 of CTD vectors at block 710 of the method 700. In an example embodiment, each CTD vector 306 is selected only from the reduced test space that excludes invalid combinations of attribute values ​​that violate the architectural constraints for the SUT 214. The CTD vectors 306 are selected based on probability values ​​assigned to the attribute-value pairs. To facilitate the probability values ​​having an impact on the selection of the attribute-value pairs, a Markov chain is used to represent each selection as a state. The mathematical construct of the Markov chain then models state changes based on the probability of a particular event (i.e., the selection of a particular attribute-value pair) occurring while the system is in a desired state (i.e., the currently selected attribute-value pair).

[0088] At block 712 of method 700, an initial set of test cases 202 are generated from the initial set of CTD vectors 306 and executed, similar to block 606 of method 600. At block 714, a determination is made as to whether any of the test cases 202 have failed, similar to the determination at block 608 of method 600. In response to a negative determination at block 714, the method 700 ends because none of the test cases 202 produced a failing execution result. However, in response to a positive determination at block 714, the method 700 proceeds to block 716, where the inverse CTD test case generation module 316 generates new test cases 318 from the selected failing test cases.

[0089] In one or more embodiments of the invention, a respective set of new test cases is generated for each attribute by changing the attribute value for that attribute in the selected failing test cases to a respective other candidate attribute value for that attribute that is not present in any of the failing test cases. For example, a single new test case is generated for each attribute in the selected failing test cases by changing the attribute value for the attribute to another attribute value that the attribute is suitable to have and that is not present in any of the failing test cases. Although there may be multiple attribute values ​​for an attribute that are not present in any of the failing test cases, each new test case generated tests only one such attribute value, such that in this technique of generating new tests, each new test case may be generated by simply varying the corresponding attribute value in the selected failing test cases to one other attribute value that is not present in any of the failing test cases.

[0090] Alternatively, in one or more embodiments of the present invention, for each attribute in the selected failing test cases, a new test case is generated for each possible attribute value for the attribute that is not present in any of the failing test cases. Thus, according to this second technique, if there are multiple attribute values ​​for an attribute that are not present in any of the failing test cases, a new test case is generated for each such attribute value.

[0091] In the example shown in Fig. 4, for example, inverse combinatorial analysis is applied to the failing test cases 402 to obtain a set of new test cases 406 according to the first technique described above. That is, a single new test case is shown to be generated for each attribute in the selected failing test cases 402 by changing the corresponding attribute value for the attribute in the selected failing test cases 402 to another candidate attribute value that is not present in any of the failing test cases. On the other hand, if the set of new test cases 406 is generated according to the second technique described above, the attribute values ​​"Ryan" and "Rachel" are also not present in any of the failing test cases, so the set of new test cases 406 includes the following additional test cases: (Ryan, blue, triangle) and (Rachel, blue, triangle).

[0092] Furthermore, the inverse CTD test case generation module 316 excludes any of the new test cases that violate an architectural restriction for the SUT 214. The set of new test cases 318 obtained by the inverse combinatorics augmentation is then reduced to a smaller set by excluding (and not excluding) one or more new test cases that violate one or more architectural restrictions. In particular, when a selected test case corresponding to the initial CTD vector 306 is selected for augmentation to obtain a set of new test cases 318 that are both failed and designed to detect and localize n-wise errors, the augmentation may be performed first without taking into account any architectural restrictions. Then, any new test cases generated from the augmentation that violate an architectural restriction may be excluded from the set of new test cases 318. In this way, it may be ensured that the augmentation provides as much coverage of the test space as possible in the neighborhood of the failing test cases.

[0093] Additionally, method 700 includes, at block 718, executing a reduced set of new tests 318, similar to block 612 of method 600. At block 720 of method 700, n-wise errors are detected and localized based at least in part on attributes and their corresponding attribute values ​​in the selected failing test cases that, when modified, produce new test cases that pass, similar to block 614 of method 600. Additionally, at block 722, a regression bucket 212 of failing test cases is generated, similar to block 616 of method 600. Each test case in the regression bucket 212 corresponds to a particular combination of attribute values ​​that comprises a fault-generating subset 324 of attribute-value pairs, such that the regression bucket 212 covers all possible combinations of attribute-value pairs that adhere to the architectural constraints for the SUT 214 and include failing-generating attribute values. Additionally, at block 724, any failing test cases that violate the architectural constraints may be removed from the regression bucket 212.

[0094] As a result, the method 700 facilitates that the regression bucket 212 of failing test cases is generated without regard to the architectural constraints, and then any test cases that violate the architectural constraints may be excluded from the regression bucket 212. That is, test cases corresponding to all possible combinations in the entire Cartesian product test space, including certain combinations of attribute values ​​that cause n-wise errors or errors of lower order, may be first generated and included in the regression bucket 212, and then the regression bucket 212 may be reduced to exclude any test cases that violate the constraints. In an alternative example embodiment, the test cases included in the regression bucket 212 may be selected from a reduced Cartesian space from which an initial set of CTD vectors is selected, where the regression bucket 212 is generated to include only combinations of attribute values ​​that include a subset of attribute values ​​that cause errors and that do not violate any architectural constraints.

[0095] In addition, the tests selected are based on an adjusted probability value for each attribute-value pair. Thus, a first attribute-value pair having a higher probability value has a higher chance of being tested than a second attribute-value pair having a lower probability value than the first attribute-value pair. Test engineers can control test selection in the CTD-based test infrastructure accordingly.

[0096] Referring now to FIG. 8, a computer system 800 is generally illustrated in accordance with an embodiment of the present invention. The computer system 800 may be an electronic computer framework that includes and / or utilizes any number and combination of computing devices and networks utilizing various communication technologies as described herein. The computer system 800 may be easily scalable, extensible, and modular, with the ability to change to different services or reconfigure some features independently of others. The computer system 800 may be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smartphone. In some examples, the computer system 800 may be a cloud computing node. The computer system 800 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system 800 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0097] As shown in FIG. 8, computer system 800 has one or more central processing units (CPUs) 801a, 801b, 801c, etc. (collectively or generally referred to as processors 801). Processor 801 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processor 801, also referred to as a processing circuit, is coupled to system memory 803 and various other components via system bus 802. System memory 803 may include read only memory (ROM) 804 and random access memory (RAM) 805. ROM 804 is coupled to system bus 802 and may include a basic input / output system (BIOS), which controls certain basic functions of computer system 800. RAM is a read-write memory coupled to system bus 802 for use by processor 801. System memory 803 provides temporary memory space for the operation of the above-mentioned instructions during operation. The system memory 803 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0098] Computer system 800 includes an input / output (I / O) adapter 806 and a communications adapter 807 coupled to a system bus 802. I / O adapter 806 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 808 and / or any other similar components. I / O adapter 806 and hard disk 808 are collectively referred to herein as mass storage 810.

[0099] Software 811 for execution on computer system 800 may be stored in mass storage 810. Mass storage 810 is an example of a tangible storage medium readable by processor 801. Software 811 is stored as instructions for execution by processor 801 to operate computer system 800 as described herein below with respect to various figures. Examples of computer program products and the execution of such instructions are discussed in more detail herein. Communications adapter 807 interconnects system bus 802 with network 812, which may be an external network, enabling computer system 800 to communicate with other such systems. In one embodiment, system memory 803 and a portion of mass storage 810 collectively store an operating system, which may be any suitable operating system, such as IBM® Corporation's z / OS® or AIX® operating systems, for coordinating the functions of the various components illustrated in FIG. 8.

[0100] Additional input / output devices are shown connected to the system bus 802 via a display adapter 815 and an interface adapter 816. In one embodiment, adapters 806, 807, 815, and 816 may be connected to one or more I / O buses that are connected to the system bus 802 via an intermediate bus bridge (not shown). A display 819 (e.g., a screen or display monitor) is connected to the system bus 802 by a display adapter 815, which may include a graphics controller for improving performance of graphics-intensive applications and a video controller. A keyboard 821, a mouse 822, speakers 823, etc. may be interconnected to the system bus 802 via an interface adapter 816. The interface adapter 816 may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Thus, as configured in Figure 8, computer system 800 includes processing capabilities in the form of a processor 801, storage capabilities including system memory 803 and mass storage 810, input means such as keyboard 821 and mouse 822, and output capabilities including speakers 823 and display 819.

[0101] In some embodiments, the communications adapter 807 may transmit data using any suitable interface or protocol, such as an Internet Small Computer System Interface, among others. The network 812 may be a cellular network, a wireless network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device may connect to the computer system 800 through the network 812. In some examples, the external computing device may be an external web server or a cloud computing node.

[0102] It should be understood that the block diagram of FIG. 8 is not intended to indicate that computer system 800 includes all of the components depicted in FIG. 8. Rather, computer system 800 may include any suitable fewer or additional components not depicted in FIG. 8 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Additionally, the embodiments described herein with respect to computer system 800 may be implemented with any suitable logic. Logic, as referred to herein, in various embodiments, may include any suitable hardware (e.g., a processor, embedded controller, or application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware.

[0103] In one or more embodiments of the invention, computer system 800 may be environment 100 where a SUT is a computer program or hardware component used by computer system 800. In one or more embodiments of the invention, computer system 800 may be SUT 214, where SUT 214 is part of a server cluster.

[0104] The invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0105] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards or mechanically encoded devices such as ridge structures in grooves that allow instructions to be recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through an electrical wire.

[0106] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0107] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may execute completely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to perform aspects of the invention.

[0108] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0109] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored in a computer readable storage medium that may instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored includes an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0110] Computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0111] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0112] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond those found in the market, or to enable other skilled in the art to understand the embodiments described herein.

[0113] Various embodiments of the present invention are described herein with reference to the associated drawings. Alternative embodiments of the present invention may be devised without departing from the scope of the present invention. Various connections and relationships (e.g., above, below, adjacent, etc.) are described between elements in the following description and drawings. These connections and / or relationships may be direct or indirect unless otherwise specified, and the present invention is not intended to be limited in this respect. Thus, connections of elements may refer to either direct or indirect connections, and relationships between elements may be direct or indirect relationships. Furthermore, various tasks and processing steps described herein may be incorporated into a more comprehensive procedure or process having additional steps or functionality not described in detail herein.

[0114] The following definitions and abbreviations shall be used for interpreting the claims and the specification. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," or "containing," or any other variation thereof, are intended to include a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or device that includes a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or device.

[0115] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" may be understood to include any integer number greater than or equal to one, i.e., 1, 2, 3, 4, etc. The term "multiple" may be understood to include any integer number greater than or equal to two, i.e., 2, 3, 4, 5, etc. The term "connected" may include both an indirect "connected" and a direct "connected."

[0116] The terms "about," "substantially," and "approximately," and variations thereof, are intended to include the degree of error associated with measurement of a particular quantity based on equipment available at the time of the filing of this application. For example, "about" can include a range of ±8%, or 5%, or 2% of a given value.

[0117] For the sake of brevity, the prior art related to making and using aspects of the present invention is not necessarily described in detail herein. In particular, various aspects of computing systems and specific computer programs that implement various technical features described herein are well known. Thus, for the sake of brevity, details of many conventional implementations are only briefly described herein or omitted entirely without providing details of well-known systems and / or processes.

Claims

1. 1. A method for detecting and localizing defects in a system under test (SUT), comprising: modeling input to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space that includes all possible combinations of attribute-value pairs to a reduced test space by eliminating one or more combinations of attribute-value pairs based at least in part on corresponding probability values; generating an initial set of test vectors that provide complete n-wise coverage of the reduced test space; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining that one or more test cases failed to execute based at least in part on the first set of execution results; generating a set of new test cases from the selected failing test cases; executing the new set of test cases to obtain a second set of execution results; Detecting and localizing the defects based at least in part on the second set of execution results; and A method comprising:

2. The method of claim 1 , wherein the Cartesian product space is further reduced based on architectural constraints of the SUT.

3. 3. The method of claim 2, wherein the architectural constraints include at least one of: a requirement that a first attribute cannot take on one or more candidate attribute values ​​if a second attribute has a particular attribute value; a requirement that the first attribute must take on a particular candidate attribute value if the second attribute has the particular attribute value; or a requirement to introduce a third attribute if the second attribute has the particular attribute value.

4. The method of claim 1 , wherein the attribute-value pairs and corresponding probability values ​​are represented as a hash.

5. The method of claim 1 , wherein a first attribute-value pair being tested is assigned a first probability value that is higher than a second probability value of a second attribute-value pair.

6. The method of claim 1 , wherein detecting and localizing the defect based at least in part on the second set of execution results comprises determining a particular combination of attribute values ​​that causes the defect.

7. The method of claim 6 , further comprising generating regression buckets of failing test cases based at least in part on the particular combination of attribute values ​​that cause the defect.

8. 2. The method of claim 1, wherein generating the set of new test cases comprises, associated with each attribute in the selected failing test cases, generating a respective subset of new test cases at least in part by changing a respective attribute value for the attribute in the selected failing test cases to a respective other candidate attribute value for the attribute that is not present in any of the one or more test cases that failed to execute.

9. 2. The method of claim 1 , wherein generating an initial set of test vectors comprises generating an initial set of test vectors that provide full n-wise coverage of the reduced test space associated with the set of attribute-value pairs, and generating a set of new test cases from the selected failing test cases comprises modifying respective attribute values ​​in the selected failing test cases to generate respective new test cases.

10. 1. A system comprising: Memory, A processor coupled to the memory, the processor configured to execute a method for detecting and localizing defects in a system under test (SUT), the method comprising: modeling input to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space that includes all possible combinations of attribute-value pairs to a reduced test space by eliminating one or more combinations of attribute-value pairs based at least in part on corresponding probability values; generating an initial set of test vectors that provide complete n-wise coverage of the reduced test space; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining that one or more test cases failed to execute based at least in part on the first set of execution results; generating a set of new test cases from the selected failing test cases; executing the new set of test cases to obtain a second set of execution results; Detecting and localizing the defects based at least in part on the second set of execution results; and the processor, A system comprising:

11. The system of claim 10 , wherein the Cartesian product space is further reduced based on architectural constraints of the SUT.

12. 12. The system of claim 11, wherein the architectural constraints include at least one of: a requirement that a first attribute cannot take on one or more candidate attribute values ​​if a second attribute has a particular attribute value; a requirement that the first attribute must take on a particular candidate attribute value if the second attribute has the particular attribute value; or a requirement to introduce a third attribute if the second attribute has the particular attribute value.

13. The system of claim 10 , wherein the attribute-value pairs and corresponding probability values ​​are represented as a hash.

14. The system of claim 10 , wherein a first attribute-value pair being tested is assigned a first probability value that is higher than a second probability value of a second attribute-value pair.

15. The system of claim 10 , wherein detecting and localizing the defect based at least in part on the second set of execution results comprises determining a particular combination of attribute values ​​that causes the defect.

16. 16. The system of claim 15, the method further comprising generating regression buckets of failing test cases based at least in part on the particular combination of attribute values ​​that cause the defect.

17. 1. A method for detecting and localizing defects in a system under test (SUT), comprising: modeling input to the SUT as a set of attribute-value pairs; assigning a probability value to each attribute-value pair from the set of attribute-value pairs; reducing a Cartesian product space that includes all possible combinations of attribute-value pairs to a reduced test space by eliminating one or more combinations of attribute-value pairs based at least in part on corresponding probability values; generating an initial set of test vectors that provide complete n-wise coverage of the reduced test space; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining that one or more test cases failed to execute based at least in part on the first set of execution results; generating a set of new test cases from the selected failing test cases, said generating including, associated with each attribute in the selected failing test cases, generating a respective subset of new test cases at least in part by changing a respective attribute value for the attribute in the selected failing test cases to a respective other candidate attribute value for the attribute that is not present in any of the one or more test cases that failed to execute; executing the new set of test cases to obtain a second set of execution results; Detecting and localizing the defects based at least in part on the second set of execution results; and A method comprising:

18. 20. The method of claim 17, wherein the attribute-value pairs and corresponding probability values ​​are represented as hash values.

19. 1. A method for detecting and localizing defects in a system under test (SUT), comprising: modeling input to the SUT as a set of attribute-value pairs; reducing a test space that includes all possible combinations of attribute-value pairs to a reduced test space by eliminating one or more combinations of attribute-value pairs based at least in part on corresponding probability values; generating an initial set of test vectors that provide complete n-wise coverage of the reduced test space associated with the set of attribute-value pairs; generating an initial set of test cases from the initial set of test vectors; executing the initial set of test cases to obtain a first set of execution results; determining that one or more test cases failed to execute based at least in part on the first set of execution results; generating a set of new test cases from the selected failing test cases, wherein generating the set of new test cases includes modifying respective attribute values ​​in the selected failing test cases to generate respective new test cases; executing the new set of test cases to obtain a second set of execution results; Detecting and localizing the defects based at least in part on the second set of execution results; and A method comprising:

20. 20. The method of claim 19, wherein the attribute-value pairs and corresponding probability values ​​are represented as a hash.

21. A computer program product causing a computer to carry out a method according to any one of claims 1 to 9 or claims 17 to 20.

22. A computer-readable storage medium having a computer program recorded thereon, the computer program being for causing a computer to carry out a method according to any one of claims 1 to 9 or claims 17 to 20.

Images