System testing infrastructure with hidden variable, hidden attribute, and hidden value detection

Abstract

Inputs to the system under test (SUT) are modeled as a set of attribute-value pairs. A set of test cases is executed using an initial set of test vectors that provides full n-wise coverage of the attribute-value pairs. Each test case execution yields a non-binary success rate (S AV ) is calculated for each attribute-value pair based on the binary execution results. Attributes are selected according to the set of success rates corresponding to the set of attribute-value pairs containing the attribute being all below a predetermined threshold. A set of test cases is run with another set of test vectors using additional values ​​for the selected attributes. For each run of the set of test cases, a second non-binary success rate (S AV ') is recorded for each attribute-value pair, where if a predetermined threshold is met, the user is notified of additional values ​​for the detected attributes.

Description

[Technical Field]

[0001] The present invention relates to testing computer systems, and more particularly to a test case infrastructure that can identify hidden variables, hidden attributes, and hidden attribute values ​​that have not been tested but should be tested. Additionally, the present invention relates to using the identified hidden variables to improve the testing infrastructure, for example, by generating additional test cases to test such variables, identifying code paths that must be executed, etc. [Background technology]

[0002] Computerized devices control almost every aspect of our lives, from writing documents, controlling traffic lights, completing e-commerce transactions, to controlling manned spacecraft missions. However, computerized devices are often prone to errors and therefore require a testing phase to discover and correct them. The testing phase is considered one of the most difficult tasks when designing a computerized device. The cost of not discovering an error can be enormous depending on how the computerized device is used, so thorough testing is important. Coverage tools for checking software provide a measure of how well the software being evaluated performed during testing, thereby providing a level of assurance that the software is of high quality. Summary of the Invention

[0003] According to one or more embodiments of the present invention, a method for detecting and localizing defects when testing a system under test (SUT) includes modeling inputs to the SUT as a set of attribute-value pairs and generating an initial set of test vectors that provide complete n-wise coverage of the test space represented by the attribute-value pairs. The method further includes generating a set of test cases from the initial set of test vectors. The method further includes executing the set of test cases to obtain a set of execution results, the execution results indicating in binary form whether the test cases passed or failed, and the set of test cases is executed multiple times. The method further includes calculating a non-binary success rate (S) based on the execution results for each execution of the set of test cases. AV ) for each attribute-value pair, where a non-binary success rate for the attribute-value pair is based on execution results of each test case using the attribute-value pair. The method further includes selecting the attribute in response to a set of success rates corresponding to a set of attribute-value pairs including the attribute being all below a predetermined threshold. A second set of test vectors is generated, including additional values ​​for the selected attribute. Further, the set of test cases is executed to obtain a second set of execution results using the second set of test vectors, and the set of test cases is executed at least a predetermined number of times. Further, a second non-binary success rate (S) is calculated for each execution of the set of test cases. AV ') is recorded for each attribute-value pair based on the execution results. In response to a second set of success rates corresponding to the set of attribute-value pairs including the attribute satisfying a predetermined threshold, a user is notified of additional values ​​for the detected attribute.

[0004] According to one or more embodiments of the present invention, a system includes a memory and a processor coupled to the memory, where the processor executes a method for detecting and localizing defects when testing a system under test (SUT). The method includes modeling inputs to the SUT as a set of attribute-value pairs and generating an initial set of test vectors that provide complete n-wise coverage of the test space represented by the attribute-value pairs. The method further includes generating a set of test cases from the initial set of test vectors. The method further includes executing the set of test cases to obtain a set of execution results, where the execution results indicate in binary form whether the test cases passed or failed, where the set of test cases is executed multiple times. The method further includes calculating a non-binary success rate (S) based on the execution results for each execution of the set of test cases. AV ) for each attribute-value pair, where a non-binary success rate for the attribute-value pair is based on execution results of each test case using the attribute-value pair. The method further includes selecting an attribute in response to a set of success rates corresponding to a set of attribute-value pairs including the attribute being all below a predetermined threshold. A second set of test vectors is generated, including additional values ​​for the selected attribute. Further, the set of test cases is executed to obtain a second set of execution results using the second set of test vectors, and the set of test cases is executed at least a predetermined number of times. Further, a second non-binary success rate (S) is generated for each execution of the set of test cases. AV ') is recorded for each attribute-value pair based on the execution results. In response to a second set of success rates corresponding to the set of attribute-value pairs including the attribute satisfying a predetermined threshold, a user is notified of additional values ​​for the detected attribute.

[0005] According to one or more embodiments of the present invention, a computer program product includes a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by the processor, cause the processor to perform a method for detecting and confining defects when testing a system under test (SUT). The method includes modeling inputs to the SUT as a set of attribute-value pairs and generating an initial set of test vectors that provide full n-wise coverage of the test space represented by the attribute-value pairs. The method further includes generating a set of test cases from the initial set of test vectors. The method further includes executing the set of test cases to obtain a set of execution results, the execution results indicating in binary form whether the test cases passed or failed, and the set of test cases is executed multiple times. The method further includes determining a non-binary success rate (S) based on the execution results for each execution of the set of test cases. AV ) for each attribute-value pair, where a non-binary success rate for the attribute-value pair is based on execution results of each test case using the attribute-value pair. The method further includes selecting an attribute in response to a set of success rates corresponding to a set of attribute-value pairs including the attribute being all below a predetermined threshold. A second set of test vectors is generated, including additional values ​​for the selected attribute. Further, the set of test cases is executed to obtain a second set of execution results using the second set of test vectors, and the set of test cases is executed at least a predetermined number of times. Further, a second non-binary success rate (S) is generated for each execution of the set of test cases. AV ') is recorded for each attribute-value pair based on the execution results. In response to a second set of success rates corresponding to the set of attribute-value pairs including the attribute satisfying a predetermined threshold, a user is notified of additional values ​​for the detected attribute.

[0006] Additional technical features and advantages are realized through the techniques of the present invention. Embodiments and aspects of the present invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, reference is made to the detailed description and drawings.

[0007] The particulars of the exclusive rights set forth herein are particularly pointed out and distinctly claimed in the claims at the end of the specification. The foregoing and other features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1] 1 shows a schematic diagram of a computerized environment in accordance with one or more embodiments of the present invention. [Figure 2] FIG. 2 is a block diagram illustrating modules that provide a testing infrastructure according to an embodiment of the present invention. [Figure 3] FIG. 1 is a schematic hybrid data flow / block diagram illustrating defect detection and localization using combinatorial test design (CTD) and generation of regression buckets of failing test cases that expose detected defects, in accordance with one or more embodiments of the present invention. [Figure 4] 1 illustrates an example set of CTD vectors, according to one or more embodiments of the present invention. [Figure 5] 1 illustrates an example CTD vector, according to one or more embodiments of the present invention. [Figure 6] FIG. 1 is a process flow diagram of an exemplary method for detecting and confining n-wise defects using CTD techniques and generating regression buckets of failing test cases that expose the detected n-wise defects, in accordance with one or more embodiments of the present invention. [Figure 7] 1 illustrates a flowchart of a method for detecting flaws in a test model using success rates of attribute-value pairs over time when testing a system under test, in accordance with one or more embodiments of the present invention. [Figure 8] 1 illustrates an example visualization of success rates (SAV) of attribute-value pairs, according to one or more embodiments of the present invention. [Figure 9] 1 illustrates a flowchart of a method for determining whether hidden attributes are present in a current test model in accordance with one or more embodiments of the present invention. [Figure 10] 1 illustrates a flowchart of a method for identifying portions of a system under test that drive hidden attributes, in accordance with one or more embodiments of the present invention. [Figure 11] 1 illustrates a flowchart of a method for identifying portions of a SUT that drive hidden attributes in accordance with one or more embodiments of the present invention. [Figure 12] 1 illustrates a computer system in accordance with one or more embodiments of the present invention. [Figure 13] 1 illustrates a cloud computing environment in accordance with one or more embodiments of the present invention. [Figure 14] 1 illustrates abstraction model layers in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] The figures shown herein are exemplary. Many variations may exist to the figures or the operations described therein without departing from the spirit of the invention. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Also, the term "coupled" and variations thereof refer to having a communication path between two elements, and do not imply a direct connection between elements with no intervening elements / connections between them. All of these variations are considered part of the specification.

[0010] Exemplary embodiments of the present invention relate, inter alia, to devices, systems, methods, computer-readable media, techniques, and methodologies for performing defect detection and localization using combinatorial test design (CTD) techniques and for generating regression buckets of test cases that expose the detected defects. In exemplary embodiments of the present invention, the detected and localized defects occur in a system under test (SUT). The SUT may be a hardware system or a software system. Furthermore, in exemplary embodiments of the present invention, defect detection and localization may be performed while adhering to architectural constraints on the SUT.

[0011] When assessing the quality of an SUT, it is essential to identify "weaker parts" of the SUT and understand why they exist. A "weaker part" in this context may be a part of the SUT that is more susceptible to failure when one or more parameters associated with the SUT change. For example, a "hidden attribute" that causes a failure may be considered a weak part of the SUT. Parameters associated with the SUT may include configurable values ​​of the SUT, such as the amount of memory, the number of processors, or any other such hardware / software that form the SUT, as well as any settings associated with the SUT's hardware and / or software. In addition, SUT parameters that may affect the success / failure of the SUT or the success / failure of one or more test cases of the SUT may include operating conditions associated with the SUT, such as temperature, humidity, power, the combination of software applications running in conjunction with the SUT, the number / type of jobs running in conjunction with the SUT, and any other such dynamic conditions that may vary across instances of the SUT.

[0012] Existing techniques for identifying weaker parts of a SUT are narrow in scope and do not adequately describe the metrics that cause failures or errors. Therefore, the metrics are not sufficient to explain the cause of one or more such failures. Some such existing techniques, such as calculating the success rate of a given test case, can provide early warning signs of underlying problems in the SUT. However, identifying underlying problems requires manual intervention and deep skill in the field.

[0013] For example, a test case failure may be due to an error in the implementation of the SUT itself, such as a bug. Alternatively, a test case failure may be due to incorrect user input. Alternatively, a test case failure may be caused by an operating condition that causes the SUT to behave in an unexpected way. Furthermore, a test case failure may be due to an untested (and therefore untuned) attribute or value in the test case. Additionally, a test case failure may occur in a first SUT with a certain configuration but not in a second SUT with a substantially identical configuration.

[0014] When testing an SUT, some errors are observed to not surface consistently due to the requirements of certain environmental or timing conditions. For example, for a software SUT, the conditions may include one or more parameters of the computer system on which the software is running. For example, the parameters may include the computer system's settings, other software running on the computer system, the computer system's hardware configuration, etc. Similarly, for a hardware SUT, the configuration of the system of which the hardware SUT is a part may affect environmental conditions. In other words, a particular error in an SUT may be the result of the system state and not the quality of the SUT itself. Some errors are sometimes referred to as "soft failures." Identifying such conditions that can lead to soft failures, referred to herein as "hidden variables," is a technical challenge. Additionally, identifying soft failures, i.e., determining that an error is a soft failure, can also be a technical challenge.

[0015] This difficulty is particularly exacerbated when hidden variables in the SUT significantly affect the test results. Hidden variables can be uncontrolled factors, and in some cases, cannot be controlled during test case execution. For example, any of the SUT's operating conditions, such as temperature, humidity, job mix, or available memory, that are not configured by the test case itself but can cause the test case to fail intermittently, could be considered hidden variables. Hidden variables are not always discoverable for a given testing infrastructure because they are outside the specific scope of the SUT. For example, stress tests may force the SUT into various resource-constrained configurations (e.g., memory, compute cycles, etc.), thereby causing unpredictable errors or failures in the SUT that are difficult, if not impossible, to reproduce consistently. This means that even if the specific functionality or features that a stress test verifies are stable, the SUT may perform improperly or fail depending on the resources on which the stress test depends.

[0016] Thus, a technical challenge exists in the testing infrastructure to determine when one or more hidden variables consistently affect the outcome of a given test case, and to further identify such hidden variables when a test case is failing due to factors beyond the testing infrastructure itself, so that resources can be adjusted or new testing infrastructure can be developed accordingly.

[0017] Embodiments of the present invention address such technical challenges in test infrastructures for computer systems. Note that the test infrastructure itself includes a computer system, including a memory device, a processing unit, and one or more computer-executable instructions. Thus, embodiments of the present invention facilitate improvements to computing technologies, test infrastructures, and SUTs. In one or more embodiments of the present invention, embodiments of the present invention may utilize pseudo-Boolean algebra, such as Heiting's algebra, to encapsulate hidden variables that consistently affect test case outcomes while quantizing the success rate of a test vector. For example, for a test vector <Andrew, Blue, 30>, which in this case has three attributes <Name, Color, Age> with assigned value combinations, the success rate indicates how often a test case succeeds while using this particular value combination. An "attribute-value pair" represents a specific value assigned to a particular attribute being tested by a test case. An attribute may cycle through multiple values ​​during testing. Furthermore, there may be multiple such attributes being tested. Quantizing the success rate of attribute-value pairs allows users, such as quality assurance engineers, to reliably identify when a test case is failing due to factors greater than or equal to the test infrastructure itself, so that resources can be adjusted or new test infrastructure can be developed accordingly. This enhanced success metric can be used to improve the efficiency and reliability of testing efforts, improving the overall stability of the SUT. Thus, embodiments of the present invention provide improvements, practical applications, and technical solutions to technical challenges in computing technology, particularly in the test infrastructure used to test SUTs.

[0018] Additionally, coverage tools for testing the SUT provide a measure of how well the SUT behaves during testing, thereby providing a level of assurance that the SUT is of high quality. There are several types of coverage known in the art, such as statement coverage, line coverage, condition coverage, path coverage, and method coverage. One further coverage method is functional coverage. Functional coverage is designed to measure the amount, portion, or similar metric of tests that examine a given functional behavior. Once functional coverage is measured, quality assurance (QA) personnel can design additional tests to examine untested behavior.

[0019] However, test infrastructure can be very costly, and without constant optimization of the product, tests, and environment, the cost per test cycle increases. As more tests are added, more test suites are operated, and the complexity of support services increases, along with the cost of each test cycle. Test environments, test dependencies, and test suites must be managed with the same rigor as the shipped product (i.e., the SUT). Therefore, just as managing the test infrastructure is a technical challenge, so is efficiently scaling it down. Note that scaling down the test infrastructure does not mean a decrease in the quality of testing the SUT. Rather, the technical challenge is that test quality should improve as the test infrastructure scales down.

[0020] Embodiments of the present invention address such technical challenges. According to one or more embodiments of the present invention, a reduced set of test cases for the SUT is run on the system. Note that the system is distinct from the SUT; the SUT is run using the system. The reduced set of test cases can be generated using combinatorial test design (CTD). Errors can be identified based on a subset of tests that fail from the reduced set. Furthermore, according to one or more embodiments of the present invention, inverse CTD is used to localize defects in the system.

[0021] According to one or more embodiments of the present invention, CTD is performed with respect to interaction requirements, such as pairwise interactions, n-tuple interactions, or others. Test planning can provide sufficient coverage tasks to ensure that interaction requirements are met. In some exemplary embodiments, test planning is based on modeling a test space using a functional coverage model. The test space can represent inputs, scenarios, configurations, the internal state of an application, or any other aspect that may need to be tested. The functional coverage model can include a set of functional attributes. In other embodiments of the present invention, the coverage model can include one of several types of coverage known in the art, such as statement coverage, line coverage, condition coverage, path coverage, and method coverage. The type of coverage does not affect the reduction of the test infrastructure techniques described herein.

[0022] Furthermore, with respect to a test plan, a test space may define potential tests that may be executed on the SUT. A test, as opposed to a coverage task, may be associated with only one potential test. In some exemplary embodiments of the present invention, several different tests may have the same potential test.

[0023] In some exemplary embodiments of the present invention, the motivation for the CTD approach is that most errors depend on interactions between a small number of functional attributes. CTD algorithms may also address scalability issues. CTD algorithms may assume that the combinatorial space is too large to be represented explicitly, and they use various techniques to attempt to generate optimized test plans without explicitly enumerating all possible combinations. Examples of such algorithms may be reduction to orthogonal arrays or covering array problems. These techniques are limited in the types of constraints they impose on value combinations or the level of interaction they require, and they still encounter scalability issues as the number of model variables increases.

[0024] In one or more embodiments of the present invention, inputs to the SUT are modeled as a set of attribute-value pairs. More specifically, inputs to the SUT may be modeled as a set of attributes, each of which may have one or more corresponding attribute values. In one or more embodiments of the present invention, the entire Cartesian product space, which contains all possible combinations of attribute-value pairs, may be reduced to a smaller set of test vectors that provide full n-wise coverage of the entire test space. For example, if four different attributes A, B, C, and D are assumed to be modeled, and if it is further assumed that attribute A can take on four distinct values, attribute B can take on three distinct values, attribute C can take on three distinct values, and attribute D can take on two distinct values, then the total number of possible combinations of attribute-value pairs is 4 × 3 × 3 × 2 = 72. Thus, in this illustrative example, the entire Cartesian product space contains 72 different combinations of attribute-value pairs. As mentioned above, these 72 different combinations of attribute-value pairs can be reduced to a smaller set of combinations that still provide full n-wise coverage of the Cartesian product space. For example, referring to the same example introduced above, if full pairwise coverage is desired, the 72 different combinations can be reduced to 12 distinct combinations that together contain every possible pairwise interaction of attribute values. The reduced number of combinations required to provide n-wise coverage can increase logarithmically as n increases.

[0025] In an exemplary embodiment of the present invention, the entire Cartesian product space including all combinations of attribute value pairs is reduced to a smaller set of CTD test vectors that provide complete n-wise coverage for a desired n. In complex hardware or software systems, the total number of attributes and corresponding candidate attribute values may be extremely large. In such cases, the total number of possible combinations of attribute values that make up the entire Cartesian product space can be astronomically large, making it virtually impossible to test all possible combinations. By reducing the entire Cartesian product space to a significantly smaller number of CTD test vectors that still provide complete n-wise coverage, it becomes possible to detect any n-wise (or m-wise for m < n) defects without directly testing all possible combinations in the test space.

[0026] In an exemplary embodiment of the present invention, a binary decision diagram or the like can be used to perform the reduction and identify a reduced set of CTD vectors that provide complete n-wise coverage. In one or more embodiments of the present invention, each generated CTD test vector includes a unique combination of attribute values, and the set of CTD test vectors together includes all possible n-wise interactions of the attribute values. In particular, each CTD vector can have dimensions corresponding to the number of attributes being modeled. In that case, each component of the CTD vector is the respective attribute value of the corresponding attribute. However, the set of generated CTD vectors may not be unique. That is, there may be multiple different sets of CTD vectors, each of which provides complete n-wise coverage. In one or more embodiments of the present invention, the CTD vectors can be randomly selected while still ensuring complete n-wise coverage. In one or more other embodiments, the initial set of CTD vectors can be selected according to specific criteria, such as considering increasing or decreasing the representation of specific attribute values in the set of CTD vectors.

[0027] In an exemplary embodiment of the present invention, once an initial set of CTD test vectors is generated, they are used to generate a corresponding set of test cases. For example, a set of CTD test vectors can be provided as input to a test case generation tool configured to generate each corresponding test case for each CTD vector. Each test case can be designed to test the interaction between specific combinations of attribute values included in the corresponding CTD vector.

[0028] In an exemplary embodiment of the present invention, then the test cases are executed. The execution of each test case results in either a successful execution result indicating that the combination of attribute values associated with the test case does not contain an n-wise (or m-wise if m < n) error, or a failed execution result indicating that the combination of attribute values associated with the test case contains an n-wise (or m-wise if m < n) error. In one or more embodiments of the present invention, then a specific failed test case is selected and the inverse combination theory is applied to the failed test case to create a new set of test cases capable of manifesting bugs. In an exemplary embodiment of the present invention, applying the inverse combination theory to the selected failed test case includes generating a respective new test case for each attribute. Thus, the number of new test cases generated can be equal to the number of attributes. In an exemplary embodiment of the present invention, in each respective new test case, the attribute value of the corresponding attribute in the selected failed test case is changed to the attribute value for an attribute that does not exist in any of the failed test cases, and the respective attribute values for each of the other attributes are not changed from those that exist in the selected failed test case.

[0029] Those new test cases that result in successful execution results can then be evaluated to detect and localize n-wise (or m-wise where m < n) errors. In particular, the specific attribute value pairs that cause an error can be identified based on the new test cases that execute successfully. A regression bucket of test cases can then be generated based on the combinations that cause an error in the attribute value pairs. More specifically, all possible combinations of attribute values that include the attribute values determined to cause an error may be determined, and the corresponding regression buckets of test cases for testing these combinations can be output for use, for example, by a manual tester. In particular, in one or more embodiments of the present invention, all test cases included in the regression bucket should fail at runtime until the bug is fixed, and after the bug is fixed, all regression bucket test cases should pass (i.e., result in a successful execution result).

[0030] In an exemplary embodiment of the present invention, architectural restrictions can be applied to the SUT. The architectural restrictions can include any of a variety of restrictions on the inputs to the SUT. For example, as an example, the restriction may be that when a given attribute has a particular attribute value, one or more other attributes are excluded from having a certain attribute value. As another example, an architectural restriction may be that when a given attribute has a particular attribute value, one or more other attributes must have a certain attribute value. As yet another example, an architectural restriction may be that a new attribute is introduced only when a particular attribute has a particular attribute value. It should be understood that the above examples of architectural restrictions are merely illustrative and not exhaustive.

[0031] In an exemplary embodiment of the present invention, architectural constraints are taken into account before performing a reduction of the entire Cartesian product space to an initial set of CTD vectors that provides the desired complete n-wise coverage. That is, in an exemplary embodiment of the present invention, certain combinations of attribute values ​​that violate any architectural constraints are first excluded from the Cartesian product space, and then a reduction to a set of CTD vectors that provides the desired complete n-wise coverage is performed. In this manner, it can be ensured that combinations of attribute values ​​that violate architectural constraints are not included in the initial set of CTD vectors. Furthermore, in an exemplary embodiment of the present invention, when a selected test case corresponding to an initial CTD vector fails and is selected for expansion to obtain a new set of test cases designed to detect and localize n-wise errors, the expansion can be performed first without taking any architectural constraints into account. Any new test cases that violate architectural constraints can then be excluded. In this manner, it can be ensured that the expansion provides as much coverage of the test space as possible in the neighborhood of the failing test cases. Additionally, in exemplary embodiments of the present invention, a regression bucket of failing test cases may be first generated without regard to architectural constraints, and then any test cases that violate the architectural constraints may be excluded from the regression bucket. That is, test cases corresponding to all possible combinations in the entire Cartesian product test space, including specific combinations of attribute values ​​that cause n-wise errors or lower-order errors, may be first generated and then reduced to exclude any test cases that violate the constraints. Alternatively, the test cases included in the regression bucket may be selected from a reduced Cartesian product space from which an initial set of CTD vectors is selected, in which case the regression bucket is generated to include only those combinations of attribute values ​​that include a subset of the error-causing attribute values ​​and that do not violate any architectural constraints.

[0032] One or more embodiments of the present invention include various technical features that produce technical effects that bring various improvements to computer technology. For example, an exemplary embodiment of the present invention includes the technical feature of reverse combinatorics for expanding a set of test cases from a selected failing test case. This technical feature enables n-wise or lower-order defects to be detected and localized within a single iteration of expansion. This technical effect represents an improvement to debugging and fault detection computer technology because it automates the generation of additional test cases around a selected failing test case in a manner specifically designed to facilitate defect detection and localization. Thus, the automated generation of new test cases according to one or more embodiments of the present invention enables more efficient defect exposure than manual test case generation and the use of a fundamentally different methodology. An exemplary embodiment of the present invention also includes the technical feature of generating a regression bucket of test cases that tests all possible combinations of attribute values, including a specific subset of attribute values ​​that cause n-wise defects. Thus, the regression bucket contains only test cases that all fail before the defect is corrected and all pass after the defect is corrected. Thus, the automatic generation of regression buckets of failing test cases according to one or more embodiments of the present invention provides an improvement to computer technology by providing a tester or automated testing algorithm with a set of test cases, each of which is sure to fail, so that the tester or automated testing algorithm can use the regression buckets of test cases to verify that defects have been corrected when all test cases ultimately pass after a debug run.

[0033] During the lifecycle of an effort to test an SUT, significant time and resources are spent building sustainable, reliable test cases and infrastructure that provides a consistent test system for executing test cases and validating the SUT. The end result of building a test system is a reliable and efficient way to continuously validate the SUT throughout its lifespan, especially as maintenance and updates are applied over time. A common way to ensure sufficient test coverage is to ensure that test coverage is sufficient through the use of Computational Testing.

[0034] A technical challenge with this paradigm is that test model designers either overlook attribute values ​​from those to be tested or define insufficient granularity for the values ​​of attributes to be tested. In such cases, the test model generated for use by the test system may be incomplete, leading to incomplete testing of the SUT. Another technical challenge is the biased use of any one test attribute variable value over time due to the specific algorithm used to generate the tests. This again results in some attribute values ​​not being executed, further resulting in incomplete testing of the SUT. In addition to missing attribute values, as previously mentioned, technical challenges also include hidden variables causing test cases to fail to test and / or identify the root cause of failures that may be occurring in the SUT.

[0035] Note that a "hidden variable" may fail intermittently due to difficult-to-control parameters, such as the temperature of the SUT, from other applications, jobs, or services running concurrently with the SUT. A "hidden attribute" is an attribute that should be included in the model to generate better test vectors. A "hidden attribute value" is a value that should be included in an attribute in the model to generate better test vectors, but was missing / overlooked during test model generation, or was not considered by the user when designing the test model. On the other hand, a "hidden value" is an environmental / external factor that affects the outcome of one or more test cases.

[0036] Embodiments of the present invention address such technical challenges by further facilitating the use of a pseudo-Boolean algebra, such as Heyting's algebra, Kripke's algebra, or any other intuitionistic logic, to automatically generate and select test cases for testing the SUT using CTD and to detect one or more hidden variables that cause one or more test case failures. Embodiments of the present invention further facilitate using the detected hidden variables to determine one or more portions of the SUT that may require further diagnosis to detect errors that may cause test case failures.

[0037] In one or more embodiments of the present invention, the success rate of each attribute-value pair over time is calculated using a pseudo-Boolean algorithm to identify missing / hidden attributes that should be tested. The success rate history is then visually displayed using an irregularly distributed visual representation. The visual representation may also be referred to as a non-uniform table or heat map of the test vector attribute values ​​of the model used to test the SUT. Alternatively, or in addition, attributes in the model containing values ​​with success rates that do not meet a predetermined threshold are identified based on the historical data. The identified attributes are used to identify and expose depth deficiencies in the set of values ​​used to test at least that attribute. By adding additional values ​​to the attribute and any other related attributes, the success rate of the attribute is again monitored over time. If a success rate threshold is met, the hidden value or hidden attribute may be considered to have been identified by the added value. The hidden value may then be used in a further set of test cases.

[0038]

[0023] Referring now to Figure 1, Figure 1 shows a schematic diagram of a computerized environment according to an embodiment of the present invention. The computerized environment 100 may include one or more computerized tools. Note that the diagram is one possible example of a block diagram and that some components may not be shown for clarity.

[0039] In one or more embodiments of the present invention, users 110, such as developers, QA staff members, testers, designers, verification engineers, etc., may interact with computerized environment 100. Users 110 may utilize man-machine interfaces (MMI) 112, such as terminals, displays, keyboards, input devices, etc.

[0040] In some exemplary embodiments of the present invention, a coverage model definition tool 102 may be utilized to define a test coverage model, such as a functional coverage model. In some exemplary embodiments of the present invention, a user 110 may define attributes to be tested, e.g., functional attributes if functional coverage is being tested. In some exemplary embodiments of the present invention, a similar tool may be utilized to define a test space. In some exemplary embodiments, a coverage model may be adapted to be utilized as a test model.

[0041] In some exemplary embodiments of the present invention, the test execution engine 108 may be utilized to test the SUT. It should be noted that the SUT may be hardware, firmware, software, a combination thereof, or any other type of computerized device. The test execution engine 108 may be a simulation-based verification tool, a test generation platform, etc. The test execution engine 108 may be operatively coupled to the test planning tool 106 and configured to execute tests according to a test plan. In some exemplary embodiments, the test planning tool 106 may provide tests for execution by the test execution engine 108. It should be noted that dynamic verification is a broader concept than testing of the SUT and further includes test planning, coverage analysis, etc. The test execution engine 108 provides only one aspect of the overall behavioral scope that may be executed during dynamic verification, and the term “dynamic verification” should not be used to interpret the term more narrowly.

[0042] In some example embodiments of the present invention, the coverage analysis tool 104 is configured to measure the coverage of a test space for the SUT based on the dynamic verification performed by the test execution engine 108. For example, the coverage analysis tool 104 may be a functional coverage analysis tool. The coverage analysis tool 104 provides coverage measurements, such as a coverage test space or a portion of a defined test plan, that indicate the coverage tasks that were covered during the dynamic verification performed by the test execution engine 108. The user 110 may review the coverage measurements, the list of covered tasks, or both.

[0043] In some exemplary embodiments of the present invention, the test planning tool 106 may define a covered test plan. In some exemplary embodiments of the present invention, the test plan may be a set of covered coverage tasks. In some exemplary embodiments of the present invention, the test planning tool 106 may provide tests that are deemed to cover the test plan based on test benchmarks, etc., stored in a data store containing tests known / presumed to cover specific aspects of the SUT. As another example, the test planning tool 106 is configured to generate tests that cover the coverage tasks. The user 110 may review the test plan, selected tests, etc. In some exemplary embodiments of the present invention, the user 110 may provide parameters for the test planning tool 106 to use in determining the objectives of the test plan, such as the desired interaction level. Embodiments of the present invention enable the elimination of any redundancies in the generated tests.

[0044] Although the illustration of FIG. 1 is described with particular components, including a coverage model definition tool 102, a coverage analysis tool 104, a test planning tool 106, and a test execution engine 108, embodiments of the present invention are not limited to these components or system configurations and may be practiced with other system configurations employing fewer or additional components.

[0045] 2 is a block diagram illustrating modules that provide a testing infrastructure according to an embodiment of the present invention. More specifically, the infrastructure includes a test generator 208. The test generator 208 accesses a testing infrastructure, such as a repository of test cases 202, which stores a suite of test cases that can be used to verify the correctness of the SUT 214. Each test case specifies an input to be applied to the SUT 214 and an expected response to that input (to demonstrate its correct behavior). Typically, the test cases are organized into sets (test suites), e.g., one test suite for each different component of the SUT 214.

[0046] The test generator 208 creates an execution bucket for each test run on the SUT 214. The bucket specifies the actions to be taken to execute the desired test cases in a machine-readable language (e.g., XML-based). In particular, in the case of an exhaustive test, all available test cases are executed on each component of the SUT 214; conversely, in the case of a regression test, the execution is limited to a selected subset of test cases. The buckets thus obtained can be saved to a file.

[0047] The test execution engine 108 controls the execution of the buckets read from the file. For each test case in the bucket 212, this involves applying the corresponding input (attribute values) to the SUT 214. In response, the SUT 214 returns the corresponding output to the test execution engine 108. The test execution engine 108 determines the outcome of the test case by comparing the output with the corresponding expected response (e.g., extracted from the file). The result of the test case (i.e., positive if the two values ​​match, negative otherwise) is saved in a log. For example, this can be achieved using a standard test tracking tool (TTT). The (current) execution results of the tests are available in the log for their analysis.

[0048] For large, complex SUTs 214, test case infrastructure 202 may contain a large number of duplicate test actions / test cases. According to one embodiment of the present invention, one technique developed to address these challenges combines the use of fingerprint repository 206 with a store of information containing a set of fingerprints corresponding to multiple regression tests stored in test repository 202. In one embodiment, the fingerprints are generated by fingerprint generator 204. For example, fingerprint repository 206 may contain fingerprints for most, if not all, of the tests stored in test repository 202, along with references to where copies of the tests are located throughout the test infrastructure, similar to an index in a book. Each fingerprint uniquely identifies a specific code path covered by the corresponding test case. In this way, the system may identify redundant and possibly duplicate fingerprints (and corresponding tests).

[0049] According to embodiments of the present invention, processing the results 216 of the regression tests generated by the test generator 208 and executed by the test execution engine 108 may include determining the code paths traversed during execution of each generated regression test and generating a fingerprint for each executed regression test case based at least in part on the code paths. In some embodiments of the present invention, these steps may be performed by the fingerprint generator 204. Here, the "code path" associated with a test case refers to the portion of the SUT 214 executed by the test case.

[0050] Redundancy analyzer 218 compares the fingerprints corresponding to all test cases generated by test generator 208 with the fingerprints stored in fingerprint repository 206. Redundancy analyzer 218 provides a list of regression test cases generated by test generator 208 that have fingerprints that match one or more fingerprints stored in fingerprint repository 206. This information is used to select and discard duplicate test cases from test repository 202 (as described below).

[0051] Figure 3 is a schematic hybrid data flow / block diagram illustrating defect detection and localization using CTD techniques and generating regression buckets of failing test cases that expose the detected defects, in accordance with one or more embodiments of the present invention. Figure 6 is a process flow diagram of an exemplary method 600 for detecting and localizing n-wise defects using CTD techniques and generating regression buckets of failing test cases that expose the detected n-wise defects, in accordance with one or more embodiments of the present invention. Figure 6 will be described in conjunction with Figures 1-5.

[0052] Exemplary methods according to one or more embodiments of the present invention, as well as corresponding data structures (e.g., modules, units, and other such components) for implementing the methods, are described herein. It should be noted that each operation of one or more methods described herein may be performed by one or more of the modules described herein. These modules may be implemented in any combination of hardware, software, or firmware described herein, or combinations thereof. In one or more embodiments, one or more of these modules may be implemented at least in part as software and / or firmware modules that include computer-executable instructions that, when executed by a processing circuit, cause the execution of one or more operations. A system or device described herein as configured to implement one or more embodiments may include one or more processing circuits, each of which may include one or more processing units or nodes. The computer-executable instructions may include computer-executable program code that, when executed by a processing unit, may access and process input data contained in or referenced by the computer-executable program code to produce output data.

[0053] 6 in conjunction with FIG. 3 , in one or more embodiments of the invention, at block 602 of method 600, input to the SUT 214 is modeled as a set of attribute-value pairs 302. Any number of attributes may be used to model the SUT input, and each attribute may have any number of candidate attribute values. At block 604, computer-executable instructions of one or more n-wise coverage CTD vector generation modules 304 are executed to generate an initial set 306 of CTD vectors that provide n-wise coverage of the entire Cartesian product space associated with the set of attribute-value pairs 302.

[0054] In particular, the entire Cartesian product space, which contains all possible combinations of attribute-value pairs 302, is reduced to a smaller set of CTD test vectors 306 that provides complete n-wise coverage of the entire test space. In one or more embodiments of the present invention, the complete n-wise coverage provided by the set of CTD vectors 306 may be complete pairwise coverage. For example, if three attributes are assumed to be modeled: the “Name” attribute, the “Color” attribute, and the “Shape” attribute shown in FIG. 4 , and if it is further assumed that the “Name” attribute can take on four distinct attributes (Dale, Rachel, Andrew, and Ryan), the “Color” attribute can take on two distinct attributes (Green, Blue), and the “Shape” attribute can take on three distinct attributes (Circle, Square, Triangle), then the total number of possible combinations of attribute-value pairs is 4 × 3 × 2 = 24. Thus, in this illustrative example, the entire Cartesian product space would contain 24 different combinations of attribute-value pairs.

[0055] The 24 different combinations of attribute-value pairs can be reduced to a smaller set of combinations (i.e., set of CTD vectors 306) that still provide complete n-wise coverage of the Cartesian product space. For example, if complete pairwise coverage is desired, the 24 different combinations can be reduced to 12 distinct combinations that together contain all possible pairwise interactions of attribute values. An example set 400 of CTD vectors is shown in FIG. 4. The example set 400 of CTD vectors contains all pairwise interactions between attribute values ​​of the attributes "name," "color," and "shape."

[0056] A binary decision diagram or the like may be used to perform the reduction and identify a reduced set 306 of CTD vectors that provides full n-wise coverage. While each CTD vector in the set 306 of CTD vectors includes a unique combination of attribute values, the set 306 of CTD vectors itself may not be unique. That is, there may be multiple different sets of CTD vectors, each of which provides full n-wise coverage. For example, while FIG. 4 illustrates an example set 400 of CTD vectors for the described example attributes and attribute values, it should be understood that alternative sets of CTD vectors including different combinations of attribute values ​​may independently provide full n-wise coverage. It should further be understood that the example set 400 of CTD vectors provides full pairwise coverage and partial 3-wise coverage, but a greater number of CTD vectors is required to provide full 3-wise coverage. More generally, as n increases, the number of CTD vectors required to provide full n-wise coverage increases logarithmically with n.

[0057] 5 illustrates an example CTD vector 500 of the type that may be included in the set of CTD vectors 306. The example CTD vector 500 includes multiple attributes 502. As mentioned above, the attributes 502 may be used to model input to the SUT 214. The attributes 502 may be associated with attribute values ​​504. In particular, each attribute 502 may have a corresponding attribute value 504, which may be one of one or more candidate attribute values ​​that the attribute can take.

[0058] In one or more embodiments of the present invention, architectural constraints are considered in method 600 before performing a reduction of the entire Cartesian product space to an initial set of CTD vectors 306 that provides the desired full n-wise coverage. That is, certain combinations of attribute values ​​that violate any architectural constraints are first excluded from the Cartesian product space, and then a reduction is performed to a set of CTD vectors 306 that provides the desired full n-wise coverage. In this way, it can be ensured that combinations of attribute values ​​that violate architectural constraints are not included in the initial set of CTD vectors 306.

[0059] Architectural constraints may include any of a variety of restrictions on inputs to SUT 214. For example, an example constraint may be that if a given attribute has a particular attribute value, then one or more other attributes are excluded from having certain attribute values. Another example architectural constraint may be that if a given attribute has a particular attribute value, then one or more other attributes must have certain attribute values. Yet another example architectural constraint may be that a new attribute is introduced if and only if a particular attribute has a particular attribute value. It should be understood that the above examples of architectural constraints are merely illustrative and not exhaustive.

[0060] Referring again to FIG. 6 , at block 606 of method 600, in one or more embodiments of the present invention, test case generation module 208 may be executed to generate a corresponding set of test cases 202 from the initial set of CTD test vectors 306, which are then executed by test case execution module 108 to produce an execution result (pass or fail) for each test case.

[0061] For example, the set 306 of CTD test vectors can be provided as input to a test case generation tool 208 that generates a respective corresponding test case for each CTD vector. Each test case within the set of test cases 202 tests the interaction between specific combinations of attribute values included in the corresponding CTD vector of the set 306 of CTD vectors. It should be understood that the set of CTD vectors and their corresponding test cases can be interchangeably described or illustrated, or both, at this point in time. For example, the set 400 as an example of a set of CTD vectors shown in FIG. 4 can be interchangeably considered as a corresponding set of test cases that test specific combinations of attribute values represented by the set 400 of CTD vectors.

[0062] In block 608 of method 600, the computer-executable instructions of the test case execution module 108 are executed to determine whether any test case within the set of test cases 202 has failed. In one or more embodiments of the present invention, the execution of each test case 202 results in either a successful execution result indicating that the combination of attribute values included in the corresponding CTD vector 306 does not include an n-wise (or m-wise where m < n) error, or a failed execution result indicating that the combination of attribute values in the corresponding CTD vector 306 includes an n-wise (or m-wise where m < n) error.

[0063] Referring to the example shown in FIG. 4, test cases corresponding to the example set 400 of CTD vectors are executed in block 606 to produce respective execution results for each test case. In particular, two test cases 402 and 404 are illustratively shown in FIG. 4 as producing failures. Failing test case 402 tests the following combination of attribute values: Dale, Blue, Triangle, which correspond to the name, color, and shape attributes, respectively. Failing test case 404 tests the following combination of attribute values: Dale, Blue, Circle, which correspond to the name, color, and shape attributes, respectively. Although "Dale" and "Blue" are present in both the CTD vector corresponding to failing test case 402 and the CTD vector corresponding to failing test case 404, it is unclear at this stage in the process whether "Dale" and "Blue" are generating a pairwise error, whether "Dale" and ("Triangle" or "Circle") are generating a pairwise error, or whether "Blue" and ("Triangle" or "Circle") are generating a pairwise error. Subsequent operations of method 600 may utilize inverse combinatorics to expand the test space around the selected failing test case to detect and localize the specific combination of attribute values ​​that is causing the n-wise error (the pairwise error in the example shown in FIG. 4).

[0064] In response to a negative determination at block 608, the method 600 may end because none of the initial set of test cases 202 failed. Because the initial set of CTD vectors 306 (based on which the set of test cases 202 was generated) provided full n-wise coverage, it may be guaranteed that there are no n-wise errors or lower order errors. However, successful execution of all test cases in the initial set of test cases 202 does not guarantee that there are no higher order errors (k-wise, where k>n).

[0065] On the other hand, in response to a positive determination at block 608 indicating that execution of the set of test cases 202 results in one or more failing test cases 314, the method 600 includes using the inverse CTD test case generation module 316 to execute and select certain failing test cases 314 and apply inverse combinatorics to the selected failing test cases 314 to create a new set of test cases 318 that can be used to detect and localize combinations of attribute values ​​that are causing n-wise errors or lower order errors.

[0066] In one or more embodiments of the invention, applying reverse combinatorial analysis to the selected failing test cases 314 includes generating a respective new test case 318 for each attribute. Thus, the number of new test cases generated may be equal to the number of attributes. In one or more embodiments, in each new test case 318, the attribute value in the selected failing test case 314 for the corresponding attribute is changed to the attribute value for the attribute that is not present in any other failing test case, and the respective attribute value for each other attribute is left unchanged from that present in the selected failing test case 314.

[0067] Referring again to the example shown in FIG. 4 , assuming that a failing test case 402 is selected in block 610, an example set of new test cases 406 may be generated. In one or more embodiments of the present invention, each example new test case 406 may be generated by changing the attribute value of the corresponding attribute in the selected failing test case 402 to a different value that is not present in any other failing test case. For example, a first new test case corresponding to the CTD vector <Andrew, Blue, Triangle> is obtained by changing the attribute value of the “Name” attribute in the failing test case 402 from “Dale” to “Andrew” while keeping the attribute values ​​for the other attributes the same as in the failing test case 402. Similarly, a second new test case corresponding to the CTD vector <Dale, Green, Triangle> is obtained by changing the attribute value of the “Color” attribute in the failing test case 402 from “Blue” to “Green” while keeping the attribute values ​​for the other attributes the same as in the failing test case 402. Finally, a third new test case corresponding to the CTD vector <Dale, Blue, Square> is obtained by changing the attribute value of the “shape” attribute in the failing test case 402 from “Triangle” to “Square” while keeping the attribute values ​​for the other attributes the same as in the failing test case 402.

[0068] It should be understood that each attribute value that is changed to obtain each new test case 406 is not explicitly present in the selected failing test case 402. Furthermore, each changed attribute value is not present in any other failing test case (e.g., failing test case 404). Specifically, the changed attribute value "Andrew" for the first new test case is not present in any failing test case, the changed attribute value "Green" for the second new test case is not present in any failing test case, and the changed attribute value "Square" for the third new test case is not present in any failing test case.

[0069] After generating a new set of test cases 318 by executing the inverse combination theory on the selected failed test case 314, the test case execution module 108 uses the new test case 318 in block 612 of method 600. Further, in block 614, one or more n-wise error localization modules 322 detect and localize n-wise errors or errors of lower order based on the attributes within the selected failed test case 314 and their corresponding original failure attribute values that result in new test cases that pass when changed. In particular, the computer-executable instructions of the n-wise error localization module 322 are executed to detect and localize n-wise errors or errors of lower order based on a new test case that results in a successful execution result by evaluating the execution results 320 for the new set of test cases 318. As used herein, an error of lower order refers to an m-wise error where m < n, assuming complete n-wise coverage by the set of CTD vectors 306.

[0070] Referring again to the example shown in FIG. 4 assuming pairwise error detection and localization, the execution of the new set of test cases 406 results in the first new test case corresponding to the CTD vector <Andrew, blue, triangle> passing, the second new test case corresponding to the CTD vector <Dale, green, triangle> passing, and the third new test case corresponding to the CTD vector <Dale, blue, square> failing. In one or more embodiments, since changing the attribute value from "triangle" in the failed test 402 to "square" in the third new test case results in a continued failure result, the n-wise error localization module 322 determines that the "shape" attribute is not a contributor to the pairwise error based on the failure execution result for the third new test case.

[0071] Meanwhile, the n-wise error localization module 322 may determine that an attribute whose attribute value was changed to obtain a new test case that passed execution contributes to the pairwise error. Specifically, in one or more embodiments of the present invention, based on the successful execution of the first new test case corresponding to the CTD vector <Andrew, blue, triangle>, the n-wise error localization module 322 determines that the “Name” attribute and the original failing attribute value “Dale” contribute to the pairwise error. Similarly, in one or more embodiments, based on the successful execution of the second new test case corresponding to the CTD vector <Dale, green, triangle>, the n-wise error localization module 322 determines that the “Color” attribute and the original failing attribute value “Blue” also contribute to the pairwise error.

[0072] Thus, based on the execution results 320 for the set of new test cases 318, the n-wise error localization module 322 may determine that the attributes "Name" and "Color," having attribute values ​​"Dale" and "Blue," respectively, are the source of the pair-wise error in the example shown in Figure 4. More generally, based on evaluation of the execution results 320 for the set of new cases 318, the n-wise error localization module 322 may determine the particular attribute-value pairs that cause the n-wise error or an error of a lower order, or more specifically, the original attribute values ​​that were changed in the failing test cases that were selected to obtain the new test cases that pass.

[0073] The example shown in FIG. 4 assumes an initial set of CTD vectors 400 that provides complete pairwise coverage, in which case a pairwise error or a lower-order error (e.g., a single attribute value causing an error) may be detected and localized in just one pass of applying reverse combinatorics to a selected failing test case to obtain a new set of test cases 318. In general, if the initial set of CTD vectors 306 provides n-wise coverage, just one pass of applying reverse combinatorics to a selected failing test case will reveal the n-wise error or a lower-order error. While a k-wise error for k>n may be detectable using an initial set of CTD vectors that provides complete n-wise coverage, this cannot be guaranteed because the initial set of CTD vectors 306 does not include all k-wise interactions. However, an initial set of CTD vectors 306 that provides full n-wise coverage (e.g., full pairwise coverage) may also provide some k-wise coverage (e.g., 3-wise coverage) for k>n, so that based on the particular CTD vectors selected, method 600 may reveal k-wise errors in just one pass or after multiple passes that provide reverse combinatorics for the selected failing test case.

[0074] 6, at block 616 of method 600, computer-executable instructions of one or more regression bucket generation modules 326 are executed to generate and output a regression bucket 212 of failing test cases that includes all possible combinations in the Cartesian product space that include the original failing attribute-value pairs. In one or more embodiments of the invention, the regression bucket generation module 326 receives as input an indication of the error-producing subset 324 of attribute-value pairs and determines and outputs a regression bucket 212 that includes only test cases that correspond to all possible combinations in the Cartesian product space that include the particular attribute value that is causing the detected error.

[0075] Referring again to the example of FIG. 4 , regression bucket generation module 326 receives as input the attributes “Name” and “Color” and the corresponding attribute values ​​“Dale” and “Blue” that are causing the pairwise error, determines all attribute value combinations in the Cartesian product space that include “Dale” and “Blue,” and populates regression bucket 212 with test cases corresponding to all of these combinations. In this example, regression bucket 212 contains the following test cases: (Dale, Blue, Triangle), (Dale, Blue, Circle), and (Dale, Blue, Square). Each of the test cases in regression bucket 212 contains the attribute values ​​“Dale” and “Blue” for the attributes “Name” and “Color,” respectively, which together cause the detected pairwise error, and each test case in regression bucket 212 is guaranteed to fail. In one or more embodiments of the present invention, regression bucket 212 may be output for use by a manual tester or another automated debugging algorithm. The regression bucket 212 of failing test cases can be used to verify whether a detected error has been corrected, because all test cases contained in the regression bucket 212 pass only when the error is corrected. Thus, if any test cases in the regression bucket 212 continue to fail despite modifications made to the SUT 214 in an attempt to correct the error, this indicates that the error has not been fully resolved.

[0076] 7 illustrates a flowchart of a method for detecting flaws in a test model using success rates of attribute-value pairs over time when testing an SUT, according to one or more embodiments of the present invention. Method 700 includes, at block 702, modeling attribute-value pairs for testing SUT 214. The modeling is similar to that performed in method 600, producing a dictionary of attributes and their corresponding values ​​that can be used to generate all combinations of test cases.

[0077] At block 704, the n-wise coverage CTD vector generation module 304 generates a set of CTD vectors 306 for testing the SUT 214 with n-wise coverage using the model. In one or more embodiments of the present invention, the CTD vectors 306 are generated from the entire Cartesian product space of attributes to be tested for the SUT 214. In an example embodiment of the present invention, each CTD vector 306 is selected exclusively from a reduced test space that excludes invalid combinations of attribute values ​​that violate architectural constraints of the SUT 214. For example, the CTD vector generation module 304 reduces the entire Cartesian product space associated with the set of attribute-value pairs 302 by identifying and excluding combinations that do not adhere to architectural constraints associated with the SUT 214.

[0078] At block 706, a set of test cases 202 is generated from the initial set of CTD vectors 306 and executed similarly to block 606 of method 600. In one or more embodiments of the present invention, generating and executing test cases 202 may include generating additional test cases 202 using reverse combination theory based on one or more failing test cases 202.

[0079] In block 708, the results of each test case 202 are recorded for at least a predetermined number of runs of the test case 202, i.e., repeated runs of the same set of test cases 202. The number of runs of a test case 202 may be limited using a predetermined number of runs, such as 10, 100, 1000, etc., or using runs over a predetermined period of time, e.g., hours, days, weeks, months, etc.

[0080] The results of a test case execution are binary, i.e., pass or fail. Accordingly, for a given configuration of the SUT 214, it can be expected that the results of the test case 202 will not change across multiple executions without any configuration changes. However, as discussed above, due to hidden attributes, the same test case that passes (or fails) in one execution with a particular configuration of the SUT 214 may fail (or pass) in another execution with the same configuration. Therefore, in block 710, the results of the test case 202 are converted to a non-binary format before storing the results.

[0081] Converting the results of the test cases 202 into a non-binary format is performed by calculating the success rate of the attribute-value pairs used by the test cases. In one or more embodiments of the present invention, the success rate is calculated using a pseudo-Boolean algebra, such as heightening algebra, to quantize a vector of Boolean results representing a pass / fail test case. In one or more embodiments of the present invention, an eigenvalue is calculated based on the success rate (i.e., heightening value) calculated for each attribute-value pair used by the test cases of the test model. All of the calculated success rates for the test model are collected into an eigenvector, which is then used to calculate the eigenvalue. The success rate (or failure rate) for each attribute-value pair (S AV ) is calculated based on the execution of each test case. Therefore, the success rate of attribute-value pairs (S AV ) is a non-binary value that describes the success / failure rate of a given attribute-value pair over time, where the success rate is calculated using pseudo-Boolean algebra. A vector containing the results of multiple runs of a test case with an attribute-value pair is denoted by S AV =f(t1,t2...tn), where S AV = success rate of attribute-value pair, ti = 1 / 0 based on success / failure of test case i, n = number of times test case is executed, and f = Boolean value ti for non-binary S AVwhere n may be the number of test cases that use the attribute-value pair, and the number of times the test case is executed.

[0082] In addition, the success rate of each test case (S T ) (or failure rate) is also monitored and recorded as a non-binary value. T ) is calculated using pseudo-Boolean algebra to quantize a vector of test case execution results over time. For example, a vector containing the results of multiple executions of a test case can be calculated as S T =g(t1,t2...tn), where S T = success rate of test case, ti = 1 / 0 based on success / failure of test case i, n = number of times test case is executed, and g = boolean ti to non-binary S T where n may be the number of times the test case is executed. In one or more embodiments of the present invention, eigenvalues ​​are calculated based on the success rates (i.e., height values) calculated for each test vector (i.e., test case) of the test model. All of the calculated success rates of the test model are collected into an eigenvector, which is then used to calculate the eigenvalues.

[0083] Tables 1 and 2 show example test vectors, where the attributes being tested include <name, color, state, age>, and the values ​​used are from the set shown in Table 1. It should be understood that this is merely an example, and that in other embodiments of the present invention, the test vectors may include different attributes and / or a different number of attributes in the test vector. Also, the values ​​of the attributes may vary from those shown herein. Table 1 shows the success rate for each attribute-value pair, and Table 2 shows the success rate for each test case. The success rate of the test case may be the same as the success rate of the test vector. The last column in Table 2 is the eigenvector of the pseudo-Boolean algebra, containing all of the success rates. The eigenvalues ​​are the numerical representations of the eigenvectors.

[0084] [Table 1]

[0085] [Table 2]

[0086] Note that a "hidden variable" may cause a test case to intermittently fail due to soft failures, such as the temperature of the SUT 214, in other applications, jobs, or services running concurrently with the SUT 214. A "hidden attribute" is an attribute that should be included in the model to generate test vectors with different combinations of attributes. Additionally, a "hidden attribute value" is a value that should be included to test attributes already in the model. That is, the combination of attributes does not change, but the combination of attribute values ​​used to test the SUT 214 changes. A "hidden value" is an environmental / external factor that affects the outcome of one or more test cases. A hidden variable is a type of hidden value.

[0087] In the example in Table 2, the test case using the attribute value combination represented by test vector #1 will fail 50% of the time, while test vectors #2 and #5 will result in the test case passing 100% of the time. Test vectors #1, #2, and #5 differ from each other by a single value, as shown.

[0088] For the sake of this example, assume that the combination of attribute values ​​<color, black> and <state, Connecticut> causes the test case to fail. If this were the case, then per pseudo-Boolean algebra, test cases with test vectors #2 and #5 would also have failed with a 50% success rate (identical to vector #1). However, since this is not the case, it can be assumed that there is a "hidden variable" in the model, i.e., that the model does not control for soft failures caused by unknown environmental conditions, for example. Furthermore, if all three test vectors #1, #2, and #5 result in a 50% success rate, it can be assumed that there is a "hidden attribute" that must be used to generate additional test vectors. Nevertheless, by adding one or more values ​​to the attribute's range and monitoring the success rate of the new attribute values ​​over time, a "hidden attribute value" can be detected if the success rate threshold is met by using one or more additional values. Also, the success rate of a particular test vector may change over time when a hidden variable exists and causes intermittent failures when the test case is executed.

[0089] At block 712, the success rate of the attribute-value pairs (S AV ) are stored in a data structure, such as a Karnaugh map, heat map, or any other form of heterogeneous data structure that can be analyzed to detect trends in attributes over time. The data structure is heterogeneous because each attribute can have a different number of values ​​that are tested.

[0090] FIG. 8 illustrates the success rate of attribute-value pairs (S AV ) visualization 800. The visualization 800 can be a heatmap, polygon, or success rate (S AV ) to visualize the non-uniform table using any other representation technique, such as success rate (S AV ) based on the data structure used to store the attribute-value pair. The visualization 800 is a non-uniform heat map, where the color or shading of each square 802 represents the success rate (S AVIn the particular example shown, the darker the square, the higher the success rate (S AV ) is lower, and conversely, the brighter the square, the higher the success rate (S AV ) is high. However, it should be understood that heatmap 800 may be shaded using different rules in other embodiments. Alternatively, or in addition, the success rate (S AV ) may be represented using different characteristics of the visualization other than color, for example, size. Furthermore, while visualization 800 is a 2D representation, it should be understood that in other embodiments of the invention, the visualization may use different dimensions, such as 3D.

[0091] By using visualization 800 to analyze a specific range of attributes, the effectiveness of each attribute can reveal a lack of depth in the attribute values ​​of the model. For example, the attribute "age" in visualization 800 may reveal the success rate (S) above (or below) a threshold for each attribute-value pair. AV ), the range of values ​​used to test the attribute may be underdeveloped. Otherwise, an attribute is considered to be sufficiently explored if adding additional values ​​to the attribute does not change the visualization 800 (i.e., the underlying success rate) by at least a predetermined threshold.

[0092] In block 714, the success rate is analyzed to determine deficiencies in the test model and / or data. The deficiencies may include hidden variables, hidden attributes, or hidden attribute values, or a combination thereof. In one or more embodiments of the present invention, the test model is used to generate test buckets that can be run. Using the results, an inverse combination set for each failing test vector can be calculated. Inverse combination theory is a known technique that facilitates identifying all failing test vectors. For each failing test vector, an inverse combination set is generated. For each inverse combination set, a new test bucket is generated to test and diagnose the SUT and determine the cause of the failing test case. Furthermore, in one or more embodiments of the present invention, the inverse test cases are run and the results are used to perform root cause analysis, thereby identifying the source of the failure in the original test model.

[0093] In one or more embodiments of the present invention, the analysis may include determining the success rate (S) of the test cases 202. T ) for example, S= T1 ,S T2 ...S TN represents the vector of success rates over time for test cases 202. Eigenvalues ​​and eigenvectors are calculated for vector S.

[0094] ​In one or more embodiments of the present invention, the eigenvectors may be visualized in multiple ways, such as line graphs. The eigenvectors and eigenvalues ​​indicate improvement and degradation of the testing process over time and also highlight focus on test case 202 execution #n+1. In one or more embodiments of the present invention, the eigenvectors representing the success rate of test cases may be used to generate unit vectors representing test model coverage over time. Furthermore, the eigenvectors and eigenvalues ​​may be used to monitor / track changes in the quality of results due to changes to existing test cases, the generation of new test cases, changes to the SUT 214, or any other changes. Thus, the eigenvalues ​​and eigenvectors may facilitate the quantification and monitoring of test case quality as the SUT 214 and test environment 100 change. This provides users with new metrics for maintaining and enhancing their testing infrastructure.

[0095] In one or more embodiments of the present invention, the success rate (S AV) are stored in the form of a Karnaugh map or any other data structure for visualization 800. Surface minima and maxima of the Karnaugh map may identify deficiencies in attribute-value pairs in testing by the current model and attribute-value pairs that are conversely covered by robustness, respectively. In one or more embodiments of the present invention, the test environment 100 may indicate that more or different resources may be needed to adequately test or stress the SUT 214. Furthermore, multiple such visualizations 800 obtained across multiple test case runs may be used to assess the SUT 214. For example, the visualization 800 may be compared to one or more versions of the visualization 800 to isolate patterns over time to indicate how much the SUT 214 and / or the test infrastructure 100 have matured and changed. The reference version used for comparison may be one or more visualizations 800 generated using the SUT 214 itself. Alternatively, or in addition, the reference visualization 800 may include a reference visualization generated offline using a different instance of the SUT 214.

[0096] In one or more embodiments of the present invention, clustering may be performed on each set of visualizations 800 in each release version of the SUT 214 to identify individual maps that transition from one cluster to another cluster that differs from the transitions of their peers. For example, hidden attributes may trigger localized hotspot transitions, which may involve assessing similar attributes / test coverage to determine whether the same failing attribute surfaces under multiple views.

[0097] Further analysis is performed to determine the success rate S of the first combination of values ​​in the test vector (e.g., Vector #1 in Table 1). TThe method further includes determining whether a test case 202, a test vector, or a test case 202 that uses the same test vector (e.g., vectors #2 and #5 in Table 1) deviates from other combinations of values ​​for the same test vector (e.g., vectors #2 and #5 in Table 1). If this condition is met, the hidden variable may be considered to be associated with the test case 202 that uses the first test vector. For example, if a first combination of attribute values ​​for a test vector has a first success rate, e.g., 50%, and all other combinations of attribute values ​​have a second success rate, e.g., 100%, then the first combination is considered to trigger a soft failure. In one or more embodiments of the invention, in block 716, the test case 202, test vector, and combination of attribute values ​​that triggers such a soft failure may be notified to the user.

[0098] Additionally, the success rate of a particular test case, S T is less than a predetermined threshold, e.g., 50%, over a predetermined number of test runs (or time period), then in block 718, one or more values ​​are added to the range of attribute values ​​and the success rate S of the new attribute values ​​over time is calculated. T ' by monitoring the success rate threshold S using one or more additional values T If the condition is met, a "hidden attribute value" can be detected.

[0099] Alternatively, or in addition, at block 718, the hidden attribute values ​​are calculated as a success rate (S) corresponding to a set of attribute-value pairs associated with the attribute that are all below a predetermined threshold. AV ) can be detected based on a set of attributes. The attributes can then be further diagnosed by adding more test vectors with additional attribute values ​​and running test cases with the additional attribute values. The success rate S of the test cases run with the additional attribute values ​​is AV If ' satisfies a predetermined threshold, a hidden attribute value condition has been identified and a hidden value has been discovered. The user is notified accordingly.

[0100] If the success rate threshold is not met, then in block 720, based on the determination of whether hidden attributes exist, the user may be notified of the hidden attributes or may be prompted to diagnose the SUT 214. That is, additional attributes may be required to execute the code path that is causing the test case to fail with a particular test vector.

[0101] 9 illustrates a flowchart of a method for determining whether hidden attributes are present in a current test model in accordance with one or more embodiments of the present invention. The method 900 calculates the success rate S of a test case. T is less than a predetermined threshold, for example, 50%. T The success rate S in all examples in this specification is 0%, since 0% indicates that the test case consistently fails. T Note that unless explicitly stated, is greater than 0%. Here, the test cases are passing and failing intermittently, and therefore the technical challenge is to determine whether the defect is in the SUT 214, the test cases, the environment (soft failure), or any other factor.

[0102] The method 900 includes, at block 902, generating an entire set of reverse-combination test cases based on the current model, the attribute-value pairs from the test vector, and the intermittently failing test cases 202. Known techniques for reverse CTD may be used to generate the set of test cases. The generated set of test cases has the same number of test cases as the intermittently failing test cases 202 and has a one-to-one correspondence between test case A from the original set of test cases and the corresponding test case A' in the reverse-combination test cases.

[0103] The inverse combination set is defined as a set of mutated test vectors in which one attribute value is replaced with another value from the original test model. For minimal mutation, n test vectors are generated, where n is the number of attributes in the initial failing test vector. For example, if the failing test vector is <Andrew, Black, New York, 20, 0> (see Table 2), the maximal inverse combination set is:

[0104] [Table 3]

[0105] The minimal inverse combination set in this example is

[0106] [Table 4]

[0107] At block 904, the set of reversed combination test cases is executed at least a predetermined number of times. At block 906, the success rate S of the set of reversed combination test cases is calculated. T The success rate of the set of reverse combination test cases S is recorded and stored. T ' is calculated using pseudo-Boolean algebra in the same way as the success rate of the original test case.

[0108] At block 908, the success rate S of the reverse combination test cases is calculated. T ' is the corresponding success rate S of the intermittently failing test case, i.e. the original test case. T In one or more embodiments of the present invention, the success rate S T The success rate of the eigenvalues ​​of the vector S T The comparison can be performed by comparing with the eigenvalues ​​of the vector of

[0109] If all of the failing tests in the reversed combination test cases fail at the same rate as in the original set of test cases, the test model may be deemed faulty at block 910. For example, if two eigenvalues ​​are identical, the two sets of test cases are deemed to be failing at the same rate. At block 912, the user is notified that in this case the test model needs to be diagnosed and revised.

[0110] Alternatively, if none of the test cases in the set of reverse-combination test cases fail in blocks 914 and 916, the hidden model attribute is considered to be added to the model and the user is notified accordingly. If the test cases have different failure rates and at least some of the test cases in the set of reverse-combination test cases fail, the user is notified in block 918 that the SUT 214 needs to be diagnosed to determine the cause of the failing test cases.

[0111] FIG. 10 illustrates a flowchart of a method for identifying portions of a SUT that drive hidden attributes, according to one or more embodiments of the present invention. As described above, the use of pseudo-Boolean algebra, eigenvectors, and non-uniform table heatmaps can help identify test cases that reveal that a model is missing one or more attributes, i.e., hidden attributes. The next technical challenge is determining which of these hidden attributes are relevant for a given model. Embodiments of the present invention address these technical challenges by using pseudo-Boolean algebra, eigenvectors, and non-uniform table heatmaps to identify and generate additional test cases that drive paths corresponding to lower pseudo-Boolean values ​​in the original set of test cases. Darker areas of visualization 800, representing attribute-value pairs with success rate values ​​below a predetermined threshold, indicate test cases that reveal hidden attributes in the model.

[0112] 10 and 1100 facilitate tracing the code paths of test case success and failure behavior using breakpoints to isolate portions, e.g., code paths, that execute those test cases. One or more portions, e.g., code paths, of SUT 214 are identified and highlighted as areas for further investigation to identify hidden attributes based on system configuration and timing windows.

[0113] The method 1000, at block 1002, determines whether a success rate (S) below a predetermined threshold is reached. AV ) in visualization 800. The attribute-value pairs are represented by darker shading, and in visualization 800, the success rate (S AV ) is used to determine the shading of the square, and S AV The lower the S, the darker the squares. In one or more embodiments of the present invention, a user selects one or more squares in visualization 800 to be further investigated using a graphical user interface (GUI). Alternatively, or in addition, a user may select one or more squares in visualization 800 that are below (or above) a threshold S AV to select squares with a threshold S AV may be provided.

[0114] Further, in block 1004, a subset of test cases from test cases 202 to be executed is identified, where the test cases from the subset use at least one attribute-value pair from the identified set of attribute-value pairs. Identifying the subset of test cases may be performed by scanning test cases 202 to identify test cases that use any of the identified attribute-value pairs. Alternatively, test execution engine 108 or test generator 208 maintains a record of test cases and corresponding attribute-value pairs.

[0115] A code path associated with the subset of test cases is determined at block 1006. The code path may be determined using the fingerprint 206 associated with each test case in the subset.

[0116] At block 1008, code path intersection regions are determined. The intersection regions further zoom or focus on portions of the SUT 214 that are causing the most intermittent success / failure of the test cases 202. In one or more embodiments of the invention, at block 1010, the code path intersection regions are highlighted in the source code of the SUT 214.

[0117] In one or more embodiments of the present invention, in block 1012, debug code is generated to diagnose the portions of the SUT 214 identified in this manner. The debug code may be provided to a user to apply and diagnose soft failures or any other malfunctions that cause intermittent success / failure of test cases. For example, if the SUT 214 is based on z / OS®, a command such as SLIP or any other such command used to investigate performance-related issues may be used to generate the debug code. It should be understood that if the SUT 214 uses any other architecture or operating system, performance-related commands from that architecture may be used to generate the debug code.

[0118] 11 illustrates a flowchart of a method for identifying portions of a SUT that drive hidden attributes in accordance with one or more embodiments of the present invention. At block 1102, method 1100 determines whether a success rate (S) below a predetermined threshold is reached. AV ) in visualization 800. The attribute-value pairs are represented by darker shading, and in visualization 800, the success rate (S AV ) is used to determine the shading of the square, and S AVThe lower the S, the darker the squares. In one or more embodiments of the present invention, a user selects one or more squares in visualization 800 to be further investigated using a graphical user interface (GUI). Alternatively, or in addition, a user may select one or more squares in visualization 800 that are below (or above) a threshold S AV to select squares with a threshold S AV may be provided.

[0119] Further, in block 1104, a subset of test cases from test cases 202 to be executed is identified, where test cases from the subset use at least one attribute-value pair from the identified set of attribute-value pairs. Identifying the subset of test cases may be performed by scanning test cases 202 to identify test cases that use any of the identified attribute-value pairs. Alternatively, test execution engine 108 or test generator 208 maintains a record of test cases and corresponding attribute-value pairs.

[0120] For each test case in the subset of test cases, a first code path associated with successful execution of the test case is determined and a second code path associated with unsuccessful execution of the test case is determined in block 1106. The code paths may be determined using the fingerprint 206 associated with each test case in the subset.

[0121] In block 1108, the first and second code paths are compared to identify portions of the code paths that differ, and those portions are highlighted for inspection by the user. In one or more embodiments of the present invention, the code paths may be compared using a text comparison to compare the source code that is within the code paths. Highlighting portions of the code paths may be performed by sending commands to a graphical user interface that identify portions of the source code to be marked and highlighted.

[0122] In one or more embodiments of the present invention, in block 1110, debug code is generated to diagnose the portions of the SUT 214 identified in this manner. The debug code may be provided to a user to apply and diagnose soft failures or any other malfunctions that cause intermittent success / failure of test cases. For example, if the SUT 214 is based on z / OS®, a command such as SLIP or any other such command used to investigate performance-related issues may be used to generate the debug code. It should be understood that if the SUT 214 uses any other architecture or operating system, performance-related commands from that architecture may be used to generate the debug code.

[0123] Embodiments of the present invention therefore facilitate focusing artistic testing on darker (or lighter) regions in visualization 800 because those regions highlight defects that cause test cases to pass / fail more intermittently compared to other regions of visualization 800. Embodiments of the present invention further facilitate using breakpoints, fingerprinting, or other techniques to highlight regions in the source code of SUT 214 to be further investigated based on visualization 800 that show weaknesses.

[0124] Using pseudo-Boolean algebra, such as Heiting's algebra, test system 100 can quantize the success rate of test cases and encapsulate hidden variables that consistently affect test case outcomes. This allows users to more reliably identify when test cases are failing due to factors beyond the test infrastructure itself, so that resources can be adjusted or new test infrastructure can be developed accordingly. This enhanced success metric can be used to improve the efficiency and reliability of stability testing of SUTs 214.

[0125] Referring now to FIG. 12 , a computer system 1200 is generally illustrated in accordance with an embodiment of the present invention. Computer system 1200 may be an electronic computer framework that includes and / or utilizes any number and combination of computing devices and networks utilizing various communication technologies, as described herein. Computer system 1200 may be easily scalable, extensible, and modular, with the ability to adapt to different services or reconfigure some features independently of others. Computer system 1200 may be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer system 1200 may be a cloud computing node. Computer system 1200 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system 1200 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0126] As shown in FIG. 12, computer system 1200 includes one or more central processing units (CPUs) 1201a, 1201b, 1201c, etc. (collectively or generally referred to as processors 1201). Processor 1201 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processor 1201, also referred to as a processing circuit, is coupled to system memory 1203 and various other components via a system bus 1202. System memory 1203 may include read-only memory (ROM) 1204 and random access memory (RAM) 1205. ROM 1204 is coupled to system bus 1202 and may include a basic input / output system (BIOS), which controls certain basic functions of computer system 1200. RAM is read-write memory coupled to system bus 1202 for use by processor 1201. System memory 1203 provides temporary memory space for the execution of these instructions during operation. The system memory 1203 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0127] Computer system 1200 includes an input / output (I / O) adapter 1206 and a communications adapter 1207 coupled to a system bus 1202. I / O adapter 1206 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 1208 and / or any other similar components. I / O adapter 1206 and hard disk 1208 are collectively referred to herein as mass storage 1210.

[0128] Software 1211 for execution on computer system 1200 may be stored in mass storage 1210. Mass storage 1210 is an example of a tangible storage medium readable by processor 1201. Software 1211 is stored as instructions for execution by processor 1201 to operate computer system 1200, as described herein below with reference to the various figures. Examples of computer program products and the execution of such instructions are discussed in more detail herein. Communications adapter 1207 interconnects system bus 1202 with network 1212, which may be an external network, allowing computer system 1200 to communicate with other such systems. In one embodiment, portions of system memory 1203 and mass storage 1210 collectively store an operating system, which may be any suitable operating system, such as IBM® Corporation's z / OS® or AIX® operating systems, for coordinating the functions of the various components shown in FIG. 12 .

[0129] Additional input / output devices are shown connected to system bus 1202 via display adapter 1215 and interface adapter 1216. In one embodiment, adapters 1206, 1207, 1215, and 1216 may be connected to one or more I / O buses connected to system bus 1202 via an intermediate bus bridge (not shown). A display 1219 (e.g., a screen or display monitor) is connected to system bus 1202 by display adapter 1215, which may include a graphics controller for improving performance of graphics-intensive applications and a video controller. A keyboard 1221, mouse 1222, speakers 1223, etc. may be interconnected to system bus 1202 via interface adapter 1216. Interface adapter 1216 may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Thus, as configured in Figure 12, computer system 1200 includes processing capabilities in the form of processor 1201, storage capabilities including system memory 1203 and mass storage 1210, input means such as keyboard 1221 and mouse 1222, and output capabilities including speakers 1223 and display 1219.

[0130] In some embodiments, communications adapter 1207 may transmit data using any suitable interface or protocol, such as an Internet Small Computer System Interface, among others. Network 1212 may be a cellular network, a wireless network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. External computing devices may connect to computer system 1200 through network 1212. In some examples, the external computing device may be an external web server or a cloud computing node.

[0131] It should be understood that the block diagram of Figure 12 is not intended to indicate that computer system 1200 includes all of the components shown in Figure 12. Rather, computer system 1200 may include any suitable fewer or additional components (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.) not shown in Figure 12. Furthermore, the embodiments described herein with respect to computer system 1200 may be implemented with any suitable logic. Logic, as referred to herein, in various embodiments, may include any suitable hardware (e.g., a processor, embedded controller, or application-specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware.

[0132] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementations of the teachings recited herein are not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0133] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or service provider interaction. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0134] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with the service provider. Wide network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., cell phones, laptops, and PDAs). Resource Sharing: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence is implied in that consumers generally have no control or knowledge over the exact location of the resources provided, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: Capabilities can be rapidly and elastically provisioned to instantly scale out, and rapidly released to instantly scale in, sometimes automatically. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time. Services are meterable: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0135] The service model is as follows: Software as a Service (SaaS): The capability offered to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to the consumer is to deploy consumer-created or consumer-acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, to the extent possible, the application hosting environment configuration. Infrastructure as a Service (IaaS): The capability offered to the consumer is the supply of processing, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does exercise control over the operating systems, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0136] The deployment model is as follows: Private Cloud: The cloud infrastructure is operated exclusively for an organization. It may be managed by the organization or a third party and may reside on-premise or off-premise. Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party and may reside on-premises or off-premises. Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services. Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0137] Cloud computing environments are stateless, low-coupling, modular, and service-oriented with a focus on semantic interoperability. At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes.

[0138] Referring now to FIG. 13, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or any combination thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks (not shown), such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or any combination thereof, as described above. This enables the cloud computing environment 50 to offer infrastructure, platform, or software, or any combination thereof, as a service without the cloud consumer having to maintain resources on their local computing device. It should be understood that the types of computing devices 54A-N shown in FIG. 13 are intended to be exemplary only, and that the computing node 10 and cloud computing environment 50 may communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0139] Referring now to Figure 14, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 13) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 14 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0140] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based server 62, server 63, blade server 64, storage device 65, and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0141] The virtualization layer 70 provides an abstraction layer at which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0142] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides protection for data and other resources as well as identity verification for cloud consumers and tasks. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to ensure requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources in anticipation of future requirements according to SLAs.

[0143] The workload layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and testing 96.

[0144] In one or more embodiments of the present invention, the computer system may be environment 100 in which SUT 214 is a computer program or hardware component used by the computer system. In one or more embodiments of the present invention, the computer system may be SUT 214, where SUT 214 is part of a server cluster.

[0145] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0146] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, punch cards, or mechanically encoded devices such as ridge structures in grooves that allow instructions to be recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through electrical wires.

[0147] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0148] Computer-readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to perform aspects of the present invention.

[0149] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0151] Computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0152] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0153] The description of various embodiments of the present invention has been presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond those found in the marketplace, or to enable others skilled in the art to understand the embodiments described herein.

[0154] Various embodiments of the present invention are described herein with reference to the associated drawings. Alternate embodiments of the present invention may be devised without departing from the scope of the present invention. Various connections and relationships (e.g., above, below, adjacent, etc.) are described between elements in the following description and in the drawings. These connections and / or relationships may be direct or indirect unless otherwise specified, and the present invention is not intended to be limited in this respect. Thus, connections between elements may refer to either direct or indirect connections, and relationships between elements may be direct or indirect relationships. Furthermore, various tasks and processing steps described herein may be combined into a more comprehensive procedure or process having additional steps or functionality not described in detail herein.

[0155] The following definitions and abbreviations shall be used for interpreting the claims and the specification. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," or "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or device that includes a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or inherent in such composition, mixture, process, method, article, or device.

[0156] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" may be understood to include any integer greater than or equal to one, i.e., 1, 2, 3, 4, etc. The term "plurality" may be understood to include any integer greater than or equal to two, i.e., 2, 3, 4, 5, etc. The term "connected" may include both an indirect "connected" and a direct "connected."

[0157] The terms "about," "substantially," "approximately," and variations thereof are intended to include the degree of error associated with measurement of a particular quantity based on equipment available at the time of filing this application. For example, "about" can include a range of ±8%, or 5%, or 2% of a given value.

[0158] For the sake of brevity, the prior art related to making and using aspects of the present invention may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs that implement various technical features described herein are well known. Therefore, for the sake of brevity, details of many conventional implementations are only briefly described herein or omitted entirely without providing details of well-known systems and / or processes.

Claims

1. 1. A processor-implemented method for detecting and localizing defects when testing a system under test (SUT), comprising: modeling input to the SUT as a set of attribute-value pairs; generating an initial set of test vectors that provide complete n-wise coverage of the test space represented by the attribute-value pairs; generating a set of test cases from the initial set of test vectors; executing the set of test cases to obtain a set of execution results, the execution results indicating in binary form whether a test case passed or failed, and the set of test cases being executed multiple times; For each execution of the set of test cases, a non-binary success rate (S) is calculated based on the execution results. ＡＶ ) for each attribute-value pair, wherein the non-binary success rate of an attribute-value pair is based on the results of execution of each test case that uses the attribute-value pair; selecting an attribute in response to a set of attribute-value pairs including the attribute having corresponding success rates all below a predetermined threshold; generating a second set of test vectors including additional values ​​for the selected attribute; executing the set of test cases to obtain a second set of execution results using the second set of test vectors, wherein the set of test cases is executed at least a predetermined number of times; and For each execution of the set of test cases, a second non-binary success rate (S) is calculated based on the execution results. ＡＶ ') for each attribute-value pair; notifying a user of the detected additional value for the attribute in response to a second set of success rates corresponding to the set of attribute-value pairs including the attribute satisfying the predetermined threshold; A method comprising:

2. The non-binary success rate S ＡＶ The method of claim 1 , wherein is calculated using pseudo-Boolean algebra.

3. outputting to the user an identification of one or more test cases that use the set of attribute-value pairs to be used to diagnose soft failures associated with the SUT in response to the second set of success rates corresponding to the set of attribute-value pairs that includes the attribute not satisfying the predetermined threshold; The method of claim 1.

4. The method of claim 3 , wherein soft failures are caused by one or more operating conditions of the SUT.

5. The method of claim 1 , wherein a Cartesian product space containing all possible combinations of the attribute-value pairs is reduced to a reduced test space using at least in part combinatorial theoretic test design.

6. The method of claim 5 , wherein the Cartesian product space is further reduced based on architectural constraints of the SUT.

7. 7. The method of claim 6, wherein the architectural constraints include at least one of a requirement that a first attribute cannot take on one or more candidate attribute values ​​if a second attribute has a particular attribute value, a requirement that the first attribute must take on a particular candidate attribute value if the second attribute has the particular attribute value, or a requirement to introduce a third attribute if the second attribute has the particular attribute value.

8. 1. A system comprising: Memory and a processor coupled to the memory, the processor configured to execute a method for detecting and confining defects when testing a system under test (SUT), the method comprising: modeling input to the SUT as a set of attribute-value pairs; generating an initial set of test vectors that provide complete n-wise coverage of the test space represented by the attribute-value pairs; generating a set of test cases from the initial set of test vectors; executing the set of test cases to obtain a set of execution results, the execution results indicating in binary form whether a test case passed or failed, and the set of test cases being executed multiple times; For each execution of the set of test cases, a non-binary success rate (S) is calculated based on the execution results. ＡＶ ) for each attribute-value pair, wherein the non-binary success rate of an attribute-value pair is based on the results of execution of each test case that uses the attribute-value pair; selecting an attribute in response to a set of attribute-value pairs including the attribute having corresponding success rates all below a predetermined threshold; generating a second set of test vectors including additional values ​​for the selected attribute; executing the set of test cases to obtain a second set of execution results using the second set of test vectors, wherein the set of test cases is executed at least a predetermined number of times; and For each execution of the set of test cases, a second non-binary success rate (S) is calculated based on the execution results. ＡＶ ') for each attribute-value pair; In response to a second set of success rates corresponding to the set of attribute-value pairs that includes the attribute satisfying the predetermined threshold, notifying a user of the detected additional value for the attribute and outputting to the user an identification of one or more test cases that use the set of attribute-value pairs, the one or more test cases being used to diagnose soft failures associated with the SUT; the processor, A system comprising:

9. The non-binary success rate S ＡＶ The system of claim 8 , wherein is calculated using pseudo-Boolean algebra.

10. 9. The system of claim 8, wherein the method further comprises, in response to the second set of success rates corresponding to the set of attribute-value pairs that includes the attribute not meeting the predetermined threshold, outputting to the user an identification of one or more test cases that use the set of attribute-value pairs, the one or more test cases being used to diagnose a soft failure associated with the SUT.

11. The system of claim 10 , wherein a soft failure is caused by one or more operating conditions of the SUT.

12. The system of claim 8 , wherein a Cartesian product space containing all possible combinations of the attribute-value pairs is reduced to a reduced test space using at least in part combinatorial theory test design.

13. The system of claim 12 , wherein the Cartesian product space is further reduced based on architectural constraints of the SUT.

14. 14. The system of claim 13, wherein the architectural constraints include at least one of a requirement that a first attribute cannot take on one or more candidate attribute values ​​when a second attribute has a particular attribute value, a requirement that the first attribute must take on a particular candidate attribute value when the second attribute has the particular attribute value, or a requirement to introduce a third attribute when the second attribute has the particular attribute value.

15. 1. A computer program product that, when executed by a processor, causes the processor to perform a method for detecting and confining defects when testing a system under test (SUT), comprising: modeling input to the SUT as a set of attribute-value pairs; generating an initial set of test vectors that provide complete n-wise coverage of the test space represented by the attribute-value pairs; generating a set of test cases from the initial set of test vectors; executing the set of test cases to obtain a set of execution results, the execution results indicating in binary form whether a test case passed or failed, and the set of test cases being executed multiple times; For each execution of the set of test cases, a non-binary success rate (S) is calculated based on the execution results. ＡＶ ) for each attribute-value pair, wherein the non-binary success rate of an attribute-value pair is based on the results of execution of each test case that uses the attribute-value pair; selecting an attribute in response to a set of attribute-value pairs including the attribute having corresponding success rates all below a predetermined threshold; generating a second set of test vectors including additional values ​​for the selected attribute; executing the set of test cases to obtain a second set of execution results using the second set of test vectors, wherein the set of test cases is executed at least a predetermined number of times; and For each execution of the set of test cases, a second non-binary success rate (S) is calculated based on the execution results. ＡＶ ') for each attribute-value pair; In response to a second set of success rates corresponding to the set of attribute-value pairs that includes the attribute satisfying the predetermined threshold, notifying a user of the detected additional value for the attribute and outputting to the user an identification of one or more test cases that use the set of attribute-value pairs, the one or more test cases being used to diagnose soft failures associated with the SUT; A computer program that executes the following:

16. The non-binary success rate S ＡＶ 16. The computer program of claim 15, wherein is calculated using pseudo-Boolean algebra.

17. 16. The computer program product of claim 15, wherein the method further comprises: outputting to the user, in response to the second set of success rates corresponding to the set of attribute-value pairs that includes the attribute, failing to meet the predetermined threshold, an identification of one or more test cases that use the set of attribute-value pairs, the one or more test cases being used to diagnose a soft failure associated with the SUT.

18. 20. The computer program product of claim 17, wherein a soft failure is caused by one or more operating conditions of the SUT.

19. 16. The computer program product of claim 15, wherein the Cartesian product space containing all possible combinations of attribute-value pairs is reduced to a reduced test space using at least in part combinatorial theoretic test design.

20. 20. The computer program product of claim 19, wherein the Cartesian product space is further reduced based on architectural constraints of the SUT.

Images