Artificial intelligence-enabled output space exploration for guiding test case generation

Abstract

A method for testing a software application in a system under test (SUT) includes constructing a reference model of the SUT that includes a computer-based neural network; training the reference model using input data and corresponding output data generated by the SUT; selecting an output value within a domain of possible output values of the SUT, the output value representing an output not represented in the output data used to train the reference model; applying the selected output value to the reference model and tracing the selected output through the reference model to identify a test input value that, when input to the reference model, produces the selected output value; and testing the system under test using the identified test input value.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 626,890, entitled “Booted Test Case Generation by Enabling AI Output Space Exploration,” filed February 6, 2018, and U.S. Provisional Patent Application Serial No. 62 / 628,557, entitled “Output Space Exploration for Black-Box Testing,” filed February 9, 2018, which are incorporated herein by reference. Technical Field

[0003] This application relates to the testing of systems. More specifically, this application relates to testing software in industrial systems. Background Technology

[0004] Software verification is crucial for delivering safe and high-quality products. In industry, testing represents the primary verification method used in software-intensive systems. As the complexity of these systems increases, the number of test cases requiring inspection increases proportionally. Therefore, identifying test cases that thoroughly test every feature of the software—those that may uncover defects critical to system quality—is essential.

[0005] The ability of a given test case to detect defects depends on the test inputs and expected outputs. This defines the test case. Test inputs are provided to the system under test (SUT) to trigger some execution of the underlying software system. The expected outputs, after processing the inputs, provide the state of the system, thus improving the system. In some cases, different test inputs can produce the same or similar system outputs. To be certain that all errors in the system are found, it is crucial to ensure that the tests meet each of the various output conditions defined by the system.

[0006] The challenge of fully testing the system under test (SUT) involves finding appropriate test inputs that produce specific system responses. The selected inputs need to generate outputs that cover as much of the system response space as possible. Furthermore, once testing has begun, the selection of additional test cases should focus on outputs that have not yet been tested. Other aspects of selecting test outputs may include focusing on values ​​close to the constraints imposed on the system. For example, the output values ​​to be tested may include extreme cases based on system specifications. Additionally, inputs should be able to be generated automatically without requiring human intervention from experts.

[0007] Several methods support determining the test inputs of interest from a black-box perspective (i.e., when internal system information is unavailable). In this regard, the most prevalent approach in industry is to perform input selection with the assistance of domain experts. Experts analyze the specification documents and use their understanding of the system to identify extreme cases to create relevant inputs that test different combinations of inputs at extreme levels, sometimes exceeding normal operating limits. While this can be an effective method, it is also very expensive and time-consuming. Furthermore, the quality of the results depends heavily on the experience of the expert or tester.

[0008] Other black-box testing methods for deriving test inputs include equivalent segmentation and randomization. Equivalent segmentation divides the system's input data into segments of equivalent data, assuming that equivalent data will produce similar system output responses. On the other hand, randomization methods, including stochastic simulation and fuzzing, work by generating a large number of possible inputs for the system to run. However, these methods focus on the inputs and do not consider the desired output when identifying the inputs to be tried. While the generation of these inputs can be initially guided, it is difficult to obtain good inputs for very specific output conditions due to the dependencies between the data and other factors. These methods require a high level of domain knowledge. Methods that focus on coverage criteria in the input space cannot provide any reliable guarantees about the degree of coverage in the output space. The implementations described herein provide alternative methods to automate the process of generating inputs that produce selected outputs. Summary of the Invention

[0009] A method for testing software applications in a system under test (SUT) includes: constructing a reference model of the SUT based on a computer-based neural network; training the reference model using input data and corresponding output data generated by the SUT; selecting an output value within a domain of possible output values ​​of the SUT, the output value representing an output not represented in the output data used to train the reference model; applying the selected output value to the reference model and tracking the selected output through the reference model to identify a test input value that produces the selected output value when input to the reference model; and testing the system under test using the identified test input value. According to an embodiment, the method further includes providing the determined input value to the reference model to generate the identified output value, providing the determined input value to the SUT to generate a test output value from the SUT, and comparing the identified output value with the test output value.

[0010] According to other embodiments, the method further includes determining whether a comparison indicates that the difference between the identified output value and the test output value exceeds a predetermined threshold. If so, the test output value is analyzed to determine whether the test output value is a legitimate system output or whether the difference indicates an error in the system. If the test output value is determined to be a legitimate value, the neural network can be retrained using a dataset containing the determined input and the identified output values, and if the test output value is determined to be an illegitimate system output, the system under test is analyzed to identify the error responsible for creating the illegitimate system output, and the identified error is corrected.

[0011] To identify output values ​​from the output space of a software application, the output generator receives the current distribution and the target distribution; and new, unexplored output values ​​or multiple outputs in the output vector y'. In other implementations, the output generator applies constraints on the adjacency region of the input space and on the output space to select only new output vectors from the unexplored output space. The output generator can further select new output vectors y' by modifying existing outputs from the training set with context-sensitive large values ​​and by visually comparing the distribution maps of the current output space coverage and the target output space distribution.

[0012] A system for testing software applications in a System Under Test (SUT) includes: a computer processor communicating with the SUT; a memory communicating with the computer processor, the memory containing executable instructions that, when executed by the processor, cause the processor to create a data structure representing a reference model approximating the functionality of the SUT; identifying output values ​​from the output space of the software application, the output values ​​being output values ​​not yet observed through previous testing; submitting the identified output values ​​to the reference model and tracking the output values ​​through the reference model to determine at least one input value or a vector of input values ​​that produces one or more of the identified output values ​​when input to the reference model. The reference model is implemented as an artificial neural network (ANN). In some embodiments, the executable instructions further include instructions to cause the processor to perform: providing the determined input values ​​to the reference model to generate the identified output values, providing the determined input values ​​to the SUT to generate test output values ​​from the SUT, and comparing the identified output values ​​with the test output values. According to some implementations, the executable instructions further include instructions to cause the processor to perform: when a comparison instruction indicates that the difference between the identified output value and the test output value exceeds a predetermined threshold, analyze the test output value to determine whether the test output value is a legitimate system output or indicates an error in the system. If the test output value is determined to be a legitimate value, the neural network is retrained using a dataset containing the determined input value and the test output value. Otherwise, if the test output value is determined to be an invalid system output, the system under test is analyzed to identify errors responsible for creating the invalid system output, and the identified errors are corrected.

[0013] Identifying output values ​​from the output space of these software applications can be performed via an output generator configured to receive the current distribution and the target distribution, and output a new, unexplored output vector y'. The output generator can be configured to apply constraints to the adjacency region of the input space and to apply constraints to the output space to select only new output vectors from the unexplored output space. Attached Figure Description

[0014] The foregoing and other aspects of the invention are best understood from the following detailed description when read in conjunction with the accompanying drawings. For the purpose of illustrating the invention, presently preferred embodiments are shown in the drawings; however, it should be understood that the invention is not limited to the specific means disclosed. The drawings include the following figures:

[0015] Figure 1 This is a block diagram of a system and method for creating test cases in a system under test, according to various aspects of specific embodiments of this disclosure.

[0016] Figure 2 This is a flowchart illustrating various aspects of creating and analyzing test cases for the system under test, based on specific embodiments of this disclosure.

[0017] Figure 3 This is a flowchart illustrating various aspects of exploring the output space of a system under test according to specific embodiments of this disclosure.

[0018] Figure 4 This is a flowchart illustrating a method for creating and analyzing test cases for a system under test, based on various aspects of specific embodiments of this disclosure.

[0019] Figure 5 This is a block diagram of a computer system for implementing systems and methods for creating and analyzing test cases for a system under test, according to various aspects of specific embodiments of this disclosure.

[0020] Figure 6A and Figure 6B It is a graphic depiction of visual comparison of output spatial coverage according to various aspects of specific embodiments of the present disclosure. Detailed Implementation

[0021] Figure 1 An overview of the method according to embodiments of this disclosure is provided. The entire process can be distinguished by three main phases: training 100, querying 120, and evaluation 130.

[0022] In the first stage (training) 100, a neural network SUT that approximates the system under test (SUT) 105 was created. NN 111. Train a neural network 109 using a dataset 101 of the input-output relationships (resp.x, and y) of SUT 105. The dataset 101 is obtained from existing or real-world scenarios that have been validated as correct. Based on the available dataset 101, the training 109 of the network can also be scaled to a subset of features of interest from the entire SUT 105. These scenarios can be described by different sources, such as test cases from validated or operational log files. Neural Network SUT NN 111 learns the input-output relationship by feeding it a dataset 101. The training phase 100 is completed when the neural network 111 is able to approximate the existing data with a 'good' level of accuracy, where 'good' is defined for each individual SUT 105 to which the method is applied.

[0023] In the second stage (query) 120, a new output value y'121 is introduced and used to utilize the neural network 111 to obtain the input x'126, such that x'126 is fed to the SUT. NN111, generating y'121. Modify Δ on the existing output y to obtain y' = y + Δ, thus deriving the definition of the new output y'121. By checking the current output space coverage with the existing input, select y'120 from the uncovered or weakly covered output space. This task can be performed manually or automated by generating randomly selected uncovered output values ​​or systematically selecting the value furthest from the existing covered output values. To receive x'126, explore the neural network 111 in reverse order 124, i.e., compute NN 111 in reverse.

[0024] In the third stage (Eval) 130, the newly derived input value x'126 from the new output y'120 is submitted to SUT 105 to produce the test output y'(SUT) 134. The test output 134 is then compared with the result from the neural network SUT. NN 111 generates the predetermined output y'(SUT) NN Compare 136 with 138. If the two outputs y'(SUT)134 and y'(SUT)138 are... NN If 136 is inconsistent, experts can examine the cause of the unexpected result. This reduces the time required by experts using traditional methods, as they can focus solely on the mismatched result. The reason why the test output 134 and the predetermined output 120 are not equal to each other can be explained by at least two possibilities. First, the unequal outputs may be correct (not due to a software error), but this scenario has not yet been learned by the neural network 111. This training phase can be revisited to add new scenarios to the neural network 111. Second, this difference can be attributed to a fault in the SUT (e.g., a software error in the SUT's control system). In this case, the fault can be analyzed, identified, and corrected.

[0025] Otherwise, in the two outputs y'(SUT) and y'(SUT) NN If the conditions are consistent, we have identified an input x' that has been processed to the same result, meaning that the output from which new explorations of test cases are derived works correctly. In this case, we add the valid value to our set of test cases and repeat the process in the query phase to qualify another output y'.

[0026] Figure 2A method for generating test cases for a software application in a system under test (SUT) according to a specific embodiment of this disclosure is described. A neural network providing a reference model to the SUT is trained 201 using a dataset including inputs and resulting outputs implemented in data generated by the actual SUT. Based on selected output values, the neural network is queried and evaluated in reverse order to identify input values ​​that will produce the reference model neural network generating the selected output 203. Evaluation 205 is performed on the input values ​​identified in step 203. Evaluation may include comparing the selected output generated by feeding the identified inputs into the neural network with the output created by the physical SUT when the same inputs are provided to the SUT. If the output from the neural network matches the output from the SUT 207, a test case is added to the test case set for the SUT, and this test case increases the coverage of the output space for the SUT. If the outputs from the neural network and the SUT do not match 209, an expert analyzes this difference 211. If the result generated by the SUT is found to be valid, meaning the input / output dataset represents a valid scenario that was not previously submitted to the neural network during training, the neural network training phase is re-entered to improve the reference model 213. If it is determined that the difference is actually due to an error within the system under test 215, then locate and correct the error, and add the correct input / output dataset to the test case set 217.

[0027] The system and method described above offer advantages over previous approaches that required manual intervention to select test cases from the perspective of the input domain space. These traditional techniques rely on educated guesswork rather than systematically focusing on the output space. By starting with the output space and targeting untested or tested regions within it, appropriate inputs can be identified without resorting to random trials to produce the desired output or requiring extensive data processing to handle numerous input combinations. In this way, the test design of the system can be automated, especially for systems involving black-box testing, such as legacy systems where the internal functionality of the system is unknown. The ability to perform output space exploration increases coverage of unexplored test scenarios and discovers test cases based on the system's output space. This, in turn, increases fault detection from the derived test cases. Furthermore, reduced test execution time is achieved because fewer test cases are required since they are specifically selected to target specific parts of the output space.

[0028] The improved systems and methods disclosed herein provide technical features to improve the ability to test and verify software systems (such as software systems that provide control for industrial systems). For example, the automatic generation of input values ​​for a given system output can be identified and tested. Coverage of the output space is systematically increased by leveraging trained neural networks. Output / input pairs of interest in the test system are identified, such as output values ​​not covered by existing test suites. The identification of input vectors that may cause system failures is increased, and test predictions are automatically generated for the predicted test cases. Using the disclosed systems and methods, higher quality products can be released due to increased test coverage. Products in which the source code is unavailable or only partially available (e.g., black box) or not fully understood (e.g., third-party products, legacy code) can be verified.

[0029] Black-box software testing is an increasingly crucial part of quality assurance for industrial products. This is particularly prevalent when the source code of the system under test is unavailable, often in cases involving legacy systems, third-party software, or when the system's behavior is too complex to be represented by a reusable design artifact used for testing. To verify the reliable behavior of such software-intensive systems, testing needs to ensure that the system produces correct outputs based on different inputs defined by corresponding input / output (In / Out) relationships. Even more critically, it needs to guide the testing to detect errors. The probability of finding errors is highest in previously untested areas, including unexpected extreme cases. The better the coverage of the input / output domains, the higher the likelihood that the product will be error-free.

[0030] Black-box testing, without knowledge of the internals of the System Under Test (SUT), helps automate test data generation. Traditionally, test data is generated randomly or through some form of system input space exploration. Input space exploration is based on using a fault ontology or partitioning and control strategies to guide the exploration. However, the nature of this input space exploration makes both the probability and timeframe of finding relevant faults highly uncertain. This leads to inefficiency in thoroughly testing the system. Furthermore, the coverage of the input space is not necessarily related to or indicative of similar coverage of the output space.

[0031] Existing testing techniques, such as equivalence class testing or combinatorial test case generation, exist within the industry. While these techniques are generally effective at providing coverage of combinations of test input sets or configuration settings, they struggle to cover all outputs of the system associated with the test inputs. Other test generation or test suite simplification techniques can be based on the discovery of the system's input / output (In / Out) relationships. These methods rely solely on generating the desired results from stimulated input data but do not guarantee the discovery of new input / output relationships and, consequently, a thorough exploration of the output space domain. Systematic testing techniques also require considerable time and resources for test case generation and execution. In contrast, random testing is a frequently used alternative approach, more feasible for running large numbers of test cases due to its simplicity and cost-effectiveness. Empirical comparisons show that random testing and segmented testing are equally effective. Adaptive Random Testing (ART) aims to randomly select test cases but also to distribute them evenly. The theory behind ART is that new test inputs further away from test inputs that did not reveal faults are more likely to reveal faults. ART has been proposed to improve the fault detection capabilities of random testing. ART utilizes information about the executed test cases (especially successful ones) to guide the generation of random test inputs. Because faulty inputs often cluster together to form one or more adjacent regions, subsequent test inputs that are close to (or similar to) successful test cases are less likely to hit the faulty region compared to those that are farther away. Therefore, ART promotes diversity among random test cases.

[0032] This disclosure provides a novel, intelligent test case generation method with a higher probability of exposing system faults. This is achieved by systematically exploring the output space of the system under test. Adversarial instances are generated via backpropagation through an artificial neural network, and effective test case generation is achieved by identifying input test values ​​corresponding to untested regions in the output domain space. The selection of appropriate output vectors based on the untested output space can be performed automatically (e.g., ...). Figure 1 (As described in reference 121). The described solution enhances the way the output space can be explored to find unexplored output vectors y'.

[0033] Figure 3This is a flowchart depicting an implementation of a method for selecting an output vector and subsequently identifying the corresponding input vector based on the output vector. At the heart of this process is an output generator 305. The output generator 305 takes the current distribution of the output space 301 as input. The current distribution 301 can be easily determined based on existing tests. This data provides the current coverage of the output space. To guide the exploration of the output space, the output generator 305 also receives a target distribution 303 as input. The target distribution 303 represents the test target in terms of the coverage of the output space provided to the output generator 305. These inputs are used to determine how to explore the output space. The result is a new, unexplored output vector y'307. The output vector y'307 is then provided to an artificial neural network (SUT'). (ANN) The system under test is represented by y'309. The output vector y'307 is backpropagated through neural network 309 and traced to a set of corresponding inputs 311. When it is forward processed through neural network 309, the set of corresponding inputs will produce the output vector y'307.

[0034] The output generator 305 can operate in several ways. Two possible techniques will now be described.

[0035] Method 1: Enhanced Adaptive Random Output Generation

[0036] Modifying the existing output y by a certain increment (delta) Δ produces y' = y + Δ, resulting in the definition of the new output y'307. The search for Δ (which transforms the covered output vector y into the uncovered output vector y'307) is automatically performed by the implemented output test generator 305. The test generator 305 is based on specific constraints on the input and on the output space. This technique assumes that failed test cases are typically grouped into adjacent regions of the input space, but their output spaces reveal disproportionate variations. This is based on the theory of discontinuous functions, where slight changes in the input produce jumps in the output. Therefore, random output generation is constrained by the uncovered output space and the adjacent regions of the input vectors.

[0037] Method 2: Rule-based output generation

[0038] In rule-based output generation, a new value y'307 is constrained by modifying existing outputs from the training set with a context-dependent large value -δ. This method involves rules that can be derived from domain knowledge or conventional testing techniques. The target output space distribution is determined by domain experts and / or test engineers using classic testing techniques including class segmentation methods, boundary value analysis, and domain-specific requirements.

[0039] Based on a visual comparison of the current output space coverage distribution map obtained from existing test data and the target output space distribution, gaps can be identified, and new y' values ​​307 can be selected from these gap regions. The visual comparison can be performed using a graphical depiction showing the categories of test cases and the number of test cases falling into each category.

[0040] Figure 6A and Figure 6B Examples of graphical depictions for providing visual comparisons of output spatial coverage according to some embodiments of this disclosure are provided. Figure 6A and Figure 6B The system under test (SUT) is shown, relating to a control system for providing braking to a vehicle (e.g., a train). Braking distances are marked on a horizontal axis 620, while multiple test cases are represented on a vertical axis 610. The measured quality (braking distance 620) is divided into multiple categories of possible outputs. That is, the braking distance is broken down into increments of 100 feet, from zero to 100 feet up to 800 feet. For each category, the number of test cases is represented by a series of bars arranged as a histogram 611, with a target number of test cases established for each category. In Figure 6, the target number of test cases for each output category is 250 test cases 601. Figure 6A Starting from the described starting point, there are 2000 pre-existing test cases distributed across each indicated output category. For example, there are 888 test cases for braking distances from zero to 100 feet. There are 392 test cases for braking distances from 101 to 200 feet, 228 for 201 to 300 feet, 189 for 301 to 400 feet, 136 for 401 to 500 feet, 100 for 501 to 600 feet, 54 for 601 to 700 feet, and 16 for braking distances exceeding 701 feet. (The last sentence appears to be incomplete and possibly refers to a different test case.) Figure 6A As seen in the diagram, braking distances greater than 200 feet are represented by fewer than the target number of test cases 601. This defines the area of ​​the output space 621 under test. Using the techniques and processes of embodiments described in this disclosure, the output space 621 under test can be purposefully targeted, and test cases 625 can be generated to satisfy the lack of test cases relative to the target level 601. These test cases covering the output space 625 can be generated using the identified output space 621, and associated input values ​​can be generated using a trained neural network representing the system under test based on the untested output space.

[0041] These output selection techniques represent a technological improvement in providing systematic exploration of the output space by considering the current and target distributions of the output data. They further provide guided output exploration by reducing the number of test input combinations and searching only uncovered outputs in the output space. This results in other improvements, including deriving test inputs that will effectively provide better output space coverage. By improving output space coverage, the likelihood of discovering new defects increases. Furthermore, testing is improved by enabling the identification of values ​​where the relationship between changes in the input vector and the resulting changes in the output space is not preserved (e.g., small changes in the input vector lead to disproportionate changes in the output space). These techniques also allow exploration of the input / output relationship to be performed before test cases have been generated. These technological improvements not only reduce the number of test cases but, more importantly, improve testing by identifying missing test cases.

[0042] During product development, testing can be a resource- and time-consuming phase. This approach aims to conserve time and resources by reducing the number of test cases while maintaining fault detection capabilities. The described method reduces the workload required for testing by providing less efficient test cases to more rigorously cover the output space. In this way, the described method prevents the generation of duplicate test cases and avoids the generation of test inputs that produce the same results. In summary, it optimizes the costs associated with testing by reducing the number of test cases evaluated.

[0043] Figure 4This is a flowchart of a method for automatically generating test cases for a system under test (SUT) according to some embodiments of the present disclosure. A reference model is created as an artificial neural network (ANN) 401 trained using data from the SUT. Training can be performed by providing the ANN with a dataset including input data values ​​and associated output values, which produce these input data values ​​and associated output values ​​when the inputs are applied to the SUT. Based on the design of the SUT, the output space is analyzed to determine the portions of the output space that remain untested or under test. A desired output value representing the untested output space is selected 403. The desired output value is then backpropagated through the ANN to map the desired output value to the input values ​​that produce the desired output according to the ANN 405. Input values ​​associated with the desired output are identified 407, and the identified input values ​​are provided to the SUT to generate test output values ​​409. The test output values ​​and their associated inputs are then compared with an input / output dataset from the reference model 408 to determine whether the output values ​​according to the ANN model match the test output values ​​generated by the SUT 411 based on the same input values. If the output values ​​match, the new input is added to an existing test suite 413. Otherwise, if the output values ​​do not match, determine whether the output is a valid output of the system under test 417. If the generated output value is valid 415, use the data to retrain the ANN, because it can be inferred that the input / output dataset represents valid data that the neural network was not exposed to during training. Otherwise, determine the output as an invalid system output representing a defect or error in the system being tested 419. Experts can be used to track down errors and fix the system under test 419.

[0044] Figure 5 An exemplary computing environment 500 in which embodiments of the present invention may be implemented is shown. Computers and computing environments (such as computer system 510 and computing environment 500) are known to those skilled in the art and are therefore briefly described herein.

[0045] like Figure 5 As shown, the computer system 510 may include a communication mechanism such as a system bus 521 or other communication mechanisms for transmitting information within the computer system 510. The computer system 510 also includes one or more processors 520 connected to the system bus 521 for processing information.

[0046] Processor 520 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processor known in the art. More generally, a processor, as used herein, is a device for performing tasks by executing machine-readable instructions stored on a computer-readable medium, and may include any one or a combination of hardware and firmware. The processor may also include memory storing machine-readable instructions executable for performing tasks. The processor acts on information by manipulating, analyzing, modifying, transforming, or transmitting information for use by an executable program or information apparatus, and / or by routing information to an output device. For example, the processor may use or include the capabilities of a computer, controller, or microprocessor, and may use executable instructions to regulate the processor to perform special functions not performed by a general-purpose computer. The processor may be coupled (electrically coupled and / or include executable components) to any other processor to enable interaction and / or communication between them. A user interface processor or generator is a known element that includes electronic circuitry or software, or a combination of both, for generating a display image or a portion thereof. A user interface includes one or more display images that enable a user to interact with the processor or other devices.

[0047] See also Figure 5 The computer system 510 also includes system memory 530 connected to the system bus 521 for storing information and instructions to be executed by the processor 520. System memory 530 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 531 and / or random access memory (RAM) 532. RAM 532 may include other dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). ROM 531 may include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM).

[0048] Furthermore, system memory 530 can be used to store temporary variables or other intermediate information during instruction execution by processor 520. A basic input / output system 533 (BIOS) containing basic routines that facilitate the transfer of information between components within computer system 510 (such as during startup) can be stored in ROM 531. RAM 532 can contain data and / or program modules that are immediately accessible and / or currently operating by processor 520. Additionally, system memory 530 may include, for example, an operating system 534, application programs 535, other program modules 536, and program data 537.

[0049] Computer system 510 also includes a disk controller 540, such as a magnetic hard disk drive 541 and a removable media drive 542 (e.g., a floppy disk drive, optical disk drive, magnetic tape drive, and / or solid-state drive), which is coupled to system bus 521 to control one or more storage devices for storing information and instructions. Storage devices can be added to computer system 510 using appropriate device interfaces such as Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or FireWire.

[0050] Computer system 510 may also include a display controller 565 coupled to system bus 521 to control a display or monitor 566, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface 560 and one or more input devices (such as a keyboard 562 and a clicking device 561) for interacting with the computer user and providing information to processor 520. For example, clicking device 561 may be a mouse, light pen, trackball, or click stick for transmitting directional information and command selections to processor 520 and for controlling cursor movement on display 566. Display 566 may provide a touchscreen interface that allows input to supplement or replace the communication of directional information and command selections by clicking device 561. In some embodiments, a user-wearable augmented reality device 567 may provide input / output functionality that allows the user to interact with both the physical and virtual worlds. Augmented reality device 567 communicates with display controller 565 and user input interface 560, thereby allowing the user to interact with virtual items generated by display controller 565 in augmented reality device 567. Users can also provide gestures that are detected by the augmented reality device 567 and transmitted as input signals to the user input interface 560.

[0051] Computer system 510 may perform some or all of the processing steps of embodiments of the present invention in response to processor 520 executing one or more sequences of one or more instructions contained in memory (such as system memory 530). Such instructions may be read into system memory 530 from another computer-readable medium such as magnetic hard disk 541 or removable media drive 542. Magnetic hard disk 541 may contain one or more data storage and data files used by embodiments of the present invention. The data storage contents and data files may be encrypted to improve security. Processor 520 may also be employed in a multiprocessing arrangement to execute sequences of one or more instructions contained in system memory 530. In alternative embodiments, hardwired lines may be used in place of or in combination with software instructions. Therefore, the embodiments are not limited to any particular combination of hardware circuitry and software.

[0052] As described above, computer system 510 may include at least one computer-readable medium or memory for storing instructions programmed according to embodiments of the present invention and for containing data structures, tables, records, or other data described herein. As used herein, the term "computer-readable medium" refers to any medium that participates in providing instructions to processor 520 for execution. Computer-readable media can take many forms, including but not limited to non-transitory, non-volatile, volatile, and transmission media. Non-limiting examples of non-volatile media include optical discs, solid-state drives, magnetic disks, and magneto-optical discs, such as magnetic hard disk 541 or removable media drive 542. Non-limiting examples of volatile media include dynamic memory, such as system memory 530. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including lines forming system bus 521. Transmission media may also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0053] The computing environment 500 may also include a computer system 510 operating in a network environment using logical connections to one or more remote computers (such as remote computing devices 580). The remote computing device 580 may be a personal computer (laptop or desktop), mobile device, server, router, network PC, peer-to-peer device, or other common network node, and typically includes many or all of the elements described above with respect to the computer system 510. When used in a network environment, the computer system 510 may include a modem 572 for establishing communication over a network 571 such as the Internet. The modem 572 may be connected to the system bus 521 via a user network interface 570 or via another suitable mechanism.

[0054] Network 571 can be any network or system commonly known in the art, including the Internet, intranet, local area network (LAN), wide area network (WAN), metropolitan area network (MAN), direct connection or a series of connections, cellular telephone network, or any other network or medium capable of facilitating communication between computer system 510 and other computers (e.g., remote computing device 580). Network 571 can be wired, wireless, or a combination thereof. Wired connections can be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection commonly known in the art. Wireless connections can be implemented using Wi-Fi, WiMAX, Bluetooth, infrared, cellular networks, satellite, or any other wireless connection method commonly known in the art. Furthermore, several networks can operate independently or communicate with each other to facilitate communication within network 571.

[0055] For example, an executable application, as used herein, includes code or machine-readable instructions for regulating the processor to perform predetermined functions (such as those of an operating system, a context data acquisition system, or other information processing system) in response to user commands or input. An executable step is a segment of code or machine-readable instructions, subroutine, or other distinct code segment or part of an executable application for performing one or more specific processes. These processes may include receiving input data and / or parameters, performing operations on the received input data and / or performing functions in response to the received input parameters, and providing the resulting output data and / or parameters.

[0056] As used herein, a graphical user interface (GUI) includes one or more display images generated by a display processor and capable of enabling user interaction with the processor or other devices, along with associated data acquisition and processing functions. The GUI also includes an executable program or application. The executable program or application modulates the display processor to generate signals representing the GUI display images. These signals are supplied to a display device that displays images for user viewing. Under the control of the executable program or application, the processor manipulates the GUI display images in response to signals received from input devices. In this way, the user can interact with the display images using input devices, thereby enabling user interaction with the processor or other devices.

[0057] The functions and processing steps described herein may be executed automatically, fully, or partially in response to user commands. Automatically executed activities (including steps) are performed in response to one or more executable instructions or device operations, without direct user initiation.

[0058] The systems and processes shown in the accompanying drawings are not unique. Other systems, processes, and menus can be derived from the principles of the invention to achieve the same purpose. Although the invention has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the present design can be made by those skilled in the art without departing from the scope of the invention. As described herein, different systems, subsystems, agents, managers, and processes can be implemented using hardware components, software components, and / or combinations thereof. Unless an element is explicitly stated using the phrase "means for...", no element of any claim herein should be construed under Section 6 of 35 USC 112.

Claims

1. A method for testing a software application in a system under test (SUT), comprising: constructing a reference model of the SUT, the reference model comprising a computer-based neural network; training the reference model using input data and corresponding output data generated by the SUT; selecting an output value within a domain of possible output values of the SUT, the selected output value representing an output not represented in the output data used to train the reference model; applying the selected output value to the reference model and tracing the selected output value through the reference model to identify a test input value that produces the selected output value when input to the reference model; testing the system under test using the identified test input value to generate a test output value from the SUT; providing the identified input value to the reference model to generate the selected test output value; comparing the selected output value from the reference model to the test output value from the SUT; and if the comparison indicates that a difference between the selected output value from the reference model and the test output value from the SUT exceeds a predetermined threshold, analyzing the test output value to determine whether the test output value is a legitimate system output or whether the difference indicates an error in the SUT. if it is determined that the test output value is a legitimate value, retraining the neural network using a data set containing the determined input and the test output value.

2. The method of claim 1, wherein, if it is determined that the test output value is not a legitimate system output, analyzing the system under test to identify an error responsible for creating the illegitimate system output and correcting the identified error.

3. The method of claim 1, wherein, identifying the output value from the output space of the software application comprises:

4. The method of claim 1, wherein, receiving a current distribution and a target distribution in an output generator; and outputting a new output vector y' that is unexplored.

5. The method of claim 4, further comprising the output generator: applying a restriction to contiguous regions of the input space; and applying a restriction to the output space to select only the new output vector from unexplored output space.

6. The method of claim 4, further comprising the output generator: selecting the new output vector y' by modifying an existing output from a training set with contextually relevant values and based on a visual comparison of a current output space coverage and a plot of a target output space distribution.

7. A system for testing a software application in a system under test (SUT), comprising: a computer processor in communication with a SUT; a memory in communication with the computer processor, the memory containing executable instructions that, when executed by the processor, cause the processor to: construct a reference model of the SUT; identify an output value from an output space of the software application, the output value being an output value that has not been observed through prior testing; submit the identified output value to the reference model; trace the output value through the reference model to determine at least one input value that produces the identified output value when input to the reference model; ​ providing the determined input values to the reference model to generate an identified output value; providing the determined input values to the SUT to generate a test output value from the SUT; comparing the identified output value to the test output value; and if the comparison indicates that a difference between the identified output value and the test output value exceeds a predetermined threshold, analyzing the test output value to determine whether the test output value is a legitimate system output or whether the difference indicates an error in the system under test, wherein the reference model is implemented as an artificial neural network, ANN.

8. The system of claim 7, wherein, if it is determined that the test output value is a legitimate value, retraining the ANN using a dataset containing the determined input and the test output value.

9. The system of claim 7, wherein, if it is determined that the test output value is not a legitimate system output, analyzing the system under test to identify an error responsible for creating the illegitimate system output, and correcting the identified error.

10. The system of claim 7, wherein, identifying the output value from the output space of the software application comprises an output generator configured to receive a current distribution and a target distribution, and output a new output vector y’ that is unexplored.

11. The system of claim 10, wherein, the output generator is configured to: apply a restriction to contiguous regions of the input space; and apply a restriction to the output space to select only the new output vector from unexplored output space.

12. The system of claim 10, wherein, the output generator is configured to: select the new output vector y’ by modifying an existing output from a training set with context-dependent values, and based on a visual comparison of a current output space coverage and a plot of a target output space distribution.

Images