Methods for automated testing (robustness / functionality testing) of applications with a graphical user interface (GUI) using instrumentation of the underlying GUI frameworks / runtime environments

Abstract

1. Methods for automated testing (robustness / functionality testing) of applications with a graphical user interface (GUI) by instrumenting the underlying GUI frameworks / runtime environments. 2. Summary 2.1 Applications with a graphical user interface have so far been largely tested by recording interactions (mouse / keyboard input) and then replaying them. These interactions can also be specified textually / graphically. In newer approaches, the graphical user interface is technically removed, and the underlying functions are tested for robustness by directly calling them. This new method enables both automated functional testing and robustness testing. 2.2 By instrumenting the underlying GUI frameworks / runtime environments, the application's controls can be read and manipulated. This allows their type and properties to be determined, enabling automated robustness testing. Furthermore, automated functional testing can be performed by specifying additional conditions. 2.3 The method enables the automated testing of applications with a graphical user interface.

Description

State of the art

[0001] There is currently a wide variety of software for testing applications with graphical user interfaces (GUIs). These offer the option of recording and replaying interactions with the program under test, and they also allow the creation of textual / graphical specifications for how the test interactions should appear. A disadvantage of this approach is that these tests must be created entirely manually, which can be very time-consuming. Furthermore, this manual process usually does not result in optimal code coverage. More recent research approaches technically truncate the GUI and call the underlying functions directly to test their robustness (fuzzing). This also collects information about code coverage (which branches have already been taken in the program and which have not). This is made possible by instrumentation code.This can be implemented either directly by modern compilers or through other methods used in closed-source software. This makes it possible to obtain feedback on the test and, at the same time, achieve greater code coverage through targeted modification of the input data. A disadvantage of directly calling functions located behind the GUI is that, on the one hand, values ​​that would otherwise be filtered by the controls are passed to the functions, which can lead to false positives (erroneously detected errors). On the other hand, the application state can become inconsistent if only individual functions of the application under test are called, bypassing the actual program logic. Structure of the invention

[0002] The invention instruments the underlying GUI frameworks or runtime environments to report the creation of new controls. The type of each created control and its properties (e.g., minimum / maximum of the selectable integer, etc.) can then be determined via the programming interface provided by the GUI framework / runtime environment. This makes it possible to automatically generate tests in which the created controls are manipulated either repeatedly in sequence or in a different order according to their type and properties (e.g., setting a value while maintaining a minimum / maximum, if applicable), or in which actions (mouse clicks, etc.) are triggered. This results in a so-called robustness test, since input data unforeseen by the programmer may lead to a crash.Additionally, conditions can be specified before starting the automated test (e.g., the value of control A equals the sum of control B and control C), resulting in a functional test. Furthermore, if the application under test is instrumented to report when new program paths are reached, the automated robustness / functional test can be optimized by manipulating the controls in a beneficial way, resulting in greater code coverage and thus a better test. Example

[0003] The invention was implemented using the Qt framework. For closed-source applications, a library necessary for further processing was integrated into the application under test using .dll hooking. If the application has source code available, a provided "dummy function" of the library can simply be called anywhere in the code of the application under test and linked to the integrated library. The integrated library receives control and information about the application under test using various types of hooking (function / V-Table hooking) and can directly manipulate and read controls. Once a control is created, it is registered within the constructor using hooking. The thus-memorized control can later be read and manipulated for testing purposes, taking its type and other properties into account.The injected library gains control by starting a timer in the initialization part of the library, which is triggered in the event loop of the program under test. The timer event handler is called within the event loop so that actions can be performed on the controls. If other events need to be processed by the event loop, the timer event handler returns after reactivating the timer, allowing further actions to be performed on the controls later.

[0004] Finally, information about the current code coverage is collected using instrumentation code generated by the translator and, based on this, new inputs for the controls are generated so that code coverage is maximized.

[0005] This example can be generalized so that, for example, Java or other types of applications can also be supported. Advantages of the invention

[0006] The invention enables automated, self-optimizing tests to be created. This is a clear advantage over the current state of the art, as a test can be created and initiated more or less with the press of a button. Furthermore, by specifying additional conditions, a functional test can be performed with little effort, something that would require significantly more effort to create using the current state of the art. Regarding the approach that "truncates" the GUI, one advantage is that only values ​​that "make sense" are used, because properties of the control elements are taken into account (minimum / maximum / ...). Furthermore, the program logic is not undermined, and inconsistent states cannot arise.

Claims

[1] the automated registration of controls (e.g. text fields, buttons, ...) a) either when they are created b) or after their creation [2] the automated manipulation / operation of controls a) either repeated in the order in which they were created b) or in a different order [3] the automated consideration of the properties (including the type) of the registered controls when manipulating / operating them [4] the automated consideration of any existing specification of conditions that must be fulfilled (functional test) [5] the automated inclusion of code coverage during testing during further manipulation / operation of the controls [6] the use of timers of the corresponding GUI framework / runtime environment, which allow the execution of test code within the timer event handler within the event loop of the program under test