Methods for testing a computer program

Abstract

The invention relates to a method for testing a computer program with a graphical user interface, wherein a test environment is provided for testing the system behavior of the computer program under test by operating at least one control element of the graphical user interface, wherein, in addition to the first image data stream for the display device, a second image data stream is provided to the test environment via the display interface of the data processing device, wherein, by means of optical character recognition, control elements operable by an input device in the graphical user interface provided via the second image data stream are recognized as an object of the graphical user interface.wherein at least the operable controls recognized in the graphical user interface are entered as input data into a large language model connected to the test environment via a data interface, with the stipulation that an action option for operating at least one control element of the graphical user interface is obtained at the data interface, wherein an input signal is generated by the test environment depending on the action option provided at the data interface for operating at least one recognized control element, and wherein the generated input signal is provided at the input interface of the data processing system in order to trigger a simulated user input for testing the computer program.

Description

[0001] The invention relates to a method for testing a computer program which generates an output in the form of a graphical user interface and displays it on a display device.

[0002] A crucial step in developing a computer program is testing its specified functionality. This involves not only verifying that the program performs correct calculations and thus delivers the necessary and, above all, accurate output data, for example, in safety-critical environments, but also the interaction between the user and the machine. This human-machine interface is now provided by graphical user interfaces, which allow humans to interact with the machine.

[0003] In such a human-machine interface, in the form of a graphical user interface, not only is information displayed as static or dynamic data, but it also provides the possibility to interact with the graphical user interface using suitable input devices, such as a mouse, a keyboard, and / or a sensitive touchscreen, in order to input data into the computer program. Such input can be made, for example, by selecting predefined data or by freely entering character strings.

[0004] Only if the desired processes are functionally represented in the graphical user interface and at the same time human use is effective and efficient, does the user interface fulfill its purpose and minimize incorrect input and operating errors.

[0005] Particularly in the development of safety-critical interfaces, such as those frequently found in aviation and aerospace, a problem arises in software testing: while experts in these industries possess in-depth knowledge of user needs, environmental conditions, and system functionalities, they lack expertise in software development and testing. Software developers, on the other hand, lack a thorough understanding of the specific challenges within their respective industries and have insufficient knowledge of the environmental parameters relevant to the safety-critical interface. Therefore, it is desirable that both professional groups can fully utilize their respective knowledge and expertise without having to delve deeply into the other's field.

[0006] For example, practical experience shows that most errors and poor usability issues in the development of novel graphical user interfaces for commercial aircraft pilots are immediately identified by the first two or three pilots. This increases the number of feedback loops and reduces pilot acceptance of new software products. Furthermore, it increases the costs of developing and implementing new software components.

[0007] It is common practice to automate software testing during development. For this purpose, program elements necessary solely for running test scenarios are integrated into the program code of the software under development. These elements are of no use to the software itself. Test scenarios can then be programmed that access these integrated program elements and execute pre-programmed tests.

[0008] The disadvantage here is that creating test scenarios requires hard-coding corresponding program elements into the program code, which must be done during the development of the computer program. This limits the development of test scenarios to these specific program elements. Furthermore, only the program developers can create a test scenario, making it difficult to integrate the expertise of other professional groups, such as pilots or clients, into the test scenario creation process.

[0009] Another disadvantage of this approach is that this useless (dead) test code, which remains in the final computer program, is prohibited by many security standards or necessitates additional security measures. If this test code is compiled only for testing purposes and not for production use, then the software being tested is not exactly the one that will later be released. This creates an irresolvable conflict: either you have dead code at runtime, or you are not performing tests on the exact software that will ultimately go into production.

[0010] It is therefore an object of the present invention to provide an improved method for testing computer programs, which eliminates the need to insert program elements into the program code solely for the purpose of creating test scenarios.

[0011] The problem is solved according to the invention using the method according to claim 1. Advantageous embodiments of the invention are described in the corresponding dependent claims.

[0012] According to claim 1, a method for testing a computer program with a graphical user interface is proposed, wherein the method generically comprises the following steps: - Providing a data processing system comprising an electronic computing unit, a display interface through which a display device is connected or connectable, and an input interface through which an input device for user input is connected or connectable; - Execution of the computer program to be tested on the data processing system, whereby the computer program continuously provides an image data stream with a graphical user interface containing a plurality of objects at the display interface of the data processing system.

[0013] According to the invention, a test environment is provided for testing the system behavior of the computer program under test, including the operation of at least one control element of the graphical user interface. - wherein, in addition to the first image data stream, a second image data stream is provided to the test environment via the display interface of the data processing device, - wherein, by means of optical character recognition, control elements operable by an input device in the graphical user interface provided via the second image data stream are recognized as an object of the graphical user interface, - wherein at least the operable control elements recognized in the graphical user interface are entered as input data into a large language model connected to the test environment via a data interface, with the requirement of obtaining an action option for operating at least one control element of the graphical user interface at the data interface, - wherein an input signal is generated by the test environment depending on the action option provided at the data interface for operating at least one detected control element, and - wherein the generated input signal is provided at the input interface of the data processing system to trigger a simulated user input for testing the computer program.

[0014] Accordingly, a data processing system is first provided on which the computer program to be tested is to be executed. When the computer program is executed on the data processing system, a SuT, i.e., a "System under Test," is created. By definition, the data processing system has an electronic processing unit (EPU), which includes a central processing unit (CPU) for executing the computer program. Furthermore, peripheral devices can be connected to the data processing system via appropriate interfaces to provide input for the EPU and to display output on a screen. For this purpose, the data processing system has a screen interface to which corresponding image data can be supplied for display on a screen connected to the screen interface.Furthermore, the data processing system has an input interface to receive input from an input device (e.g., mouse, keyboard, touchscreen, etc.) and forward it to the electronic processing unit. The input is then processed by the computer program, whereupon the output is displayed on the display device (e.g., screen) and adjusted as necessary.

[0015] To test the computer program, it is executed on the data processing system, which continuously provides an image data stream to the display interface of the data processing system. This image data stream can then be displayed as a graphical user interface on the display device. This image data stream is provided, for example, at a predetermined refresh rate, preferably the frequency of the display device. However, it is not strictly necessary for a display device to actually be connected to the display interface to execute the method. Likewise, it is not strictly necessary for an input device to actually be connected to the input interface to execute the method.

[0016] Preferably, the computer program to be tested does not contain any program elements that form an interface for testing the computer program using another test program. Instead, according to the invention, the computer program is tested using the following procedure.

[0017] First, a test environment is provided to test the computer program under test. This test environment can also be a computer program itself, for example, one that runs on the data processing system. The test environment is designed to replicate the system behavior of the computer program under test, including the operation of at least one control element of the graphical user interface.

[0018] A second image data stream is provided to the test environment, which can be accessed from the display interface of the data processing unit. This second image data stream is parallel to the first image data stream, which is sent to the display device to render the graphical user interface. The test environment thus receives a copy of the current display on the display device, preferably at the same frequency.

[0019] Next, optical character recognition (OCR) is used to analyze the graphical user interface (GUI) provided to the test environment via the second image data stream. In addition to recognizing static information (data fields, labels, etc.), the focus here is on identifying interactive controls (buttons, drop-down menus, text fields, checkboxes, etc.), especially those controls relevant to the system behavior under test and thus part of the system behavior definition. The analysis not only identifies the controls themselves but also their position within the GUI, which can be represented using a two-dimensional coordinate system.

[0020] Optical character recognition can be configured in such a way that it receives a context beforehand, specifying which objects and, in particular, which character strings identifying the objects are to be identified and, if applicable, which properties these objects possess.

[0021] Accordingly, optical character recognition is used to identify control elements operable by the input device in the graphical user interface provided via the second image data stream, depending on the control elements contained in the system behavior to be tested.

[0022] At least these recognized controls are now inputted into the large language model with the requirement of receiving an action option for operating at least one of these controls. For this purpose, the test environment has a data interface through which it is connected to the large language model. The large language model can be part of the test environment, or it can simply be connected to such an external large language model via the data interface.

[0023] The requirement to obtain a corresponding course of action is derived from the specified system behavior of the computer program being tested.

[0024] At the data interface, after inputting the recognized controls as input data, at least one action option for operating at least one of the controls is provided, with this action option resulting as output from the large language model. The large language model thus delivers a corresponding action option as output from the input of the recognized controls, with such an action option encompassing the operation of one of the controls.

[0025] Based on the action option provided at the data interface for operating at least one of the detected controls, the test environment now generates an input signal, which is provided at the input interface of the data processing system. By providing such an artificially generated input signal at the input interface of the data processing system, a simulated user input is triggered to test the computer program, thereby eliciting the corresponding system behavior.

[0026] The generated input signal corresponds to a signal from an input device that can trigger input into the data processing system. This input signal can be provided to the input interface of the data processing system to generate input. However, the generated input signal does not originate from an actual input device (such as a mouse or keyboard), but was generated virtually and is now provided to the input interface of the data processing system to trigger simulated user input for testing the computer program.

[0027] The generated input signal thus simulates the operation of the control element on the graphical user interface without requiring any actual input from a physical user. Instead, the control element is operated purely from within the test environment by providing the generated input signal at the input interface.

[0028] By simulating the operation of the control element on the graphical user interface, a response from the computer program is triggered, leading to a change in the graphical user interface. In a subsequent step, the changed graphical user interface can then be analyzed again by analyzing the second image data stream and compared with the specified system behavior in the test scenario to determine whether the computer program behaves according to the specified test scenario or not.

[0029] The present invention thus makes it possible to test a computer program (SuT) without having to integrate corresponding program elements into the computer program. Rather, the computer program can be developed without regard to this, with the specification of the test scenario then being carried out by persons with the relevant expertise, without requiring any prior knowledge of testing such computer programs.

[0030] This reduces the development costs of computer programs and allows graphical user interfaces to be tested at the earliest possible stage, without the need for subsequent beta testing in the field by the intended end users. This reduces the number of feedback loops, thereby decreasing the effort required for changes and increasing software acceptance.

[0031] Large language models are language models that can analyze natural language texts, including some aspects of their semantics. These models are trained using large datasets, particularly natural language texts. The recognized character strings are provided to the large language model, which then outputs an action option for operating one of the recognized controls.

[0032] Such a large language model can be based on an artificial neural network that has been trained using substantial amounts of data. This large language model could, for example, have been trained in the same way as the well-known LLMs, such as those sold under the trade names "ChatGPT" or "OpenAI." The large language model is a computational linguistic probability model that learns statistical word and sentence sequence relationships from a large number of text documents through a training process.

[0033] Individual words, word groups, or an entire test scenario can be entered as input data into the large language model, as well as one or more recognized strings from the graphical user interface, whereby the large language model then provides an action option.

[0034] According to one embodiment, it is provided that, by means of optical character recognition, an operating area of ​​the recognized operable control element, describable by coordinates, is also determined in the provided graphical user interface, within which the operable control element can be triggered by a user input at the input device, wherein the input signal is further generated depending on the operating area determined.

[0035] This identifies not only the control element itself, but also the operable area within which the control element can be operated on the graphical user interface and thus trigger an action. The operable area is preferably a two-dimensional area defined by its surface and position within the graphical user interface.

[0036] According to one embodiment, the test environment is provided for on the data processing system or on an external device connected to the data processing system.

[0037] The test environment is a separate, second computer program from the one under test. It runs on a data processing system and tests the program under test. The test environment can therefore run on the same data processing system as the program under test. However, it is also possible for the test environment to run on a separate, second data processing system and be connected to the first data processing system running the program under test. This connection is necessary to access the second image data stream at the display interface and to provide the generated input signal to the first data processing system via the input interface, thus triggering simulated user input.

[0038] According to one embodiment, it is provided that information displayed in the graphical user interface provided via the second image data stream is recognized as objects by means of optical character recognition, wherein the information recognized in the graphical user interface is also entered into the large language model via the data interface with the proviso of obtaining an action option for operating at least one control element of the graphical user interface at the data interface.

[0039] This allows, in particular, static information displayed on the graphical user interface, which does not change during program execution or only changes due to user interaction, to be validated against the information contained in the system behavior. This ensures that the graphical user interface displays the specified information that is also defined in the system behavior.

[0040] In addition to the recognized controls of the graphical user interface, this displayed information is passed as input data to the large language model, so that the large language model has a further context of the graphical user interface and can thus provide more targeted action options.

[0041] According to one embodiment, the test environment has at least one test scenario which includes system behavior of the computer program under test with operation of at least one control element of the graphical user interface, wherein, depending on the test scenario, at least one instruction is generated for the large language model and this is also entered into the large language model via the data interface in order to obtain an action option for the operation of at least one control element of the graphical user interface at the data interface.

[0042] In such a test scenario, the system behavior can be described by corresponding instructions for the computer program, including not only the operation of at least one control element but also the expected display. This definition in the test scenario can preferably be done using natural language, allowing corresponding instructions to be generated and then entered as further input data into the larger language model. In the simplest case, the definitions in the test scenario can be transferred one-to-one to the larger language model.

[0043] According to one embodiment thereof, it is provided that after the simulated user input is triggered, a modified second image data stream with a modified graphical user interface is provided, wherein objects in the modified graphical user interface are recognized by means of optical character recognition and wherein the modified graphical user interface is verified with regard to the system behavior described in the test scenario depending on the recognized objects.

[0044] It may be provided that the steps up to the triggering of the simulated user input are repeated multiple times, with each repetition taking into account the previously modified graphical user interface from a previous simulated user input.

[0045] In this case, the graphical user interface changes after user input is triggered by a simulated operation of the control element specified in the test scenario, which is a reaction to the operation of the control element. This change in the graphical user interface after the simulated operation of the control element is then validated by analysis using optical character recognition and the system behavior specified in the test scenario.

[0046] This enables a complete test including prerequisites, operation and response, without the computer program requiring corresponding integrated program elements.

[0047] According to one embodiment, optical character recognition is used to first recognize character strings of the objects contained in the graphical user interface, and then to assign the objects describing the system behavior in the test scenario to the corresponding objects within the graphical user interface depending on the recognized character strings of the objects contained in the graphical user interface.

[0048] Preferably, a mapping is performed in which an attempt is made to assign each object recognized in the graphical user interface to an object in the test scenario. Both the objects in the graphical user interface and the objects in the test scenarios are defined by corresponding strings, so the mapping is performed via these strings. This means that a string recognized in the graphical user interface, which identifies a corresponding object within the graphical user interface, is assigned to a string in the test scenario that defines a corresponding object within the test scenario.

[0049] According to one embodiment, the test scenario is provided in such a way that the system behavior described thereby is defined by means of natural language, whereby it may be provided that the system behavior defined in natural language is translated into an artificial description of the system behavior by means of a parser, wherein the artificial description is then entered into the large language model.

[0050] Describing system behavior using natural language makes it possible for experts in a given field to create test scenarios, for example, without requiring prior knowledge of computer program development. The parser is designed to translate the system behavior defined in natural language into an artificial description of that behavior, which is then executed by the test environment.

[0051] Such a test scenario can be specified by domain experts and the client, and can consist of individual steps formulated in prose. Appropriate prefix words can be used for each step to simplify parsing the specified test scenario and ensure a reliable process.

[0052] Such test scenarios are human-readable and therefore comprehensible to both the client and the domain expert, even without programming knowledge. Placeholders, stored in local variables, can also be used. The parser then translates this into a machine-readable description of the system behavior, allowing the test scenario to be executed step-by-step by the test environment.

[0053] The invention is explained in more detail using the attached figures as examples. They show: Fig. 1. Schematic representation of a system to be tested with test environment; Fig. 2 Schematic representation of the test procedure according to the invention.

[0054] Fig. Figure 1 shows a data processing system 10, which has an electronic computing unit 11 and a data storage device 12 (volatile and non-volatile). A computer program 20 is executed on the computing unit 11, which interacts with the elements of the data processing system 10.

[0055] The data processing system 10 has a display interface 13, via which a display device 14 is connected, so that the computer program 20 can output a corresponding graphical output via the display interface 13 on the display device 14. Such a graphical output is a graphical user interface.

[0056] The data processing system 10 also has an input interface 15 to which an input device 16 is connected. Using the input device 16, inputs for the computer program 20 can be generated and provided accordingly, thus enabling interaction with the computer program 20. Based on the interaction and the inputs generated via the input device 16, the output on the display device 14 can then be adjusted accordingly, which is also initiated by the computer program 20. Such input devices 16 can be, for example, a mouse, a keyboard, control buttons, and / or touch displays.

[0057] Furthermore, according to the invention, the data processing system 10 provides a test environment 30 which is located in Fig. 1 is represented as a component of the data processing system 10. This could, for example, also be a computer program running on the computing unit 11. However, it is also conceivable that it is an external device connected to the data processing system 10 via an interface (not shown).

[0058] The test environment 30 is connected to both the display interface 13 and the input interface 15. The test environment receives a copy of the current image data stream from the display interface 13, which is then displayed on the display device 14. The test environment 30 receives the corresponding inputs from the input interface 15, which were entered at the input device 16 for interaction with the computer program 20.

[0059] The connection between the test environment 30 and the input interface 15 is bidirectional, so that the test environment 30 not only receives a copy of the inputs from the input device 16, which are forwarded to the computer program 20, but can also itself provide input signals at the input interface 15 in order to trigger an interaction with the computer program 20.

[0060] Furthermore, the test environment 30 is connected via a data interface 18 to a large language model 45 in order to interact with it and to obtain an action option for operating the computer program under test by entering instructions, information and recognized operated elements.

[0061] Fig. Figure 2 schematically illustrates the procedure executed by the test environment using the example of an aerospace computer program. This computer program under test generates a display on a display device 14 (see Figure 2). Fig. 1) A graphical user interface 40, which is to be reviewed and validated with regard to the requirements. This user interface has several graphical objects 41, 42 with which a user of the computer program can interact by making corresponding inputs at an input device. Such a graphical object 42 of the graphical user interface 40 can, for example, be a button that can be operated by a mouse or a touch display or by a keyboard shortcut.

[0062] In the exemplary embodiment of the Fig.Let the graphical object 42 of the graphical user interface 40 be a button labeled with the string "Route". When this graphical object 42 is actuated by input, an action should be executed in which all waypoints of the aircraft should be displayed.

[0063] To test this, 43 different test scenarios are first created by an industry expert, domain expert, or the client and made available to the test environment. The test environment includes optical character recognition (OCR) and output, and is also connected to a large language model via a corresponding interface.

[0064] The test scenarios 44 contain the expected system behavior of the computer program with regard to its graphical user interface 40 and are formulated in human-readable prose. Such a scenario could, for example, be formulated as follows: Scenario: Show route Given that the flight management computer is in the main view When the route button is pressed Then all waypoints will be displayed.

[0065] This scenario checks whether pressing the Route button triggers the desired action by the computer program, namely displaying all waypoints on the flight management computer. The prerequisite for this action is defined in the first line: the flight management computer must be in its main view.

[0066] Using the familiar prefixes "Given, If, Then", test environment 30 can now use a parser to convert this test scenario, formulated in natural language, into a machine-readable description of the system behavior. In doing so, the corresponding objects, such as the "Route button", are identified.

[0067] To execute the test procedure, the computer program is now started, so that the graphical user interface 40 is displayed on the display device 14. Using optical character recognition 46, this graphical output is then analyzed and corresponding character strings on the graphical user interface are recognized. These character strings belong to corresponding objects 41, 42 on the graphical user interface 40, such as the character string "Route" of button 42. At the same time, the corresponding area within which the object responds to user input is also identified.

[0068] Both the recognized character strings from the optical character recognition 46 and instructions generated from the corresponding test scenario, which is executed by the test environment 30, are now provided to a large language model 45 so that it can determine a corresponding action option. In the example above, the large language model 45 identifies the input such that the control element labeled "Route" should be operated by an input, whereupon the graphical representation should change by displaying the waypoints.

[0069] The large language model 45 thus provides the operation of the control element labeled "Route" as an action option.

[0070] This action option is now executed by the test environment 30, generating a corresponding input signal and making it available at the appropriate interface, thus triggering a simulated user input.

[0071] This changes the graphical user interface 40, which is also analyzed in a second pass. For this purpose, the detected elements on the graphical user interface are provided to the large language model 45 together with a corresponding instruction "Waypoints visible?" generated from the test scenario, which returns a corresponding validation as a response.

[0072] In another example, an autonomous testing process will be demonstrated, explicitly mentioning the dialogues between the test environment and the large language model. The test scenario's task is to autonomously execute navigation to the coordinates 54° latitude and 10° longitude.

[0073] The computer program to be tested is displayed in a graphical representation that shows 3 control elements in the form of buttons: Start Navigation, Cancel and Settings.

[0074] Optical character recognition (OCR) detects and identifies these three buttons. The task is known from the test scenario, so the following input is provided to the large language model: "Your task is: Start navigation to the coordinates latitude 54 and longitude 10. Currently, the following buttons are visible: Start navigation, Cancel, and Settings."

[0075] Furthermore, additional information or instructions can be passed to the large language model, such as: "Which button are you pressing?"

[0076] And, if applicable, information regarding the formats to be delivered: "Please reply in JSON format"

[0077] The large language model analyzes this information and responds with an action option: “Start navigation”

[0078] The test environment recognizes (e.g., using a Levenshtein distance) that the large language model has suggested clicking the "Start Navigation" button. This recognition of the suggestion from the large language model can be implemented using a simple parser in the test environment.

[0079] Based on the previously performed optical character recognition and the identification of the pixel coordinates of the respective buttons, the test environment can now generate an input signal that triggers a simulated click on the "Start Navigation" button. This input signal is then provided at the corresponding input interface.

[0080] After the simulated user input is triggered, the computer program under test changes its user interface, and the test environment begins a new run by first reloading the modified graphical user interface and analyzing it using optical character recognition. The modified graphical user interface now displays two text fields: one labeled "Lat." and the other labeled "Lon."

[0081] In the next step, the test environment communicates again with the large language model and transmits a corresponding instruction: "The user interface has changed. There are now the following text fields: Label 'Lat' and Label 'Lon'. What do you enter into these text fields? Please answer in JSON format."

[0082] Based on the target requirement, the large language model transmits the input of the specific coordinates in JSON format: {“Lat.” : “54”, “Lon.” : “10”}

[0083] The test environment automatically transfers these inputs and placed values ​​into the corresponding text fields.

[0084] Afterwards, the user interface is checked again by the test environment to verify whether the navigation to coordinates 54, 10 was started successfully or whether further interaction is necessary.

[0085] The prose texts that the test environment passes to the large language model, as well as the interpretation of the response from the large language model, were performed entirely automatically and are highly standardized. This means that even though human-understandable communication is practiced in the aforementioned examples, no human interaction is necessary or required. The only input for the test created manually by a human is the content of the test scenario.

[0086] Studies have shown that 80% of user interface errors are found by the first three to four pilot users in a user study. The present invention makes it possible to detect a significant proportion of these errors beforehand, before valuable resources are required. Simultaneously, the efficiency of a user interface can be determined directly from the user's perspective. Furthermore, the coupling between the software under test and the test environment is minimal, as no direct intervention in the source code of the software under test is necessary. Reference symbol list 10 Data processing system 11 electronic computing unit 12 Data storage devices 13 Display interface 14 Display device 15 Input interface 16 Input device 18 Data interface 20 Computer program (the one to be tested) 30 Test environment 40 graphical user interface 41 a first object of the graphical user interface 42 a second object of the graphical user interface 43 creators of the test scenario 44 test scenarios 45 large language model 46 optical character recognition 47 Test issue

Claims

Method for testing a computer program (20) with a graphical user interface (40), the method comprising the following steps: - providing a data processing system (10) having an electronic computing unit (11), a display interface (13) via which a display device (14) is connected or connectable, and an input interface (15) via which an input device (16) for user input is connected or connectable; - executing the computer program (20) to be tested on the data processing system (10), wherein the computer program (20) continuously provides an image data stream with a graphical user interface (40) comprising a plurality of objects at the display interface (13) of the data processing system (10); characterized in that,that a test environment (30) is provided for testing a system behavior of the computer program (20) under test with operation of at least one control element of the graphical user interface (40),- wherein the test environment (30) is provided with a second image data stream in addition to the first image data stream for the display device (14) via the display interface (13) of the data processing device,- wherein, by means of optical character recognition (46), control elements operable by an input device (16) in the graphical user interface (40) provided via the second image data stream are recognized as an object of the graphical user interface (40),- wherein at least the operable control elements recognized in the graphical user interface (40) are entered as input data into a large language model (45) connected to the test environment (30) via a data interface (18) with the stipulation thatan AI / as - PD002 / 20.10 action option for operating at least one control element of the graphical user interface (40) at the data interface (18),- wherein an input signal is generated by the test environment (30) depending on the action option provided at the data interface (18) for operating at least one detected control element, and- wherein the generated input signal is provided at the input interface (15) of the data processing system (10) to trigger a simulated user input for testing the computer program (20). Method according to claim 1, characterized in that by means of optical character recognition (46) an operating area of ​​the recognized operable control element in the provided graphical user interface (40) is also determined, which can be described by coordinates, within which the operable control element can be triggered by a user input at an input device (16), wherein the input signal is further generated depending on the operating area determined. Method according to claim 1 or 2, characterized in that the test environment (30) is performed on the data processing system (10) or on an external device connected to the data processing system (10). Method according to one of the preceding claims, characterized in that information displayed in the graphical user interface (40) provided via the second image data stream is recognized as objects by means of optical character recognition (46), wherein the information recognized in the graphical user interface (40) is also entered into the large language model (45) via the data interface (18) with the proviso of obtaining an action option for operating at least one control element of the graphical user interface (40) at the data interface (18). Method according to one of the preceding claims, characterized in that the test environment (30) has at least one test scenario (44) which includes system behavior of the computer program (20) to be tested with operation of at least one control element of the graphical user interface (40), wherein, depending on the test scenario (44), at least one instruction to the large language model (45) is generated and this is also entered into the large language model (45) via the data interface (18) in order to obtain an action option for the operation of at least one control element of the graphical user interface (40) at the data interface (18). Method according to claim 5, characterized in that after triggering the simulated user input, a modified second image data stream with a modified graphical user interface (40) is provided, wherein objects in the modified graphical user interface (40) are recognized by means of optical character recognition (46) and wherein the modified graphical user interface (40) is verified with regard to the system behavior described in the test scenario (44) depending on the recognized objects. Method according to claim 5 or 6, characterized in that the steps up to triggering a simulated user input are repeated with a modified graphical user interface (40) in order to generate a further modified graphical user interface (40). Method according to one of claims 5 to 7, characterized in that, by means of optical character recognition (46), character strings of the objects contained in the graphical user interface (40) are first recognized, wherein the objects describing the system behavior in the test scenario (44) are then assigned to the corresponding objects within the graphical user interface (40) depending on the recognized character strings of the objects contained in the graphical user interface (40). Computer program (20) with program code means, configured to carry out the method according to one of the preceding claims, when the computer program (20) is executed as a test environment (30) on a data processing system (10). System comprising a data processing system (10) on which a computer program (20) to be tested can be executed with a graphical user interface (40), and a test environment (30) for testing the executed computer program (20), wherein the system is configured to carry out the method according to one of claims 1 to 8 when the computer program (20) to be tested is executed on the data processing system (10).

Images