Method for automating a software test

Abstract

The present disclosure relates to a computer-implemented method for automating a software test, in particular a software module test, an integration test and / or a software end-to-end (E2E) test, comprising: identifying, in particular locating, at least one object using at least one bounding box on the basis of at least one initial image which shows a user interface to be tested, in particular a graphical user interface (GUI), assigning at least one label to the at least one identified object in order to generate at least one marked image, and processing the marked image using a control instruction, and, on the basis of the processing, generating at least one trigger instruction which, when executed, is configured to initiate at least one interaction with the at least one identified object, in particular in order to carry out the software test.

Description

[0001] Methods for automating a software test

[0002] The present invention relates to a computer-implemented method for automating one or more software tests or for, in particular, automatically testing software, especially software module tests, integration tests and / or software end-to-end (E2E) tests.

[0003] In software development, the final validation of the software plays a particularly important role. Besides so-called module tests, in which the software manufacturer tests, for example, their individual software components (also called modules or software modules), end-to-end tests (E2E tests) are especially important, in which, for example, the final software is tested from the end user's perspective. These tests can be described and documented in test suites such as Jira Xray. The tests can be described in natural language so that a person without subject-matter expertise, and therefore not a software expert, can execute and interpret them.

[0004] Furthermore, integration tests, or software integration tests, can be important. Integration tests are a type of software test where multiple components and / or modules of an application are tested together to ensure, for example, that they work together correctly. These tests can focus on verifying interfaces and / or interactions between the individual modules, rather than concentrating on the functionality of the individual modules themselves.

[0005] The description of these tests can follow a defined syntax: First, the action to be performed is described. Optionally, the required data can be specified. Finally, the expected result of the action, which can also be called an "assertion," is recorded. An example of such a description could look like this: The website www.mybank.de is open. The user logs in with the username "Max Mustermann" and the password "123456," e.g., "I log in with the username: Max Mustermann and the password: 123456" (action). The expected result (assertion) is that the "Account Overview" window opens.

[0006] In a standard test run, a human tester would click through this step and, if successful, confirm it with "Test passed" or similar. The test suite would then save the result with a timestamp. However, existing systems for conducting these tests often require manual intervention and extensive preparation to identify the software elements to be tested and to create the corresponding test scripts. This results in significant time and cost expenditure, especially in complex software environments with numerous user interfaces and interaction elements.

[0007] According to established methods, software testing is often performed by writing and executing test scripts that simulate specific scenarios and interactions with the software. These test scripts must be created and maintained manually, which is error-prone and time-consuming. Furthermore, they require detailed knowledge of the software under test and its user interface. While automated testing tools exist, they are often limited to specific software environments and do not offer a comprehensive solution for the automatic identification of and interaction with various user interface elements. This results in limited flexibility and efficiency in software testing.

[0008] Automating these tests is also difficult and / or time-consuming, as any further development, adaptation, and / or revision of the software could lead to input fields such as "password" shifting and / or receiving a different ID in the software. Hardcoded tests then require extensive adjustments. Layout changes also cannot be considered, compensated for, or balanced out. Consequently, test teams must be commissioned to perform the tests manually. Depending on the budget and complexity, these testers work on-site or offshore.

[0009] However, this approach has several disadvantages. Costs are high, and the testers' availability is limited. Due to these limited resources, it is often only possible to perform a small number of tests from a large number of possible test combinations. Furthermore, there are planning risks in test management due to staff shortages. Errors sometimes only become apparent to the customer.

[0010] Language barriers complicate both the creation of the tests, which may have to be in English (or another language), and their administration, for example when assessing whether the Arabic language version is correct.

[0011] The categories of tests under consideration include functional tests, layout tests, and / or logical tests. Functional tests, for example, check whether the account balance in an online account changes correctly after a transfer. Layout tests ensure, for instance, that no images are missing from a user manual and / or that there are no unnecessary blank lines in the text. Logical tests check, for example, whether a vehicle's user manual contains chapters that are irrelevant to that model, such as a chapter about a convertible top in a manual for a model that isn't available as a convertible.

[0012] The rapid development of Large Language Models (LLMs), such as ChatGPT, and multimodal LLMs, such as GPT-V or similar, has opened up new technical possibilities. However, none of these systems alone enables partial or complete test automation. Current solutions like GitHub's "CoPilot" or OpenAL's "ChatGPT" and "Codex" can offer support, but they also have their drawbacks. These include limitations in the adaptability of the underlying model, data privacy and security risks, and limited support for specialized tasks.

[0013] The development of new AI models is progressing rapidly. A model that seems ideal today can be superseded by a better one tomorrow. Despite the aforementioned disadvantages or missing functionalities, existing solutions often attempt to bind customers to the offered model, even though other AI models might be better suited to the customer's specific use case.

[0014] The object of the present invention is to overcome the disadvantages of the known prior art and in particular to provide an improved computer-implemented method for automating software testing, especially software module testing, integration testing and / or software end-to-end (E2E) testing.

[0015] The problem is solved by the features of the independent claims. The dependent claims describe preferred embodiments. Further aspects, advantages, and features become apparent from the dependent claims, the description, and the accompanying drawings.

[0016] One aspect of the invention relates to a computer-implemented method for automating software testing, in particular software module testing, integration testing and / or software end-to-end (E2E) testing, comprising:

[0017] Identifying, in particular locating, at least one object using at least one bounding box, based on at least one initial image showing a user interface to be tested, in particular a graphical user interface (GUI),

[0018] Assigning at least one label to the at least one identified object to generate at least one tagged image, and

[0019] Processing the selected image using a control instruction, and, based on the processing,

[0020] Generating at least one trigger instruction that is configured to initiate at least one interaction with the at least one identified object upon execution, in particular for the purpose of performing the software test.

[0021] One aspect of the invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause it to execute the method according to the invention according to one or more of the aspects and / or embodiments described herein.

[0022] One aspect of the invention relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to execute the method according to the invention according to one or more of the aspects and / or embodiments described herein.

[0023] The invention relates to a computer-implemented method for automating software testing, in particular software module testing, integration testing, and / or end-to-end (E2E) software testing. One objective may be to make such software tests faster, more cost-effective, more efficient, and / or more reliable, as well as to create a precise and / or automated test environment.

[0024] A "software test" can be the testing, verification, inspection, and / or evaluation of the execution or function of a software or (software) application and / or the fulfillment of one or more functions, components, modules, and / or aspects of the software or (software) application. Software tests can ensure or guarantee error-free operation and quality, as well as ensure that the software meets the necessary requirements and / or that any errors are eliminated. They help to identify defects, verify usability, and / or ensure the stability of the software. Software tests are generally performed manually or automatically (especially traditionally using hardcode) and can cover various test levels that focus on different parts of the software product. The most important types of tests include software unit tests, integration tests, and / or end-to-end (E2E) tests.

[0025] A software module test, also called a software unit test or unit test (these terms can be used interchangeably here), can be understood as a test in which, for example, individual components, units, parts, and / or modules of a software are checked, evaluated, verified, and / or tested, particularly in isolation, individually, and / or separately. A module can be the smallest testable unit of a software, typically a single function and / or method. Verifying the correct functionality of each individual module can ensure that each module meets the specified requirements. These tests can aim to ensure early error detection in the development phase in order to minimize the costs and effort of error correction.

[0026] A software integration test, or integration test, can be understood as a test in which the interactions between multiple modules—e.g., individual components, units, and / or parts of the software—are checked, evaluated, verified, and / or tested. The modules can be integrated or tested step by step, for example, to detect errors early on. Integration tests can be performed at various levels, from the integration of individual modules to the integration of large (sub)systems. The goal of integration tests can be to ensure that the modules work together correctly when integrated, to verify the correct interaction of modules, to identify errors that arise from the integration of modules, or to ensure that interfaces between modules are correctly implemented.

[0027] An end-to-end (E2E) software test can be understood as a test in which the entire (software) application is examined from beginning to end. E2E tests can essentially simulate real-world usage scenarios and / or test the software or (software) application under conditions that reflect actual user or end-user interaction. These tests can cover virtually all levels of the system, from the user interface and business logic to databases and / or external systems. The goal of E2E tests can be to ensure that the software functions correctly from the end-user's perspective, to verify the application's entire workflow processes, to ensure that all system components work together, and / or to examine, evaluate, and / or verify the user experience and system functionality under real-world conditions.The software tests may include one or more of the following: functional tests to verify at least one function of the at least one object, and / or.

[0028] Layout tests for the visual verification of one or more elements of the user interface (Ul) under test, in particular the graphical user interface (GUI).

[0029] Functional tests can be used to verify the functionality of software. These tests check whether the software correctly performs the expected functions. They focus on verifying the specifications and requirements placed on the software. Layout tests focus on the visual verification of the user interface (GUI - Graphical User Interface). They ensure that the software's visual elements are displayed correctly and that usability is guaranteed. Layout tests include, among other things, the following aspects: checking whether all visual elements (such as buttons, text, images, etc.) appear correctly; ensuring that the elements appear in the correct positions on the screen and / or are displayed correctly at different screen sizes and resolutions.Checking whether texts are legible / error-free and whether the user interface is accessible to all users, including people with disabilities. Ensuring that the user interface is displayed correctly on different devices (e.g., desktop, tablet, smartphone, etc.).

[0030] By combining one or more of the different types of tests described herein, the quality, efficiency and / or reliability of the software can be ensured throughout all development phases.

[0031] The method according to the invention may include:

[0032] Identifying, in particular locating, at least one object using at least one bounding box, based on at least one initial image showing a user interface to be tested, in particular a graphical user interface (GUI), and

[0033] Assigning at least one label to the at least one identified object to generate at least one tagged image.

[0034] "Identification" can be understood as recognizing, detecting, capturing, and / or determining at least one object, for example, in at least one source image—that is, confirming that this object is present in the source image. Alternatively or additionally, identification can also involve localization, i.e., determining the object's position, location, and / or coordinates, for example, within the source image. In an optional, i.e., non-restrictive, example, identification can be achieved using a sliding window within the at least one source image. This window, e.g., of a fixed / known size (length, width) or a variable size, can be used, which scans the at least one source image, for example, by moving it. However, other methods of identification are also conceivable, e.g., through the AI ​​model itself.The identification described herein can optionally include (additional) text recognition of the at least one source image, in particular optical character recognition (OCR), to recognize text on / in the at least one source image and / or the at least one object. For example, this allows a button labeled "Button" to be recognized as such and / or more quickly. In this example, the identification, in particular localization, using the at least one bounding box and the identification by means of text recognition can be performed essentially simultaneously, in particular in parallel, which can save additional time and resources.

[0035] A "bounding box" can be a frame, typically rectangular, used to enclose, contain, and / or define a specific object, for example, in an image and / or on a user interface. This bounding box can be used to identify and / or track the position, arrangement, and / or size of one or more objects. For example, a bounding box can be drawn around an object or element, such as a button in a user interface, to mark / label it for further analysis or processing. Using a bounding box, specific objects / elements within complex user interfaces can be precisely identified, which is crucial for test automation. The bounding box can serve as a visual aid that indicates the exact coordinates and / or dimensions of the identified object.This can be done by passing at least one output image to an image processing algorithm or to an AI model, e.g. an object recognition model, in particular as input (see below), which generates the bounding box, performs operations assigned to it, e.g. the creation of the number vector described herein, and / or reports back or assigns the position of the object.

[0036] It may happen that more than one bounding box, in particular several or a plurality of bounding boxes, is used / provided for / inserted, e.g., at least two. In this case, the procedure may include:

[0037] Adjusting and / or setting the number of bounding boxes in the original image. This adjustment may include:

[0038] Merging at least partially overlapping bounding boxes in the original image, and / or

[0039] Removing bounding box artifacts from the original image.

[0040] However, it should also be noted that at least one bounding box can be adjusted, modified and / or changed, e.g., in terms of position, alignment / orientation and / or dimensions (e.g., size).

[0041] The bounding box artifacts can comprise one or more of: a bounding box that is inaccurately placed with respect to the at least one object, an ambiguous bounding box that defines the at least one object, a bounding box that partially defines the at least one object, a bounding box that is falsely detected and does not contain an object and / or is not associated with an object, a bounding box that is faulty and / or insufficiently dimensioned with respect to the at least one object, wherein in particular the at least one bounding box has dimensions that are substantially defined or specified by the at least one object.

[0042] A "user interface" or "interface" can be an interface or part of a technical system with which a user can interact to control, configure, monitor, and / or use one or more functions of that system. The user interface can include means by which the user can input information and / or receive information from the system. A user interface can be at least a part, section, and / or area of ​​a computer screen (e.g., computer monitor) or a mobile device screen (e.g., mobile phone, smartphone) and / or be configured to display a software and / or web application, a website, an application ("app"), or the like. In software development, the user interface can refer to the graphical representation of menus, buttons, text fields, scroll bars, etc.Scroll bars, input fields, controls / input elements, buttons, icons, menus, checkboxes, radio buttons, tabs, text areas, sliders, and / or other interactive elements that enable interaction with the software. A "Graphical User Interface," or GUI for short, can be a specific type of user interface that uses visual indicators and / or graphical elements to facilitate interaction with the software. GUIs make software operation more intuitive by using icons, windows, menus, and / or other graphical elements that are displayed to the user visually and / or, if applicable, haptically (e.g., through touchscreens, vibrations). A GUI allows users to interact with the software through visual cues and / or direct manipulation of screen objects, rather than entering text-based commands.

[0043] A source image, such as a screenshot (e.g., a JPEG), can be a static representation of the current state of a user interface, such as a computer screen, and / or an image at a specific point in time. A screenshot can be a visual representation of the state of the user interface (UI) or the GUI. Screenshots are often used to document or record the exact visual state of a screen and / or window at a particular moment. The source image, such as the screenshot, can be captured, created, and / or generated by the client. It is also possible to have the source image, such as the screenshot, generated by the client; for example, a client can trigger the creation of the source image.

[0044] The at least one object can be an input element, a setting element, and / or a control element of the software environment under test, in particular the graphical user interface (GUI), for example, an interactive element. Such elements can, for example, allow a user to interact with at least one part of the software environment or the user interface or GUI and / or to make changes, settings, modifications, and / or adjustments. An input element can, for example, allow a user to enter and / or submit data or information, e.g., in the form of text, text fields, input fields, text areas, drop-down menus, checkboxes, etc. A setting element or rule element can, for example, allow a user to configure settings, values, and / or parameters, e.g., a scroll bar, scroll bars, tabs, sliders, etc.A control element can be a slider that allows users to adjust values, data, information, and / or parameters within a specific range, or radio buttons that allow users to customize behavior. A control element can, for example, allow a user to perform actions within the application and / or start, stop, and / or control specific functions or processes within the software. Examples include buttons, icons, menus, and checkboxes that the user can click to perform an action (e.g., "Submit," "Cancel," "Start"); menus that offer the user various commands or actions to choose from. Other interactive elements are also possible.

[0045] After identification, at least one object can be assigned at least one label, thereby generating at least one tagged image. A "label" can be understood as an identifier, marker, and / or indication that creates a clear or unambiguous reference for the object, thus ensuring traceability, attribution, and / or characterization. The at least one label can be used for image annotation, e.g., of the at least one source image.

[0046] The method according to the invention may include:

[0047] Processing the selected image using a control instruction, and, based on the processing, and

[0048] Generating at least one trigger instruction that is configured to initiate at least one interaction with the at least one identified object upon execution, in particular for the purpose of performing the software test.

[0049] "Processing" can be understood as checking, analyzing and / or processing the marked image, for example in connection with a control instruction which, for example, directs, influences, coordinates, controls and / or manages the processing process.

[0050] For example, during processing, it can be recognized that at least one object is a button, e.g., a green button, and which label is assigned to this button. It should be noted that this refers to an example of a "green button" in a user interface. However, this is merely exemplary or optional and is only intended to clarify the present invention. Numerous examples / applications are conceivable, and the present invention is not limited to "green buttons."

[0051] The control instruction can be a prompt-based instruction, in particular a prompt, i.e., for example a text command or an input request, especially a text-based one, which can influence the processing.

[0052] The following is a non-restrictive example of a control instruction. It should be clear that the individual statements / elements mentioned in the prompt are not to be understood as restrictive; that is, what follows is merely optional and refers to the "green button" example for illustrative purposes. For example, the prompt mentions two screenshots; however, the invention is not limited to two screenshots, meaning these are by no means essential. The same applies to the interaction types, labels, etc. For the following example, it is assumed that a green button in a user interface, such as a computer screen or a GUI, is to be clicked.The control instruction can, for example, comprise a first part, which can be called the system prompt, and a second part, which can be called the user prompt, although the two-part structure is optional, and the user prompt can additionally include a task (or several tasks or a list of tasks).

[0053] System prompt

[0054] You are an AI agent with capacity to see the user's screen, click buttons, type text and scroll. The user can ask you to carry out tasks:

[0055] You will see the user's screen, then you will first think in high level what are the steps you will need to follow to carry out the task. Then, you will be given annotated screenshots with codes mapping buttons, text, input fields and so on. You can choose to click on buttons and text, type in input fields, scroll up or down, and get a refreshed update of the screen to continue until the task is completed. In order to type in an input field, you MUST first click on it. Never attempt to submit a form unless you are told to do so.

[0056] You are always very short and concise in your writing.

[0057] User-Prompt

[0058] Task: I click on the green button

[0059] Here are two screenshots of the screen. One is tagged with labels like A1, B4, A17 on each interactive item, please do two things:

[0060] A. High level list of steps to follow, using a numbered list in English text

[0061] B. A list of actions to execute, each action being one of:

[0062] - [CLICK A 1] (To click the A 1 button for example)

[0063] - [TYPE "message"] (To type "message") - [SCROLL <direction> <speed>] (To scroll up or down, at normal or quick speed. For example [SCROLL down quick])

[0064] Those are the ONLY options you have, work with that. Do NOT give actions you consider optional. Make sure the order of actions follows the order of the high-level list. If you want to tick a checkbox or select an option, ONLY use labels that start with the letter A, such as A1 or A7. If you want to click or type on an element, that is not or will not be on the screen, return a list that consists only of a [REFRESH]. If the user asked you to scroll until an element is visible, but the element is already visible, return a list that consists only of a [TRUE]. Do not scroll unless the user specified so.

[0065] Your response should be divided up in sections A and B like the prompt.

[0066] The control instruction can be configured to enable and / or trigger the assignment of at least one label to at least one identified object. The control instruction can also enable the identification of which object and / or corresponding label is meant. This can trigger a recognition action, such as finding or locating the green button in a user interface, and at least partially providing or assigning a corresponding action or interaction. This can be done, for example, via a backend and / or a program or application that is, for example, stored in the backend.

[0067] As can be seen, the control instruction can include multiple interaction types, such as a list of interaction types, e.g., clicking, typing, entering, scrolling, or the like. The control instruction can be configured to trigger an output of at least one interaction type from among the multiple interaction types. The control instruction can thus ensure that, for example, during processing, an interaction type is selected that is best suited for the at least one object. This at least one interaction type can either be output directly, e.g., as output, or integrated into the trigger instruction. This trigger instruction can therefore also be referred to as a result, or rather, it can include the at least one interaction type as a result. The trigger instruction can thus include at least one result.The trigger instruction can be indicative of at least one interaction type with the at least one identified object and can be indicative of the at least one label assigned to this, i.e., the at least one identified object. The procedure can optionally also include the step of outputting the at least one trigger instruction or the at least one result. The at least one interaction type in the trigger instruction can be linked to the at least one label and / or the at least one object (e.g., via the at least one bounding box). An optional classification of the at least one identified object, e.g., as a button, can also be included. This trigger instruction can then be (further) used, processed, and / or executed to actually initiate the at least one interaction, e.g., by a program or application.An application is stored in a backend. A backend can be described as the functional part responsible for processing. The backend can be local, e.g., on-premises, or external, e.g., outsourced. However, it is also conceivable that the trigger instruction is (further) used, processed, and / or executed by another program, application, or computing device, e.g., a computer, in order to actually initiate at least one interaction.

[0068] An example of a trigger instruction could therefore look like this:

[0069] [CLICK A10]

[0070] *Where in this example "CLICK" represents the interaction type "click" and "A10" can correspond to at least one label, which, for example, has been assigned to at least one object "green button". The number "10" can, for example, refer to the presence of other, e.g., nine, additional labels.

[0071] It is clear that the label with the value "A10" can also change (the same applies analogously to the interaction type "CLICK"). For example, the layout of the user interface, e.g., the GUI, can change, and the exemplary green button can be positioned differently, e.g., if the green button is replaced by an icon, a menu, etc., or if the green button disappears from the user interface, or similar. Accordingly, a different label can be assigned to the green button, e.g., depending on the object / labels in the at least one initial image. The method according to the invention is thus robust against layout adjustments of the user interface and can dynamically / flexibly react to and / or compensate for any adjustments, extensions, changes, shifts, innovations, additions, modifications, replacements, and / or (partial) deletions, e.g., of one or more input, control, and / or setting elements.It should be noted that the original image can be used for processing, particularly in addition to the labeled image. The process can therefore include processing the labeled image and the at least one original image using a control instruction. This can make it possible to take into account image content or image components that are, for example, obscured by the at least one label, so that this image content or image components are not lost. The corresponding result of the processing is thus more accurate, precise, and / or complete. During processing, for example, a comparison of the labeled image (e.g., the labeled image) with the at least one original image (e.g., the unlabeled or label-free original image) can take place.

[0072] The method according to the invention may include:

[0073] Initiating at least one interaction with the at least one identified object, particularly for performing software testing. This initiation can be, for example, clicking a button and can be performed using a computing device, such as a computer, an application, and / or a backend, such as the functional part, which processes the trigger instruction. Accordingly, providing the at least one trigger instruction to a computing device to initiate the at least one interaction can also be provided. Alternatively, the trigger instruction can be stored in memory, for example, for later use. This trigger instruction can also be used for or contribute to hardcoded automation, for example, via another application on the computing device.

[0074] Identification can include:

[0075] Assigning at least one, in particular multidimensional, number vector to the at least one bounding box, and / or

[0076] Classifying at least one object in at least one source image.

[0077] In other words, the at least one bounding box can comprise a numerical vector, particularly a multidimensional one, consisting of, for example, several elements, which is indicative of a position (and / or orientation) of the at least one bounding box and / or the at least one identified object in the at least one source image. Classification can involve determining a type, genus, shape, color, dimension or size, and / or at least one property of the at least one object. The classification can include a confidence level, e.g., in the range of 0-100%. For example, it can be identified that the at least one object in the at least one source image is a green button, what its dimensions / shape are, and / or where it is located, positioned, and / or how it is oriented.

[0078] Identification can include:

[0079] Creating at least one initial image, in particular by means of a screenshot, of the user interface to be tested (e.g., a computer screen, a website, an application ("APP"), or the like), especially the GUI, wherein the at least one initial image is created at the beginning of each software test. As already mentioned, the creation can be performed by the customer or triggered by the customer, e.g., a client can initiate the creation of the initial image. The initial image can be essentially unlabeled, e.g., without a label or before the label is assigned. The at least one initial image can be a label-free initial image. This approach can, for example, ensure that any layout adjustments to the user interface or GUI, e.g., adjustments, extensions, changes, shifts, innovations, additions, modifications, replacements, and / or (partial) deletions, are not affected.B. from one or more input, control and / or setting elements. At the beginning of each process, particularly immediately before carrying out the inventive method or each software test, a description can be provided, or it can mark the beginning of the inventive method, e.g., the creation of the initial image. If the method is executed again, an initial image can first be created again, and so on.

[0080] Identification and / or assignment can be performed using a neural network. It should be clear that the sub-steps of identification and / or assignment, such as localization, assigning the numerical vector, classification, creating the initial image, optional traversal using a sliding window, optional text recognition, adjusting and / or setting the number of bounding boxes in the initial image, etc., can also be performed using this network. The neural network can be an object recognition model based on a transformer architecture, specifically configured for real-time and / or end-to-end object recognition. A suitable, yet non-restrictive, neural network could be "RT-DETR" (Real-Time Detection Transformer), which can correspond to a transformer-based model for object recognition. RT-DETR can extract contextual information from an image, e.g.,RT-DETR understands an initial image, which can lead to more accurate detections. RT-DETR is designed to perform object detection essentially in real time. Unlike traditional object detection models that require multiple steps such as region proposal, feature extraction, and classification, RT-DETR operates in a single step. This simplifies the architecture and can increase processing speed. RT-DETR can therefore quickly and accurately detect objects in images and / or videos, making it particularly useful for applications that require fast and / or precise detection. It can deliver the required results in a fraction of the time needed by traditional models.However, a different neural network can also be used, so the customer is not unnecessarily tied to specific models like RT-DETR, but rather the models can be easily interchangeable, for example, because another model is / would be more suitable (e.g., in the future). The neural network can be part of the backend.

[0081] The neural network can be pre-trained using image data, particularly open-source data, that is indicative of one or more different user interfaces, especially GUIs, where the image data is publicly available and / or accessible. For example, source images, e.g., screenshots (e.g., as JPEGs), showing several different user interfaces, especially GUIs, can be used for training or pre-training.

[0082] The term "open-source" can be understood to mean that the corresponding training data is publicly accessible, freely usable, modified, and / or distributed, e.g., usable by anyone for training purposes, particularly without legal restrictions or limitations. This can fulfill the following (basic) criteria:

[0083] - Available for free download;

[0084] Commercial use is not excluded (license fees may apply)

[0085] - May be copied and / or modified.

[0086] - Disclosure of training resources and / or

[0087] - Disclosure of the training code.

[0088] The neural network, in particular the object recognition model, can also be referred to as a KL model, and these terms are interchangeable within this context.

[0089] The procedure may optionally include determining, in particular setting, the AI ​​model, especially:

[0090] Selecting and / or defining the AI ​​model based on the nature of the task to be performed

[0091] Software testing, and setting and / or adjusting model parameters, especially hyperparameters, for the AI ​​model.

[0092] The term "determining" can therefore be understood as i) selecting, specifying, and / or defining a AI model and / or, in particular subsequently, ii) setting, specifying, and / or adjusting hyperparameters for the AI ​​model. Determining, in particular according to i) and / or ii), can be carried out based on the software test to be performed, especially at least partially.

[0093] When selecting the AI ​​model according to i), various factors / criteria can be taken into account, e.g. one or more of the following:

[0094] - the type of AI model, e.g. neural network, decision tree method, support vector machine, open-source model and / or model derivative based on a transformer architecture, etc.

[0095] - the size of a KL model

[0096] - Architecture of the AI ​​model;

[0097] - Accessibility, e.g., open source models or documentation of the models;

[0098] - Trained, or at least partially trained or untrained models;

[0099] - Security and / or data protection aspects; and / or

[0100] - Quality or quantity of the training data.

[0101] Alternatively or additionally, when setting up the Kl model according to ii), in particular after selecting a suitable Kl initial model, one or more hyperparameters can be used to further customize the model.

[0102] "Hyperparameters" can be values, parameters, data and / or configurations that can influence the training process and / or the behavior of the AI ​​model.

[0103] Hyperparameters can be configured to remain unchanged by the training data being processed; instead, they can be external to the model and / or set before training begins. Consequently, while hyperparameters govern the neural network's training process, they may not be part of the resulting base model, and / or it may not be possible to retrospectively determine which hyperparameters were used to train a model. The hyperparameters of a computer model can be important because they provide a controllable way to tailor a model's behavior to produce the desired outcome for a specific use case. Rather than incurring the considerable effort and expense of developing a custom model, the hyperparameter tuning process offers the opportunity to reconfigure a base model to better meet specific expectations.

[0104] The model parameters, especially the hyperparameters, for the neural network, e.g. RT-DETR, can be chosen, set, provided and / or adapted as follows in a purely optional and non-restrictive example:

[0105] General training hyperparameters

[0106] Optimizer: AdamW

[0107] Base Learning Rate: e.g., 1e-4 (or in the range of e.g., 1e-5 to 1e-3) Learning Rate of Backbone: e.g., 1e-5 (or in the range of e.g., 1e-6 to 1e-4) Freezing Batch Normalization (BN): e.g., True (Boolean)

[0108] Linear Warm-up Start Factor: e.g., 0.001 (or in the range of e.g., 0.0001 to 0.01) Linear Warm-up Steps: e.g., 2000 (or in the range of e.g., 1000 to 5000)

[0109] - Weight Decay: e.g., 0.0001 (or in the range of e.g., 0.00001 to 0.001) Clip Gradient Norm: e.g., 0.1 (or in the range of e.g., 0.01 to 1.0) EMA Decay: e.g., 0.9999 (or in the range of e.g., 0.99 to 0.9999)

[0110] Hybrid Encoder Specific Hyperparameter

[0111] - Number of AI Fl (Attention-based Intra-scale Feature Interaction) layers: e.g. 1 (or in the range of e.g. 1 to 3)

[0112] - Number of RepBlocks in CCFF (CNN-based Cross-scale Feature Fusion): e.g. 3 (or in the range of e.g. 1 to 5)

[0113] Embedding dimension: e.g., 256 (or in the range of e.g., 128 to 512) Feedforward dimension: e.g., 1024 (or in the range of e.g., 512 to 2048)

[0114] - Number of heads in Multi-head Attention: e.g. 8 (or in the range of e.g. 4 to 16)

[0115] - Number of feature scales: e.g. 3 (e.g. Fixed)

[0116] Decoder Hyperparameter

[0117] - Number of decoder layers: e.g. 6 (or in the range of e.g. 3 to 12)

[0118] - Number of queries: e.g., 300 (or in the range of, e.g., 100 to 500) Number of decoder points: e.g., 4 (for example, Fixed)

[0119] Loss Function Hyperparameter

[0120] Class Cost Weight: e.g. 2.0 (or in the range of e.g. 1.0 to 5.0)

[0121] - Alpha in Class Cost: e.g., 0.25 (or in the range of e.g., 0.1 to 0.5) Gamma in Class Cost: e.g., 2.0 (or in the range of e.g., 1.0 to 3.0) Bounding Box Cost Weight: e.g., 5.0 (or in the range of e.g., 1.0 to 10.0) Global Cost Weight: e.g., 2.0 (or in the range of e.g., 1.0 to 5.0) Class Loss Weight: e.g., 1.0 (or in the range of e.g., 0.5 to 2.0)

[0122] - Alpha in Class Loss: e.g., 0.75 (or in the range of e.g., 0.5 to 1.0) Gamma in Class Loss: e.g., 2.0 (or in the range of e.g., 1.0 to 3.0) Bounding Box Loss Weight: e.g., 5.0 (or in the range of e.g., 1.0 to 10.0) GloU Loss Weight: e.g., 2.0 (or in the range of e.g., 1.0 to 5.0)

[0123] Denoising Hyperparameter

[0124] Denoising Number: e.g., 200 (or in the range of e.g., 100 to 300) Label Noise Ratio: e.g., 0.5 (or in the range of e.g., 0.1 to 0.5) Box Noise Scale: e.g., 1.0 (or in the range of e.g., 0.5 to 2.0)

[0125] SAHI (Slicing Aided Hyper Inference) hyperparameter

[0126] Overlap Height Ratio: e.g. 0.8 (or in the range of e.g. 0.1 to 0.9)

[0127] Overlap Width Ratio: e.g., 0.8 (or in the range of e.g., 0.1 to 0.9) Post-process Class Agnostic: e.g., False (e.g., Boolean) Slice Width: e.g., 5 (or in the range of e.g., 1 to max) Post-process Merge: e.g., True (e.g., Boolean)

[0128] Post-process NMS loU Threshold: e.g. 0.5 (or in the range of e.g. 0.3 to 0.7)

[0129] The adaptation, and in particular the training, of the AI ​​model can be done using open-source training data, or it can already be trained or pre-trained (and possibly refined with it). This open-source training data can also be referred to as "the open-source training data assigned to the user interface, especially the GUI."

[0130] The adaptation according to the invention may include:

[0131] Adjusting and / or fine-tuning through prompt tuning, fine-tuning and / or Retrieval Augmented Generation (RAG).

[0132] Prompt tuning allows the AI ​​model to be refined by using prompts and / or instructions, for example, during training. These prompts can provide the AI ​​model with guidance on what kind of tasks it should solve and how to solve them. For example, prompts can help the AI ​​model adapt to specific applications and / or requirements. Fine tuning can be understood as fine-tuning the AI ​​model for more specific data and / or tasks after it has already been trained on more general data.

[0133] With "Retrieval Augmented Generation (RAG)," the AI ​​model can be trained both to generate text and to search for relevant information in large text datasets. This allows the AI ​​model, for example, to generate text based on existing knowledge while simultaneously retrieving relevant information to improve its output.

[0134] The neural network, and in particular the object recognition model, could be an open-source model such as RT-DETR. Other open-source models would also be conceivable; that is, the invention is not limited to this specific model. Choosing an open-source model avoids unnecessarily binding the customer to specific models and allows for easy interchangeability, for example, if another model becomes more suitable in the future.

[0135] Adapting, and especially training, the AI ​​model can include refinement. This refinement can be based on a low-rank adoption (LoRA) approach. "Refinement" can be understood as further refinement or retraining aimed at improving and / or adapting the AI ​​model to achieve, for example, better performance or adaptability for specific use cases. The LoRA approach, for instance, can be based on the premise that the AI ​​model can be improved by introducing low-rank adaptations or modifications. This means that specific parts of the model are adapted in a way that reduces complexity or enhances specific features to improve the model's performance without, for example, significantly increasing the overall size of the model.

[0136] The process may include:

[0137] Creating at least one test completion image of the user interface to be tested, in particular the GUI, after initiating at least one interaction or after completion of the interaction, and if necessary.

[0138] Storing, specifically saving, the test completion image in memory. A test completion image can be generated, for example, using a screenshot, such as a JPEG. The test completion image can indicate the end or completion of the software test or interaction, or at least indicate that the software test was (at least partially) successful. The test completion image can be generated automatically, for example, via the backend.

[0139] The processing of the selected image (and, if applicable, the source image) using the control instruction and / or the generation of the at least one trigger instruction can be accomplished using a language model, in particular a Large Language Model (LLM), preferably a multimodal language model. The language model can be part of the backend. A language model can be, for example, an online model, e.g., GPT-40, or an offline model. A Large Language Model (LLM) can be configured to understand, generate, and / or respond to language. The term "multimodal" can indicate that the model is capable of processing or integrating different types of data, information, formats, and / or modalities. These modalities can include, for example, text, images, audio, video, and / or other forms of data.A multimodal LLM can therefore not only work with text-based input, but also understand and react to visual and / or auditory information. A multimodal LLM can process various types of input.

[0140] For example, it can process text, images, and audio simultaneously or sequentially. It can also generate various types of output, such as text-based answers to text-based questions, descriptions of images, answers to questions related to images, and so on. The model can combine information from different sources to provide a more comprehensive and accurate answer. For example, it can analyze image data and use this information to provide a text-based explanation or description, etc.

[0141] A "test suite" can be understood as a software application or working environment that provides users with tools and / or functions. At least one test suite can be maintained on the computing device on which the user interface (GUI) being tested is located.

[0142] The method according to the invention and / or one or more steps of the method according to the invention can be accessed and / or executed via the test suite in the form of a plug-in. A plug-in can be understood as an extension or an additional module that is integrated or integrable into the test suite to provide, for example, additional functions or support for specific tasks. Plugins can thus extend the functionality of the test suite. Users can thereby access the method according to the invention and / or one or more steps of the method according to the invention directly within their familiar test suite, without having to use a separate application. The plug-in extends the functions of the test suite to enable the integration and / or use of the method according to the invention and / or one or more steps of the method according to the invention. Examples of test suites include, for example...Jira Xray or Azure DevOps Services.

[0143] The method according to the invention can also include a caching algorithm that can increase efficiency and performance and avoid unnecessary computational and processing steps. In the caching described herein, data, information, values, and / or parameters can be temporarily stored to enable quick access to them instead of having to be recalculated each time.

[0144] The method according to the invention may include:

[0145] Generating a hash value based on at least one output image and the control instruction, and, preferably

[0146] Storing the generated hash value in a memory location.

[0147] A hash value is a unique code generated from data, such as the source image and the control instruction, using a special mathematical function (the hash algorithm). This value represents the data in a compact form and is often used for quick data access or to verify data integrity. An example hash value might consist of multiple hexadecimal characters and take the form "3d180ab86e". A hash value is unique for a specific combination of source image and control instruction; that is, changing a single pixel in the source image and / or a single element, such as a letter or punctuation mark, in the control instruction will result in a different hash value.

[0148] The method according to the invention may include:

[0149] Comparing a hash value associated with at least one output image and a control instruction with at least one hash value in memory, and, in the event of a match, in particular directly, outputting, especially from memory, a test completion image of the user interface under test, in particular the GUI, which is linked to the hash value in memory. It should be clear that the hash value in memory also corresponds to a combination of at least one output image and a control instruction and can accordingly be referred to as a stored hash value or reference hash value.

[0150] The method according to the invention can therefore utilize a caching algorithm to improve efficiency by generating a hash value based on an output image and a control instruction. This hash value can be stored in memory. If the same combination of output image and control instruction, i.e., the same hash value, occurs again later, the system can quickly retrieve the stored hash value, perform a comparison based on it, and immediately skip some steps of the method according to the invention. This eliminates the need for reprocessing or repeating some process steps, thus improving efficiency, speed, and performance.

[0151] "Direct" output can be understood as the output of the test result for the user interface under test, particularly the GUI, immediately, directly, and / or immediately following the comparison. Direct output can therefore occur without executing the steps of identification, assignment, processing, and creation (and any associated sub-steps). Caching does not necessarily have to occur before identification; it can, for example, also occur before assigning the at least one label. However, it is preferred that the comparison and / or, in particular, the direct output is performed before the identification of the at least one object.

[0152] The method according to the invention may include:

[0153] Generating an output based on an evaluation of a stored and / or a generated test completion image, and at least one assertion that is associated with the interaction with the at least one identified object.

[0154] Evaluation can be understood as verification that an action or interaction with at least one object has taken place. An assertion can be a result, particularly an expected one, such as an instruction or input that is indicative of at least one interaction with the at least one identified object. If an interaction with a user interface element, e.g., a UL element or GUI element, such as clicking the green button, triggers a specific action or interaction, the assertion can verify whether the expected result has occurred. For example, after clicking the green button, an assertion can appear, e.g., in the form of a message, such as a confirmation message, indicating "Subject clicked the button." This can be passed to the language model, especially the LLM or the multimodal LLM, along with the stored and / or generated test completion screen, e.g.,for processing / analyzing.

[0155] The output, which is generated, for example, by processing, can indicate that the user interface under test, especially the GUI, functions essentially without errors, and / or includes the value True. The output can indicate that the user interface under test, especially the GUI, has at least one error and / or is at least partially faulty, and / or includes the value False. The output can indicate that the user interface under test, especially the GUI, needs to be updated, at least partially, and / or includes the value Refresh.

[0156] In this step, an additional control instruction, in particular a prompt-based instruction, can be used. The following is again only an example and not a limiting one. As an example, it is again assumed that a green button in a user interface or GUI is to be clicked. The control instruction can, for example, comprise a first part, which can be called a system prompt, and a second part, which can be called a user prompt. However, this two-part structure is optional. The user prompt can comprise a task (or several tasks or a list of tasks).

[0157] System prompt

[0158] You are an AI agent with capacity to see the user's screen. The user can ask you to verify assumptions.

[0159] You will see the user's screen, then you will first think in high level what needs to be true so that the assumption is valid. Then you will decide whether the assumption is true or false. You are always very short and concise in your writing.

[0160] User-Prompt

[0161] Task: I should see the text "Subject clicked the button"

[0162] Here is a screenshot of the screen, please do two things:

[0163] A. High level list of steps to follow, using a numbered list in English text

[0164] B. Result of the assertion, being one of:

[0165] - [REFRESH] (if you want to make an assertion about an element, that is not or will not be on the screen)

[0166] - [TRUE] (if the assumption made by the user is true)

[0167] - [FALSE] (if the assumption made by the user is false)

[0168] Those are the ONLY options you have, work with that. Do NOT give booleans you consider optional.

[0169] Your response should be divided up into sections A and B like the prompt.

[0170] The output can then include, for example, one of: [TRUE], [FALSE], [REFRESH]

[0171] As an optional step, the procedure can include passing the output to the user interface, or GUI, or the test suite, e.g., Jira Xray, but this is not mandatory.

[0172] BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Exemplary embodiments of the invention are now described with reference to the accompanying drawings. To ensure a detailed understanding of the features of the present disclosure mentioned above, a more detailed description of the disclosure, which was briefly summarized above, can be obtained by referring to exemplary embodiments. The accompanying drawings relate to embodiments of the disclosure and are described below: Fig. 1 shows at least one initial image of a user interface to be tested, in particular a GUI, in the form of a computer screen containing at least one object, e.g., a button;

[0174] Fig. 2 shows the identification or localization of at least one object in the initial image of Fig. 1 using a bounding box;

[0175] Fig. 3 shows several bounding boxes in the initial image of Fig. 1;

[0176] Fig. 4 shows the assignment of at least one label to the at least one identified or localized object from Fig. 2 to generate a labeled image;

[0177] Fig. 5 shows a flowchart of the steps of processing and generating the trigger instruction according to the method according to the invention; and

[0178] Fig. 6 shows a flowchart of the steps for generating the marked or labeled image based on the initial image.

[0179] DESCRIPTION OF PREFERRED EXECUTION FORMS

[0180] The invention will now be explained in more detail with reference to embodiments shown in the drawings, wherein in all drawings essentially functionally identical elements have the same reference numerals.

[0181] The drawings are schematic and not to scale. Some elements in the drawings may have exaggerated dimensions to emphasize aspects of the present disclosure and / or for greater clarity of presentation. For the sake of simplicity, identical reference numerals are used to identify identical elements that are common to all drawings. It is intended that elements and features of one embodiment may be advantageously incorporated into other embodiments without further mention. In general, only the differences between individual embodiments are described.

[0182] Each embodiment serves to illustrate the disclosure and should not be understood as limiting the disclosure. Furthermore, features presented or described as part of one embodiment may be used in conjunction with other embodiments to create a further embodiment. It is intended that the description includes such modifications and variations.

[0183] It should be noted that the following figures refer, by way of example, to a "green button" in a user interface. This is merely exemplary and optional, intended only to illustrate the present invention. Numerous examples and applications are conceivable, and the present invention is not limited to "green buttons."

[0184] Fig. 1 shows at least one initial image 100 of a user interface 110 to be tested, in particular a graphical user interface (GUI) 110, e.g., a computer screen 110. The initial image 100 is shown in Fig. 1, for example, as a screenshot 100, i.e., a particularly static image of the current state of a user interface 110 or a visual representation of the state of the user interface (Ul 110) or the GUI 110.

[0185] The user interface 110 shown in Fig. 1 can be an interface or part of a technical system with which a user can interact. The user interface 110 can include means by which the user can input information and / or receive information from the system. As can be seen in Fig. 1, the exemplary user interface 110 can include a graphical representation of interactive elements 130 such as menus, buttons, text fields, input fields, control / input elements, icons, tabs, text areas, etc., some of which are highlighted in Fig. 1 and allow interaction with them. The user interface in Fig. 1 is represented as a "Graphical User Interface" (GUI), which uses visual indicators and / or graphical elements to facilitate interaction with the software.

[0186] Within the user interface 110 to be tested, there is an element or at least one object 120, the function or functionality of which is to be checked or tested. The at least one object is shown in Fig. 1 as a button, for example as a green button, although other types would also be conceivable.

[0187] Fig. 2 shows the identification or localization of the at least one object in the initial image 100 of Fig. 1 using at least one bounding box 140. In a non-restrictive example, identification can be achieved using a sliding window within the initial image 100. The identification described herein can optionally also include text recognition of the initial image 100, in particular optical character recognition (OCR), to recognize text on / in the initial image 100 and / or the initial object 120. For example, the text of the button 120 can be recognized as "Button" and / or assigned to or linked with the object 120. This information can also contribute to or be used for the classification of the initial object 120.

[0188] The bounding box 140 is shown in Fig. 2 as a substantially rectangular frame used to enclose, contain, and / or define the object 120, for example, as tightly as possible. This bounding box 140 can be used to identify the position, arrangement, and / or size of the object 120 within the user interface 110. For example, the bounding box 140 can be drawn around the button 120 of the user interface 140 to mark / label it for further analysis or processing.

[0189] As can be seen in Fig. 3, it is possible that more than one bounding box, in particular several or a plurality of bounding boxes 140A-C, is used or assigned in the initial image 100. For example, a first bounding box 140A and a second bounding box 140B can be assigned to button 120. In this case, the procedure can include:

[0190] Adjusting and / or setting the majority of bounding boxes, i.e., at least the first bounding box 140A and the second bounding box 140B in the initial image 100, for example by merging them. As a result of merging the first bounding box 140A and the second bounding box 140B from Fig. 3, the bounding box 140 in Fig. 2 can be obtained, for example. Since new techniques are becoming increasingly precise, inaccurate assignments are less frequent, and adjustment / setting may become obsolete in the future.

[0191] Fig. 4 shows the assignment of at least one label 150 or 150A-C to the at least one identified or localized object 120 or the bounding boxes 140 or 140A-C from Fig. 2 or Fig. 3 to generate a labeled image 160. The labeled image 160 can, for example, essentially correspond to the original image 100, except that the labeled image 160 has at least one label 150 or 150A-C. As can be seen in Fig. 4 (e.g., in comparison to Figs. 2 and 3), the bounding box 140A can be assigned the label 150A, with the exemplary value "A7", and the bounding box 140B can be assigned the label 150B, with the exemplary value "A5". Furthermore, the label 150 can be assigned to the at least one identified or localized object 120 or the bounding box 140 from Fig. 2 with the exemplary value "A10". It should be noted that Fig. 4 merely illustrates the two examples from Fig. 2 and Fig. 3, i.e.It is sufficient if at least one bounding box, e.g., bounding box 140 from Fig. 2, and at least one corresponding label, e.g., label 150, are created. For the sake of completeness, it should be mentioned that bounding box 140C can be assigned label 150C.

[0192] Fig. 5 shows a flowchart of a method 200 according to the invention, in which the marked image 160 (optionally in combination with the initial image 100) of Figs. 1-4 can be used. Fig. 6 then shows how the marked image 160 is obtained from the initial image 100, which is indicated by the dashed arrow 230 or the method step 230.

[0193] The method according to the invention 200 can comprise:

[0194] Processing 210 of the marked image 160 (and possibly also of the original image 100) using a control instruction 170 and, based on the processing, generating 220 at least one trigger instruction 180, which is set up to initiate at least one interaction with the at least one identified object 120 when executed, in particular for carrying out the software test.

[0195] The processing 210 of the marked image 160 (and optionally the initial image 100; as indicated by the dashed line) using the control instruction 170 and / or the generation 220 of the at least one trigger instruction 180 can be carried out using a language model 190, in particular a Large Language Model (LLM) 190, preferably using a multimodal language model 190.

[0196] For example, during processing 210, it can be recognized that at least one object 120 is a button, e.g., a green button 120 (see Fig. 1-4), and which label is assigned to this button. The language model 190 can therefore recognize an assignment of the label 150, e.g., "A10" (see Fig. 4), to the green button 120.

[0197] The control instruction 170 can be a prompt-based instruction, in particular a prompt, i.e., for example, a text command or an input prompt, especially a text-based one, which can influence the processing 210 by the language model 190. A non-restrictive example of a control instruction 170 is given below. It should be clear that the individual statements / elements mentioned in the prompt are not to be understood as restrictive; that is, what follows is merely optional and refers to the "green button" example for illustrative purposes. For example, two screenshots are mentioned in the prompt; however, the invention is not limited to two screenshots, i.e., these are by no means to be considered essential. The same applies to the interaction types, labels, etc. As an example, it is assumed below that a green button in a user interface or GUI is to be clicked.

[0198] Control instruction 170, for example, can comprise a first part, which can be called the system prompt, and a second part, which can be called the user prompt, although the two-part structure is optional, and the user prompt can additionally include a task (or several tasks or a list of tasks):

[0199] System prompt

[0200] You are an AI agent with capacity to see the user's screen, click buttons, type text and scroll. The user can ask you to carry out tasks:

[0201] You will see the user's screen, then you will first think in high level what are the steps you will need to follow to carry out the task. Then, you will be given annotated screenshots with codes mapping buttons, text, input fields and so on. You can choose to click on buttons and text, type in input fields, scroll up or down, and get a refreshed update of the screen to continue until the task is completed. In order to type in an input field, you MUST first click on it. Never attempt to submit a form unless you are told to do so.

[0202] You are always very short and concise in your writing.

[0203] User-Prompt

[0204] Task: I click on the green button Here are two screenshots of the screen. One is tagged with labels like A1, B4, A17 on each interactive item, please do two things:

[0205] A. High level list of steps to follow, using a numbered list in English text

[0206] B. A list of actions to execute, each action being one of:

[0207] - [CLICK A 1] (To click the A 1 button for example)

[0208] - [TYPE "message"] (To type "message")

[0209] - [SCROLL <direction> <speed>] (To scroll up ordown, at normal or quick speed. For example [SCROLL down quick])

[0210] Those are the ONLY options you have, work with that. Do NOT give actions you consider optional. Make sure the order of actions follows the order of the high-level list. If you want to tick a checkbox or select an option, ONLY use labels that start with the letter A, such as A1 or A7. If you want to click or type on an element, that is not or will not be on the screen, return a list that consists only of a [REFRESH]. If the user asked you to scroll until an element is visible, but the element is already visible, return a list that consists only of a [TRUE]. Do not scroll unless the user specified so.

[0211] Your response should be divided up in sections A and B like the prompt.

[0212] Control instruction 170 can be configured to enable and / or trigger the assignment of at least one label 150 to at least one identified object 120. Control instruction 170 can also enable the identification of which object 120 and / or which corresponding label 150 is meant. This can trigger a response, such as finding or locating the green button in a user interface, and at least partially providing or assigning a corresponding interaction.

[0213] As can be seen, control instruction 170 can include a plurality of interaction types, e.g., a list of interaction types, such as clicking, typing, entering, scrolling, or the like. Control instruction 170 can be configured to trigger an output of at least one interaction type from among the plurality of interaction types. Control instruction 170 can thus ensure that, for example, during processing 210, an interaction type is selected that is best suited for the at least one object 120. This at least one interaction type can either be output directly, e.g., as output, or can be integrated into the trigger instruction 180. This trigger instruction 180 can therefore also be referred to as a result, or rather, it can include the at least one interaction type as a result.The trigger instruction 180 can be indicative of at least one interaction type with the at least one identified object 120 and indicative of the at least one label 150 assigned to this, i.e., the at least one identified object 120. An optional classification of the at least one identified object 120 can also be included. In the example of Fig. 1-5, the trigger instruction 180 could therefore include the interaction type "click" (CLICK) as well as the label 150, which in Fig. 4 has been assigned the value "A10". Optionally, a classification of the at least one object 120 as a button or, if applicable, a green button could be included. An example trigger instruction could therefore look like this:

[0214] [CLICK A10]

[0215] *where in this example "CLICK" represents the interaction type "click" and "A10" can correspond to at least one label, which, for example, has been assigned to at least one object "green button". The number "10" can, for example, refer to the presence of other, e.g., further labels (see Fig. 4).

[0216] As can be seen in Fig. 5, the source image 100 can be used for processing 210, in particular additionally. The method 200 can therefore comprise: processing 210 of the marked image 160 and the at least one source image 100 using a control instruction 170. This can make it possible to take into account image content or image components that are, for example, obscured by the at least one label 150 or 150A-C, so that this image content or image components are not lost.

[0217] Fig. 6 shows process step 230 of the inventive method 200 from Fig. 5, in particular how the marked image 160 can be generated 270 based on the initial image 100 (cf. Figs. 1-4). The inventive method 200 can comprise:

[0218] Identifying, in particular locating, at least one object 120 using at least one bounding box 140 or 140A-C, based on at least one initial image 100 showing a user interface 110 to be tested, in particular a graphical user interface (GUI) 110, e.g., a computer screen (see Fig. 1-4), and assigning at least one label 150 or 150A-C to the at least one identified object 120 to generate 270 at least one labeled image 160. The identification and / or assignment can be performed using a neural network 195, for example, using a sliding window (not shown, also optional). The neural network 195 can be an object recognition model based on a transformer architecture, wherein, in particular, the neural network is configured for real-time object recognition and / or end-to-end recognition.A suitable, yet non-restrictive, neural network could be RT-DETR (Real-Time Detection Transformer), which can correspond to a transformer-based model for object recognition. The object recognition model 195 can thus quickly and accurately detect at least one object 120 in the output image 100, for example, via at least one bounding box 140 or 140A-C, and assign at least one label 150 or 150A-C to this object. The at least one output image can be provided to the object recognition model 195 as input 240. The procedure can therefore include: providing or passing 240 the at least one output image 100 to an object recognition model 195 as input.

[0219] The neural network 195 can be pre-trained using, in particular open-source, image data 196 that is indicative of one or more different user interfaces, especially GUIs, wherein the image data 196 is publicly available and / or accessible. For example, 250 source images, e.g., screenshots, for example as JPEGs, can be used for training or pre-training, showing several, e.g., different user interfaces, especially GUIs.

[0220] Procedure 200 or step 230 may include:

[0221] Creating 260 of at least one initial image 100, in particular by means of a screenshot, of the user interface 110 to be tested, in particular the GUI 110, wherein the at least one initial image 110 is created at the beginning of each software test.

[0222] Although the foregoing is directed to embodiments of the disclosure, other and further embodiments of the disclosure can be developed without deviation from its basic scope, the scope being defined by the following claims. List of reference numerals

[0223] 100 Initial image

[0224] 110 User interface

[0225] 120 objects

[0226] 130 Interactive Element

[0227] 140 Bounding Box

[0228] 140A-C Bounding Box

[0229] 150 labels

[0230] 150A-C Label

[0231] 160 Labeled Image

[0232] 170 Tax Instruction

[0233] 180 Trigger instructions

[0234] 190 Language Model

[0235] 195 Object Recognition Model

[0236] 196 training data

[0237] 200 procedures

[0238] 210 Processing

[0239] 220 Produce

[0240] 230 Arrow / Procedure Step

[0241] 240 Input Provision

[0242] 250 Training

[0243] 260 Creating the initial image

[0244] 270 Create marked image< / speed> < / direction> < / speed> < / direction>

Claims

1. PATENT CLAIMS 1. Computer-implemented method for automating software testing, in particular software module testing, integration testing and / or software end-to-end (E2E) testing, comprising: Identifying, in particular locating, at least one object using at least one bounding box, based on at least one initial image showing a user interface to be tested, in particular a graphical user interface (GUI), Assigning at least one label to the at least one identified object to generate at least one tagged image, and Processing the selected image using a control instruction, and, based on the processing, Generating at least one trigger instruction that is configured to initiate at least one interaction with the at least one identified object upon execution, in particular for the purpose of performing the software test.

2. Computer-implemented method according to claim 1, further comprising: Initiating at least one interaction with the at least one identified object, in particular for the purpose of carrying out the software test.

3. Computer-implemented method according to any of the preceding claims, wherein the identification further comprises: Assigning at least one, in particular multidimensional, number vector to the at least one bounding box, and / or Classifying at least one object in at least one source image.

4. Computer-implemented method according to any of the preceding claims, wherein the identification further comprises: Creating at least one initial image, in particular by means of a screenshot, of the user interface to be tested, in particular the GUI, wherein the at least one initial image is created at the beginning of each software test.

5. Computer-implemented method according to any of the preceding claims, wherein the identification and / or assignment is performed by means of a neural network, wherein the neural network is an object recognition model based on a transformer architecture. based, in particular the neural network is set up for real-time object recognition and / or end-to-end recognition.

6. Computer-implemented method according to claim 5, wherein the neural network is pre-trained using, in particular open-source, image data that is indicative of one or more different user interfaces, in particular GUIs, wherein the image data is publicly available and / or accessible.

7. Computer-implemented method according to one of the preceding claims, wherein, for processing, in particular additionally, at least one initial image is used.

8. Computer-implemented method according to one of the preceding claims, wherein the at least one trigger instruction is indicative for - at least one type of interaction with the at least one identified object and - at least one label assigned to it, the procedure including in particular: Issue at least one trigger instruction.

9. Computer-implemented method according to any one of the preceding claims, further comprising: Creating at least one test completion image of the user interface to be tested, in particular the GUI, after initiating at least one interaction or after completion of the interaction, and if necessary. Storing, in particular saving, the test completion image in a memory location.

10. Computer-implemented method according to one of the preceding claims, wherein the processing and / or generation is carried out using a language model, in particular a Large Language Model (LLM), preferably using a multimodal language model.

11. Computer-implemented method according to one of the preceding claims, wherein the output image corresponds to a substantially unlabeled image, in particular a label-free output image.

12. Computer-implemented method according to one of the preceding claims, wherein the at least one bounding box is a, in particular multidimensional, number vector includes, which is indicative of a position of at least one bounding box and / or at least one identified object in the original image.

13. Computer-implemented method according to one of the preceding claims, wherein the at least one object is an input element, a setting element and / or a control element of the software environment to be tested, in particular the graphical user interface (GUI).

14. Computer-implemented method according to any of the preceding claims, wherein the software tests comprise one or more of: functional tests to verify at least one function of the at least one object, and / or Layout tests for the visual verification of one or more elements of the user interface under test, in particular the graphical user interface (GUI).

15. Computer-implemented method according to any of the preceding claims, wherein the control instruction is a prompt-based instruction, in particular a prompt.

16. Computer-implemented method according to one of the preceding claims, wherein the control instruction is configured to enable and / or trigger an assignment of the at least one label to the at least one identified object.

17. Computer-implemented method according to one of the preceding claims, wherein the control instruction comprises a plurality of interaction types, wherein the control instruction is configured to elicit an output of at least one interaction type from the plurality of interaction types.

18. Computer-implemented method according to any one of the preceding claims, further comprising: Generating a hash value based on at least one output image and the control instruction, and, preferably Storing the generated hash value in a memory location.

19. Computer-implemented method according to any one of the preceding claims, further comprising: Comparing a hash value associated with at least one output image and the control instruction with at least one hash value in a memory, and, in the event of a match, in particular a direct one, Output, in particular from memory, a test completion image of the user interface under test, in particular the GUI, which is linked to the hash value in memory.

20. Computer-implemented method according to claim 19, wherein the output, in particular direct output, is performed without performing the steps of identifying, assigning, processing and generating.

21. Computer-implemented method according to one of claims 19-20, wherein the comparison and / or the, in particular direct, output is carried out prior to the identification of the at least one object.

22. Computer-implemented method according to any one of the preceding claims, further comprising: Generating an output based on an evaluation of a stored and / or a generated test completion image, and at least one assertion that is associated with the interaction with the at least one identified object.

23. Computer-implemented method according to claim 22, wherein the output is either - This is indicative if the user interface to be tested, especially the GUI, functions essentially without errors, and / or includes the value True, - is indicative of this if the user interface to be tested, in particular the GUI, has at least one error and / or is at least partially faulty, and / or includes the value False, or - This is indicative of the fact that the user interface to be tested, especially the GUI, must be at least partially updated and / or includes the value Refresh.

24. Computer program product comprising instructions which, when the program is executed by a computer, cause it to execute the method according to any one of claims 1-23.

25. Computer-readable storage medium comprising instructions which, when executed by a computer, cause it to execute the method according to any one of claims 1-23.

Images