Test assembly and method for testing a test object, and computer program for carrying out the method

EP4470162A1Pending Publication Date: 2024-12-04FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
EP2023702250
Authority / Receiving Office
EP · EP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-01-27
Filing Date
2023-01-24
Publication Date
2024-12-04

AI Technical Summary

Technical Problem

Technical devices, such as industrial automation components and IoT devices, are rarely fully tested for vulnerabilities due to complexity and the use of components from different suppliers, leading to inadequate security testing, which is often manual and resource-intensive, failing to account for internal functions and resulting in missed vulnerabilities.

Method used

A test arrangement that creates a model of the technical device through interaction, generates test cases based on this model, and continuously updates it using real test results to automate the testing process, enabling more efficient and targeted vulnerability detection.

Benefits of technology

This approach allows for comprehensive and efficient testing of technical devices, identifying vulnerabilities and improving test automation components, reducing manual effort and enhancing resilience to attacks by continuously updating the model and generating new test cases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 1.1
    Figure 1.1
Patent Text Reader

Abstract

Embodiments of the invention relate to a test assembly (101) for testing a test object (104, 410), for example for testing a technical device or an industrial device or a component of a device or a component having a communication interface, for example a wired or wireless network interface and / or a web interface. The test assembly (101) is designed to provide a model (107) on the basis of an interaction with the test object (104) in order to generate one or more test cases (102, 420, 440) using the model (107) and in order to improve the model (107) of the test object (104, 410) on the basis of real test results (109) based on the test cases (102, 420, 440) that have been generated.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] A test arrangement and a method for testing a test object, and a computer program for carrying out the method

[0002] Description

[0003] Technical area

[0004] The invention relates to a test setup for testing a test object. The invention can be used, for example, to examine test objects or technical devices for vulnerabilities. These include, among others, industrial automation components, Internet of Things devices (IoT devices, lloT devices), and medical devices. The invention relates to model-based security testing for technical devices.

[0005] Technical problem

[0006] Technical devices perform tasks. Attacks can prevent or hinder technical devices from performing their tasks. These attacks, such as virtual attacks or attacks via the network / internet, exploit vulnerabilities in the technical device. To prevent these attacks, the vulnerabilities can be discovered and remediated. Security tests are used to discover vulnerabilities. A test environment (TE) automatically tests the technical device as a device under test or subject under test (SUT).

[0007] Technical devices are rarely fully tested for vulnerabilities. This is due, on the one hand, to the complexity of the technical devices themselves and, on the other, to the fact that technical devices are composed of various components. These components can come from different suppliers and are not always visible in all details.

[0008] If the technical devices are tested at all, this is done based on experience from previous tests. These tests are conducted either manually by humans, for example by so-called penetration testers, or using test environments.

[0009] Manual testing requires a high level of human effort, especially if the tests are to be performed regularly. For this reason, in practice, they are rarely performed continuously on all devices under test or subjects under test (SUT).

[0010] Test cases for automated tests can, for example, be generated based on experience from previous tests and grammars. However, these are static except for random input data and, in particular, do not take into account the internal workings of the SLIT. It was recognized that SLIT failures are only presented as results of the investigation and are not incorporated into the creation of new, specialized test cases.

[0011] It was determined that the internal functioning of the SUT can be described using a model. This model includes all functionalities of the SUT, not just the parts required to perform the SUT's tasks. It can therefore be used to construct test cases.

[0012] Previous approaches to model-based testing of technical devices require a complete model of the SUT. This model can, for example, be derived from the design documents. For the reasons mentioned above, this approach is not readily feasible for composite technical devices. Examples of these approaches are the work of Bringmann et al. [1] and the work of Bauer et al. [2].

[0013] Manually constructing the model is very time-consuming and often impossible. Manually constructing a model also requires a significant amount of personnel, and it is rarely possible to provide a complete model. This is especially the case when individual components of the system have been purchased from third parties.

[0014] Approaches to a test environment for the automated testing of technical devices have been documented in various patents. Some embodiments relate to the general generation of test cases based on a grammar and to the monitoring of a square wave as the output of the component under test. This makes it possible to check whether the device under test has entered an error state or whether a general state transition has occurred.

[0015] US 8,132,053 B2 and US 8,006,136 B2 relate to the testing of a system with software and / or hardware components. The exemplary embodiments present a test environment or test framework that generates test cases for a system under test using a grammar. The test framework tests the system with the test case and receives a response to the test case from the system under test. The inputs and outputs are then compared with the expected results to determine whether the system under test is functioning correctly.

[0016] US 8,433,542 B2 and US 9,026,394 B2 also relate to testing a system with software and / or hardware components. The test framework of the aforementioned patents generates test cases for a system under test using a grammar. The test framework tests the system with the test case and receives a response to the test case from the system under test. The inputs and outputs are then compared with the expected results. The data can then be interpreted in the grammar system and / or used as input for a fault isolation system. This can be used to check whether the device under test has entered a faulty state or whether a general state transition has occurred.

[0017] The various components, such as test case generation and / or monitoring, are also summarized, for example, in the joint patent US 9,400,725 B2. US 9,400,725 B2 generally relates to automated testing of a system with software and / or hardware components. The test framework generates test cases for a system under test using a grammar. The test framework tests the system with the test case and receives a response to the test case from the system under test. The inputs and outputs are then compared with the expected results to determine whether the system under test is functioning correctly. In particular, the system under test can be analyzed to determine whether it is capable of properly processing control instructions and input signals and / or generating expected output control signals and additional control / feedback information.The data can then be interpreted in the grammar system and / or used as input for a fault isolation system to detect anomalies in the system under test. The objective of the present invention is to create a concept for testing test objects that provides an improved compromise between test effort and test scope or test reliability.

[0018] This object is achieved by a test arrangement or test environment (TE) according to claim 1, by a method according to claim 43, or by a computer program according to claim 44.

[0019] Summary of the invention

[0020] Embodiments of the invention provide a test arrangement for testing a test object, such as for testing a technical device or an industrial device or a component of a device or a component with a communication interface, for example with a wired or wireless network interface and / or with a web interface.

[0021] The test setup is designed to create a model based on an interaction with the test object, to generate one or more test cases using the model, and to improve the model of the test object using real test results based on the generated test cases.

[0022] The test suite (TE), for example, actively and automatically creates a model of the SUT based on its interaction with the test object or test unit (SUT). The model is then used by the TE, for example, to specifically generate new test cases. The generated test cases or tests are then executed by the TE against the SUT. During and after the test, the TE checks whether the SUT fulfills or can fulfill its task. To do this, it monitors the SUT, checks its outputs, and also uses the SUT model. The SUT model is continuously updated during the tests. A special technical feature, for example, is that the TE automatically generates a model from the interaction with an opaque SUT and uses this model automatically for adaptive test case generation.

[0023] For example, the TE creates an initial model of the SUT using interactions with the SUT. Based on the initial model, the TE creates test cases. The TE executes the test cases against the SUT to generate real-world test results. The model is improved based on the test results. New test cases are continuously created based on the new model, and the SLIT model is continuously updated.

[0024] The effect of automated derivation of a SLIT model, for example, is that knowledge about the SUT is derived that was not previously explicitly available. With this additional knowledge, the TE can, for example, generate new test cases more efficiently and purposefully. This allows the entire SUT test to be carried out more efficiently and purposefully. Furthermore, this knowledge can be used, for example, to determine whether the SUT's behavior exhibits an anomaly. This can be an indication of a vulnerability in the SUT.

[0025] The most important advantage, for example, is that testing of automation components can be carried out more efficiently and specifically thanks to the automatically generated model. Thus, the test setup creates more and more test cases that test the test object as comprehensively as possible for vulnerabilities.

[0026] In embodiments, the test arrangement is designed to receive one, several or no responses given to a generated test case from a test object that is unknown and / or non-transparent with regard to its behavior via one or more communication paths.

[0027] The test setup is designed to detect all possible responses from the test object. By receiving as many responses from the test object as possible, the model of the test object becomes more accurate based on the test results, even though the test object's behavior is unknown or non-transparent.

[0028] In embodiments, the test arrangement is designed to check the functionality of the test object or the functionality of one or more functions of the test object by means of one or more monitors and / or monitoring modules.

[0029] The test object can respond to a test case or a request with a change in behavior in addition to or as an alternative to a response. The test setup can monitor this change in behavior, for example, using one or more monitors and / or monitoring modules. In embodiments, the test setup is designed to evaluate the received responses and / or the results of the monitors and to generate test results based on this evaluation.

[0030] For example, all behavioral changes and / or responses of the test object in response to a test case are summarized under one test result. This can simplify subsequent processing of the test results.

[0031] In embodiments, the test arrangement is designed to evaluate the quality of the test case based on the test result and / or on a probability of a malfunction of the test object tested with the test case.

[0032] The test setup simulates, for example, virtual attacks against the technical device or the test object in order to discover vulnerabilities in the device. The more vulnerabilities a test case in the test setup discovers, or the higher the probability that a test case will discover a vulnerability, the better the test case is, or the better the test case is evaluated. The quality of a test case can be measured, for example, based on the number of unexpected error messages and / or malfunctions. Based on the evaluation, test cases (e.g. for an actual application on the test object) can be selected (e.g. preferentially) that are most likely to uncover vulnerabilities in the test object, thereby improving test efficiency.

[0033] In embodiments, the test arrangement is designed to predict or estimate or, for example, approximate the test results of the test cases based on the model.

[0034] The test suite can use the model to predict the test results of a test case and thus the test case's quality without executing the test case against the SUT. This can save testing time, for example, by appropriately selecting the test cases actually executed.

[0035] In embodiments, the test arrangement is designed to selectively select the test cases in which a malfunction of the test object or a malfunction of one or more functions of the test object was predicted, and to test the test object with the selected test cases in order to generate test cases that generate as many malfunctions of the test object as possible as a real test result.

[0036] The test suite can use the model to predict the test results of a test case and thus the test case's quality. This prevents the test suite from executing all test cases against the SUT. Reducing the number of test cases executed against the SUT saves resources, such as time.

[0037] In embodiments, the test arrangement is designed to determine, using the model, whether a real test result is judged to be anomalous.

[0038] The test suite can use the model to predict the test results of a test case. The test results of the executed test cases are compared with the expected results. If there is a discrepancy, the test result is considered anomalous. Anomalous test results indicate vulnerabilities. Thus, test results considered anomalous can be selectively or preferentially reported or output (e.g., flagged) to a user of the test suite.

[0039] In embodiments, the test arrangement is designed to judge the real test result as abnormal if the real test result indicates a malfunction of the test object or a malfunction of one or more functions of the test object, or if the real test result deviates from the test result predicted by the model.

[0040] The test setup can use the model to predict the test results of a test case. The test results of the executed test cases are compared with the expected results. If there is a discrepancy between an actual test result and an expected test result, or if the test object malfunctions, the test result is considered anomalous. Anomalous test results indicate weaknesses in the model and / or the test object and are particularly valuable, for example, in error analysis.

[0041] In embodiments, the model of the SLIT includes a state machine.

[0042] Working with a state machine is easy because the number of states in a state machine is often relatively small or within a manageable range, the states are independent of each other, and the transition between states follows a logical rule. For example, it is assumed that the test object remains in a state until a state-changing request is executed.

[0043] In embodiments, a state of the state machine comprises at least one test case that leads to the state and at least one resulting test result. In embodiments, the test arrangement is designed, for example, to calculate a value for a state of the test object using a hash function and to identify the state with the calculated value.

[0044] For example, the test setup is designed to add a new state to the state machine that forms the model of the test object if this state does not yet exist.

[0045] A hash function is an injective mapping that maps, for example, an input of varying lengths to a target set or hash value of a fixed length. The length of the hash value remains fixed, regardless of the length of the input values ​​and / or the length of the test results. A hash value is well-suited for use as an identification number, identifying, for example, a state in a space-efficient yet typically unique form.

[0046] In embodiments, the test arrangement is designed to bring the test object into a predetermined state and to test the test object in the predetermined state for one or more different test cases.

[0047] For example, the test setup is designed to provoke and / or identify one or more state changes starting from the specified state. Thus, the SUT model is supplemented with the newly identified state changes and / or the newly identified states.

[0048] In embodiments, the test arrangement is designed to identify state-changing test cases that lead to a change in the state of the test object and to update the model based thereon.

[0049] By continuously updating the model, the test setup creates a positive feedback loop. An improved model increases the number of state-changing test cases. An increase in the number of state-changing test cases leads to a better SUT model.

[0050] In embodiments, the test arrangement is configured to identify the state-changing test cases using a heuristic. The heuristic is configured to consider a frequency of execution of a test case and / or a frequency of identification of a test case as a state-changing test case and / or a category or type of a test case. Using a heuristic, the test arrangement can efficiently identify new state-changing test cases. The statistics about and / or experience with previous test cases can lead to the discovery or identification of new state-changing test cases.

[0051] In embodiments, the test arrangement is designed to assess whether a state has been sufficiently explored. The test arrangement is designed to generate new test cases that induce this state if the state has not been sufficiently explored, or to generate new test cases with a significant difference from the already known test cases if the state has been sufficiently explored. In this case, the difference is preferably large enough, for example, to provoke and / or identify one or more state changes.

[0052] This allows the test suite to systematically explore each state sufficiently. Once a state has been tested with enough test cases, the test suite will, for example, provoke a state change and generate new test cases for the new state.

[0053] In embodiments, the test arrangement is designed to model a decision as to whether known states should be further investigated or whether new states should be discovered as a multi-armed bandit problem, and to solve the multi-armed bandit problem using a reinforcement learning method, such as the upper confidence bound (UCB) method, to make the decision.

[0054] It was recognized that a multi-armed bandit problem can well describe the dilemma between further investigation of known states and the search for new states. One advantage of the multi-armed bandit problem is that it can be mathematically solved using various methods.

[0055] In embodiments, the test arrangement is designed to repeatedly test the test object with a test case in order to determine whether the test result of the test case changes over time and to identify a state change based thereon.

[0056] Thus, the test setup can test whether the states are truly independent of each other and / or whether the states are time-dependent. In embodiments, the test setup is designed to identify test cases with which the test object was tested between the test case and the repeated test case as potentially state-changing test cases.

[0057] For example, the test suite assumes that the states are interdependent and designates all executed test cases between the test case and the repeated test case as potentially state-changing test cases. The potentially state-changing test cases can be further investigated, for example, by the test suite.

[0058] In embodiments, the test arrangement is designed to generate invalid and / or unexpected and / or random inputs when generating the test cases.

[0059] The test setup is thus designed, for example, to discover vulnerabilities in the test object using generated test cases. The test setup is successful, for example, if it finds test cases and / or inputs that the test object's designer or constructor had not considered, such as invalid and / or unexpected and / or random inputs.

[0060] In embodiments, the test setup is designed to use a mutation algorithm when generating test cases, which generates new test cases based on old test cases. Thus, test cases can be developed successively or step by step, for example, by changing only part of a test case during a mutation.

[0061] In embodiments, the test arrangement is designed to use one or more machine learning algorithms or “machine learning algorithms” when the test cases are mutated.

[0062] In embodiments, the test arrangement is designed to generate a plurality of intermediate test cases, e.g., by means of a mutation, based on one or more predefined and / or random test cases. The test arrangement is designed to assign a fitness value to each of the intermediate test cases, to select the intermediate test cases depending on the respective assigned fitness value, and to reuse the selected intermediate test cases as new test cases.

[0063] For example, the intermediate test cases with the highest fitness values ​​or e.g.

[0064] For example, intermediate test cases with a fitness value above a predefined threshold can be selected and reused as new test cases. This allows, for example, test cases to be successively modified (e.g., only partially or in individual values ​​or subsets of values) or further developed, while, for the actual test execution, test cases can be selected that, due to their assigned fitness value, are particularly capable of causing a malfunction of the test object.

[0065] In embodiments, the test arrangement is designed to assign a fitness value to each intermediate test case based on a probability of a malfunction of the test object tested with the intermediate test case.

[0066] The test setup is designed to discover vulnerabilities in the test object using selected intermediate test cases. The intermediate test cases are evaluated based on their (e.g., expected) ability to cause a malfunction of the test object (described, for example, by the fitness value). This allows for the discovery of as many vulnerabilities as possible.

[0067] In embodiments, the test arrangement is designed to use one or more machine learning algorithms when determining the fitness value.

[0068] In embodiments, the test arrangement is designed to use one or more machine learning algorithms, such as support vector machine algorithms, or support vector machine algorithms and / or random tree algorithms, or random tree algorithms and / or neural network algorithms.

[0069] The use of well-known algorithms simplifies the application of the machine learning concept. Well-known algorithms are easy to implement.

[0070] In embodiments, the test arrangement is designed to retrain the machine learning algorithm or the machine learning algorithms based on the test results of the executed test cases.

[0071] A continuously learning algorithm improves itself over time and always remains up-to-date. Improvements to the model can thus be exploited, for example, to generate new test cases and / or to evaluate test results. In embodiments, the test setup is designed to execute a sequence of test cases and use their test results as an observation sequence for training a hidden Markov model, and to estimate or approximate states of a state machine using the hidden Markov model (HMM) based on an observation sequence.

[0072] It was recognized that the HMM is well suited, for example, for modeling states and transitions with different probabilities between the states, for example in a test object with unknown behavior.

[0073] In embodiments, the test arrangement is designed to bring the test object into the individual states of the hidden Markov model and to generate several test cases per state that bring the test object, starting from the current state, into the other states of the hidden Markov model in order to achieve good path coverage of the hidden Markov model.

[0074] Good path coverage helps the test suite understand the internal functioning of the test object and thus generate better test cases. Errors can occur at every state transition, and thus potential points of attack can be identified.

[0075] In embodiments, the test arrangement is designed to calculate the most probable execution path of a test case in the hidden Markov model by a Viterbi algorithm, and to calculate the path coverage of the test cases based thereon.

[0076] Using the Viterbi algorithm, an HMM can calculate the path most likely taken by the model during the execution of a test case. This path approximates the execution path of the SLIT and is used to quantify the behavior of the SLIT, which in turn is used to generate new test cases. Path coverage can also be used, for example, as a criterion or measure for the quality of the test or test case, or for test coverage.

[0077] In some embodiments, the test setup is designed to continually refine the model and / or the generated test cases. The test setup thus creates, for example, a positive feedback loop in which the new test cases improve the model, and new test cases are generated based on the improved model.

[0078] The test setup can test various test objects, such as web applications, network protocols, or network interfaces. These specific examples are explained below.

[0079] In embodiments, the test arrangement is designed to generate one or more test cases for a web application and to receive and / or evaluate the response of the web application.

[0080] The test arrangement can be designed to generate the test case or the test cases in the form of inputs into one or more input fields of the web application and / or activation of one or more buttons of the web application and / or activation of one or more links of the web application.

[0081] The test arrangement can be designed to receive and / or evaluate the response of the web application. The response of the web application can take the form of activating or deactivating one or more input fields of the web application and / or activating or deactivating one or more buttons of the web application and / or activating or deactivating one or more links of the web application and / or adding or removing one or more input fields and / or one or more buttons and / or one or more links, and / or changing one or more texts or numbers.

[0082] In other embodiments, the test arrangement is designed to generate one or more test cases for a test object using a network protocol, and to receive and / or evaluate the response of the test object.

[0083] The test arrangement may be designed to generate the test case or test cases in the form of one or more inputs in a sequence of one or more network packets.

[0084] The test setup can be designed to receive and / or evaluate the test object's response in the form of a sequence of one or more network packets and / or in the form of information regarding code coverage of the input. Code coverage is defined, for example, as the ratio of actually obtained statements in a test compared to the theoretically possible statements, or also as the proportion of software blocks executed (and thus tested).

[0085] The test setup can be designed to build a state machine based on the test cases and code coverage based thereon, to calculate a value for a state of the test object using a hash function, and to identify the state with the calculated value.

[0086] The test setup may be designed to use a sequence of communication between the test setup and the test object as an observation sequence for training a hidden Markov model, and to estimate or approximate the code coverage with the hidden Markov model based on the observation sequence.

[0087] The test setup can be designed to bring the test object into the individual states of the Hidden Markov Model as an initial state and to generate several test cases per state that bring the test object, starting from the current state, into the other states of the Hidden Markov Model in order to achieve good path coverage of the Hidden Markov Model.

[0088] The test setup may be designed to calculate the most probable execution path of a test case in the Hidden Markov Model by a Viterbi algorithm and to calculate the path coverage of the test cases based thereon.

[0089] In further embodiments, the test arrangement is designed to generate test cases in the form of inputs of a network packet or a sequence of network packets for a network interface of a test object, and to evaluate the functionality of the test object or the functionality of one or more functions of the test object by means of monitors.

[0090] Further embodiments according to the present invention provide corresponding methods or computer programs.

[0091] Brief description of the figures Embodiments according to the present invention are explained in more detail below with reference to the accompanying figures.

[0092] Fig. 1 shows a schematic representation of a sequence of an embodiment of a test arrangement;

[0093] Fig. 2 shows a flow diagram of an embodiment of a positive feedback loop;

[0094] Fig. 3 shows a flow diagram of an embodiment of a network protocol with access to information regarding code coverage;

[0095] Fig. 4 shows a schematic representation of an embodiment of an evolutionary algorithm.

[0096] Detailed description of the embodiments according to the figures

[0097] In the following, exemplary embodiments are described in more detail with reference to the figures. Furthermore, method steps relating to a specific feature of a device are interchangeable with that same feature of the device, and the same applies vice versa.

[0098] Figure 1 shows a schematic representation of a process flow 100 of an embodiment of a test setup (TE) 101. The TE 101 is linked to a test object (SUT) 104 via one or more communication paths 103. The TE 101 has one or more monitors 106 or monitoring modules 106 and is additionally linked to a model 107 of the test object 104.

[0099] The model 107 of the SLIT 104 is or was created, for example, by the TE 101 on the basis of an interaction 108 or an initial interaction 108 with the test object 104.

[0100] The TE 101 is designed to create one or more test cases 102 based on the model 104 and to receive one, several, or no responses 105 given to the generated test case 102. The TE 101 is further designed to check the functionality of one or more functions of the SLIT using one or more monitors 106. The received responses 105 and / or the results of the monitors 106 are evaluated to generate test results 109. Based on the test results 109, the TE 101 improves the model 107 of the SLIT 104. This improves the newly generated test cases 102, and the results 109 given thereon, in turn, refine the model 107.

[0101] In other words, Figure 1 illustrates the schematic flow of an embodiment. The test setup (TE) 101 generates one or more new test cases 102 and sends them to the test object (SUT) 104 via a communication path 103. The SUT 104 then responds with one, several, or no responses 105 via one or more communication paths 103. The TE 101 evaluates the received responses 105 and the results of the monitors 106. The model 107 of the SLIT 104 is derived from the sent test cases 102, the received responses 105, and the results of the monitors 106 and is, for example, continuously updated. Based on the (possibly improved) model 107 and the evaluation of the received responses 105 and / or the results of the monitors 106, the TE 101 generates one or more new test cases 102 and sends them to the SUT 104.

[0102] The test setup 101 uses test cases 102 to simulate virtual attacks to discover vulnerabilities in the test object 104 and also logs the test cases 102 that led to an anomaly in the behavior of the SUT 104. The decision as to whether an anomaly exists is made, for example, based on the model 107. By using the model 107, the security tests can be fully performed.

[0103] Furthermore, the creation and updating of model 107 is automated, eliminating the need for manual intervention or analyzing the internal functionality of SUT 104. This allows security tests to be performed with less effort. This enables greater test coverage for discovering or eliminating multiple vulnerabilities, thus increasing robustness against virtual attacks.

[0104] The test setup creates a positive feedback loop in which the new test cases 102 improve the model 107 and, based on the improved model 107, in turn generate new, improved test cases 102. This feedback loop will be explained in more detail in Figure 2. Figure 2 shows a flowchart of an embodiment of a positive feedback loop 200, which can optionally be used, for example, in test setups according to embodiments of the present invention. The feedback loop 200 has, for example, five steps.

[0105] The first step 210 comprises an execution of a query or a test case, such as test case 102 in Figure 1. The first step 210 is linked, for example, to the second step 220 and the fifth step 250.

[0106] The second step 220 comprises a summary of the answers, such as answer 105 in Figure 1, or the test results. The second step 220 is linked, for example, to the first step 210 and to the third step 230.

[0107] The third step 230 involves identifying state-changing requests or test cases. The third step 230 is linked, for example, to the second step 220 and the fourth step 240.

[0108] The fourth step 240 comprises a derivation or improvement of the state machine. The fourth step 240 is linked, for example, to the third step 230 and the fifth step 250.

[0109] The fifth step 250 comprises a selection of the next query or test case, such as test case 102 in Figure 1. The fifth step 250 is linked to the fourth step 240 and to the first step 210.

[0110] The TE executes a test case or query in the first step 210. The execution 210 of the test case provokes a test result, which is generated or summarized in the second step 220 from one, multiple, or no responses and / or a behavior change. The state-changing test cases are identified in the third step 230 based on the provoked test results. The model of the SLIT or the state machine is updated or derived in the fourth step 240 using the identified state-changing test cases. In the fifth step 250, new test cases are generated based on the already executed test cases and the updated model. The newly generated test cases are again executed in the first step 210, thus starting the feedback loop 200 from the beginning.Until a meaningful set of queries or test cases is reached, the TE cannot yet make a sound statement about the behavior of the test object, and the three steps on the right side of the diagram have no effect. This case is represented by the dashed arrow.

[0111] Concrete implementations

[0112] The concrete implementation of the above-mentioned solution can be achieved in various ways. Various exemplary implementations are presented below. This is not an exhaustive list, and combinations of implementations are possible. The diversity of the examples illustrates that the invention can be implemented in many ways.

[0113] One implementation of the invention represents or includes so-called "fuzzing" (robustness testing) of a web application based on a state machine. In this case, the SUT corresponds to a web application provided on a technical device or test object, the model is formed by a state machine, and the test strategy is based on "fuzzing." In this case, "fuzzing" refers to the insertion of many different values ​​and the SUT's reaction to these potentially unexpected values. In the specific case of a web application, this could be, for example, the input of special characters or program code, such as JavaScript code, into a text field. In this implementation, the SUT is considered a so-called "black box" (i.e., a closed system with its internal structure neglected), meaning no information about the SUT's internals is known.In this implementation, a test case consists, for example, of an HTTP request.

[0114] The concrete procedure is illustrated in Figure 2. The cycle begins in the top left of the diagram. First, for example, the initial request is sent to the web application; for example, the homepage of the web application is called up. Until a meaningful number of requests have been made, no well-founded statement can be made about the behavior of the web application. Therefore, the three steps on the right-hand side of the diagram initially have no effect. This case is represented by the dashed arrow. The next requests are now derived based on the queried homepage. For example, newly found hyperlinks are selected or buttons are pressed. The respective responses are summarized in so-called response clusters. For example, similar responses are grouped together in a cluster or group.This is the case, for example, for websites that only differ in the current time, but are otherwise identical in content.

[0115] For example, the same requests are deliberately made more frequently during the investigation. This makes it possible to determine whether the web application's response to the same request changes over time. If, for example, a response to a request differs from the previous response to the same request, it is assumed that an internal state transition of the web application occurred between these two requests.

[0116] This can be clearly seen in an online shop, for example. If I haven't added anything to my shopping cart yet and click "My Shopping Cart," I'll be taken to a web page that displays my empty shopping cart. However, if I add an item to my shopping cart and thus change the internal state of the web application, I'll receive a different response when I make the same request for my shopping cart. For example, it might now be possible to click the "Checkout" button, which wasn't possible previously with an empty shopping cart.

[0117] All requests that occur between the two requests under consideration, such as between the two visits to the "My Shopping Cart" page, can trigger the state change or be identified as triggering a state change. These state-changing requests are identified, for example, based on a heuristic. This heuristic includes, for example, how often the request has been identified as state-changing in the past, how often the request has already been executed, and the type of response or message, such as GET or POST.

[0118] Based on the information obtained, the current state machine is created or improved. It is assumed that the web application remains in a certain state until a state-changing request is executed.

[0119] The cycle continues. Further possible requests are selected, and the state machine is refined further and further. Once the hyperlinks, input fields, and buttons accessible via the website have been exhausted, the robustness testing algorithm, or fuzzing algorithm, takes over the selection of the next request. For this purpose, available URL parameters and input values ​​are tested against a wide range of values. The web application's response is compared with the state machine. If the web application behaves differently than the model, for example, this is evaluated as an anomaly, which may indicate a vulnerability.

[0120] Another implementation of the invention is also based on a state machine, but aims to implement a network protocol. In this implementation, a test case consists of a sequence of one or more network packets. Furthermore, this implementation assumes access to information regarding the code coverage of a specific input. In this specific case, this means that after inputting a specific sequence of network packets, the TE or algorithm receives information about which code blocks the SUT has executed in response to this sequence of network packets. This information is used to evaluate the quality of a test case. For example, a test case that triggers many new locations in the code is rated with a high quality. The flow of this example is illustrated in Figure 3.

[0121] Figure 3 shows a flowchart 300 of an implementation of a network protocol with access to information regarding code coverage. The flowchart 300 has a loop with, for example, four steps.

[0122] The first step 310 comprises, for example, providing or executing a sequence of network packets or a test case, such as test case 102 in Figure 1. The first step 310 is linked to the second step 320 and the fourth step 340.

[0123] The second step 320 comprises, for example, a calculation of the SUT's behavior or a code coverage analysis. The second step 320 is linked, for example, to the first step 310 and to the third step 330.

[0124] The third step 330, for example, comprises a check and, if necessary, update of the state machine. The third step 330 is linked, for example, to the second step 320 and to the fourth step 340. The fourth step 340, for example, comprises a selection of the next sequence of network packets. The fourth step 340 is linked, for example, to the third step 330 and to the first step 310.

[0125] The TE executes a test case in the first step 310. In this implementation, the test case consists of a sequence of one or more network packets. After inputting a specific sequence of network packets (e.g., into the SUT), the TE receives information (e.g., from the SUT) about which code blocks the SUT has executed in response to the sequence of network packets. Based on this information or this code coverage, the behavior of the SUT is analyzed or calculated, for example, in the second step 320. This analysis leads, for example, to a review and, if necessary, updating of the state machine of the SUT model in the third step 330. Based on the updated model, the next sequence of network packets is generated or selected, for example, in the fourth step 340, which is then executed in the first step.

[0126] The concrete procedure of the example is shown in Figure 3. Here, too, the cycle begins at the top left. The initial sequence of network packets is specified in this case by a starting value or a so-called "seed". This can be, for example, a single valid network packet, a sequence of network packets, or even an empty set. This start is then sent to the SUT, for example, and the information about the code coverage is received. Based on this code coverage, a value for the behavior of the SUT is calculated using a hash function, for example. Based on this behavior, a state machine is then constructed, for example. For example, it is assumed that the system is in a certain state until contradictory behavior is observed. This is based, for example, on an L* algorithm.

[0127] The selection of the next sequence is based on a so-called seed selection algorithm. This selection also includes, for example, a mutation algorithm that generates and / or executes new test cases based on the old ones. Furthermore, the selection of the next test case includes, for example, the state machine.

[0128] For example, if a state has not yet been sufficiently explored, new sequences are generated that induce precisely this state in order to learn how the SUT behaves in this state. Once all currently known states have been sufficiently explored, new test cases are generated that differ significantly from previously known test cases. These should help achieve new states.

[0129] Alternatively, the decision between exploitation and exploration—that is, whether to further explore known states or to discover new ones—can be modeled as a multi-armed bandit problem. For example, established reinforcement learning methods such as the Upper Confidence Bound (UCB) algorithm can be used to make this decision.

[0130] The SUT is continuously monitored during testing. If, for example, a program crash is detected during testing, this information is saved along with the previously submitted test case.

[0131] Figure 3 can also illustrate a further implementation of the invention in which the code coverage is illustrated based on a hidden Markov model (HMM).

[0132] In the latter implementation, for example, it was assumed that information on code coverage was available. This allows the behavior of the SUT to be quantified and used to construct a state machine. An alternative to this is the implementation described below. In this case, for example, the objective, the SUT, the algorithm for constructing the state machine, as well as the mutation and selection of the next sequence of network packets, remain the same. The calculation of the behavior of the SUT is different. Now it is assumed that information on code coverage is no longer directly available. Therefore, this information is also approximated using a model. A hidden Markov model (HMM), for example, is used as the model type for this.

[0133] At the beginning of the test process, for example, the first test cases are executed without the use of a state machine. For this purpose, the starting value or seed described above and, for example, the available mutations are used. The resulting network traffic is used, for example, to train the HMM. For example, a sequence of network packets from the communication between the test device and the SUT is viewed as an observation sequence for the HMM. A sequence of network packets refers to or describes, for example, the communication between a TCP connection establishment and a TCP connection termination. Each network packet is viewed as an individual observation, which in some cases makes the use of a multivariate HMM necessary.

[0134] A good choice for the number of internally used states can be made, for example, by repeatedly training the HMM with different numbers of nodes and then evaluating its quality, e.g., using the Bayesian Information Criterion. The trained HMM now represents a model of the internal processes of the SUT and can therefore be used, for example, to approximate code coverage as follows. For example, in an HMM, the path most likely taken by the model during the last execution can be calculated (or estimated) using the Viterbi algorithm. This path approximates, for example, the execution path of the SUT. This information can therefore be used, for example, in the same way as the information about code coverage in the previous implementation.For example, it is used again to quantify the behavior of the SUT, which in turn is used to create or update the state machine.

[0135] The selection of the next sequence of network packets is also based on the so-called seed selection algorithm. This selection also includes, for example, a mutation algorithm that generates and / or executes new test cases based on the old ones. This algorithm is explained below.

[0136] Evolutionary test case determination according to Figure 4

[0137] Figure 4 shows a schematic representation of an embodiment of an evolutionary algorithm 400, which can be used, for example, alone or in conjunction with the embodiments described herein.

[0138] A population of test cases 420 of the algorithm 400 is linked, for example, to the SUT 410. The algorithm comprises, for example, a machine learning algorithm 460 and a loop of the population of test cases 420, a mutation 430, offspring test cases or "offsprings" 440, and a selection 450. In other words, for example, a population of test cases can be generated in the loop, and then a mutation can occur, thus obtaining offspring test cases, and then a selection or selection can occur.

[0139] For example, the machine learning algorithm 460 is linked to the population of test cases 420, the mutation 430 and the selection 450.

[0140] The population of test cases 420 is executed, for example, against the SUT 410 to collect test results, for example, using monitors. The machine learning algorithm 460 is trained, for example, based on the test cases 420 and the test results.

[0141] Through mutation 430, for example, the machine learning algorithm 460 creates new test cases, the so-called "offsprings" 440, based on the population of test cases 420. Each test case is evaluated by the machine learning algorithm 460 using a fitness value. Based on the fitness value, a new population of test cases is selected 450, which is then executed against the SUT 410 to collect test results.

[0142] In other words, the evolutionary algorithm is used, for example, to generate new test cases. This starts with a starting value (seed), which represents the first population of the evaluation. New individuals are then created from the individuals of this population through mutation. These individuals are called "offspring" or offspring. A fitness value is then assigned to each of these individuals in the "offspring" (e.g., to the offspring) using a fitness function. Subsequently, the individuals with the highest fitness values ​​are included in the new generation of the population. In this process, each individual corresponds to a test case for the test procedure. The new generation of test cases is executed, for example, against the SUT, and the response is observed, for example, using monitors.This implementation of the invention consists, for example, of a combination of an evolutionary algorithm and a machine learning (ML) algorithm. The ML algorithm can be, for example, a support vector machine (SVM), a random tree (RT), or a neural network (NN).

[0143] In this implementation, for example, the network interface of an automation component can be tested. The test cases consist, for example, of a network packet (or multiple network packets). In principle, it is also possible to extend the approach so that a test case consists of a sequence of network packets. The model of the automation component is represented, for example, by the respective ML algorithm and models, for example, the behavior of the automation component in response to a test case. This behavior is described, for example, by the respective failed monitors. In the case of an automation component, these monitors consist, for example, of checking whether the automation component is still responding to ping requests and / or whether the web server of the automation component is still accessible.

[0144] The selected ML algorithm can, for example, be used in the evolutionary algorithm mentioned above to determine the fitness value. The ML algorithm is trained to approximate a function that maps a test case to the monitor outputs. For example, it estimates which monitors are likely to fail in a given test case.

[0145] One possible result of the estimation could be that the automation component still responds to ping after the test case has been executed, but the web server is no longer accessible. This estimation can be used, for example, to determine the quality, i.e. the fitness, of a test case or of an "individual". The more monitors fail as a result of the test case (or the more failures are determined as probable), the higher the quality. This makes it possible to predict whether (or with what expected probability) the test case will lead to a monitor failure. This is particularly useful in the case of automation components, since executing a test case here takes a relatively long time. The ML algorithm's estimate of the probable response of the automation component can, for example, be calculated much more quickly.For example, the ML algorithm is initially trained on the SUT's response to the test cases contained in the initial values, or "seed." It is then continuously retrained on the actually executed test cases. Here, the necessary evaluations, markings, or "labels" for the test cases are also available, for example, based on the subsequently measured behavior of the monitors.

[0146] In addition, the ML algorithm also influences the mutation of the test cases, for example. The exact nature of this influence depends, for example, on the selected ML algorithm. With a NN, for example, the gradient can be calculated that points in the direction of a (local) maximum of the approximated function. This local maximum then corresponds, for example, to a test case that will affect the highest possible number of monitors. By adapting the test cases according to this gradient, new test cases can be generated that have a high probability of causing many monitors to fail. It is similar with the SVM; here, too, vectors can be calculated that point to the test cases that cause a high number of monitors to fail.

[0147] Gradients or vectors cannot be calculated for an RT, but it can still be used for mutation. By tracing the path a test case has taken through the tree, conclusions can be drawn about necessary changes to a test case. For example, it can be checked which decisions in the RT led to the estimate that the test case will not cause many monitors to fail. These decisions are based, for example, on thresholds of certain properties of the test case. Now, for example, new test cases can be generated that fall above or below this threshold. These test cases then have a higher probability of causing a large number of monitors to fail.

[0148] As shown in Figure 4, the entire process begins with a starting value or “seed”, which is used, for example, as an initial population. The test cases of this population are executed against the SUT, and the reactions of the monitors, for example, are recorded. This is how the algorithm receives the evaluations or markings or “labels” for the test cases in the population. The combination of test cases and associated labels is now used, for example, to carry out an initial training of the ML algorithm. This is then used, for example, to carry out mutation and selection. This creates, for example, a new population, which is then executed against the SUT. For example, the reactions of the monitors are recorded again, and the ML algorithm is retrained, for example, with the newly generated data set. This is how a robustness test process orA fuzzing process has been established in which test case generation is supported, for example, by mutation, and the selection of the generated test cases is supported by an ML algorithm. This ML algorithm, for example, represents a model of the SUT by approximating a function that describes the behavior of the SUT in response to a test case.

[0149] Alternative implementation examples

[0150] The embodiments described above merely illustrate the principles of the present test assembly. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the test assembly be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein.

[0151] Although some aspects have been described in connection with a device, it should be understood that these aspects also represent a description of the corresponding method, so that a block or component of a device can also be understood as a corresponding method step or as a feature of a method step. Analogously, aspects described in connection with or as a method step also represent a description of a corresponding block, detail, or feature of a corresponding device. Some or all of the method steps can be carried out by (or using) a hardware apparatus. In some embodiments, some or more of the key method steps can be carried out by such an apparatus.

[0152] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray Disc, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, a hard disk, or other magnetic or optical storage device storing electronically readable control signals that can interact or cooperate with a programmable computer system to perform the respective method. Therefore, the digital storage medium may be computer-readable.

[0153] Some embodiments according to the invention thus comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system such that one of the methods described herein is carried out.

[0154] In general, embodiments of the present invention may be implemented as a computer program product having a program code, wherein the program code is effective to perform one of the methods when the computer program product is run on a computer.

[0155] The program code can, for example, also be stored on a machine-readable medium.

[0156] Other embodiments include the computer program for performing one of the methods described herein, wherein the computer program is stored on a machine-readable carrier.

[0157] In other words, an embodiment of the method according to the invention is thus a computer program which has a program code for carrying out one of the methods described herein when the computer program runs on a computer.

[0158] A further embodiment of the method according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded.

[0159] A further embodiment of the method according to the invention is thus a data stream or a sequence of signals that represents the computer program for carrying out one of the methods described herein. The data stream or the sequence of signals can be configured, for example, to be transferred via a data communication connection, for example, via the Internet.

[0160] A further embodiment comprises a processing device, for example a computer or a programmable logic device, which is configured or adapted to carry out one of the methods described herein.

[0161] A further embodiment comprises a computer on which the computer program for performing one of the methods described herein is installed.

[0162] A further embodiment according to the invention comprises a device or system designed to transmit a computer program for performing at least one of the methods described herein to a recipient. The transmission can be electronic or optical, for example. The recipient can be, for example, a computer, a mobile device, a storage device, or a similar device. The device or system can, for example, comprise a file server for transmitting the computer program to the recipient.

[0163] In some embodiments, a programmable logic device (e.g., a field-programmable gate array, an FPGA) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This may be general-purpose hardware such as a computer processor (CPU) or method-specific hardware such as an ASIC.

[0164] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein.

[0165] [1] Bringmann, Eckard, and Andreas Krämer. "Model-based testing of automotive systems." 2008 1st international conference on software testing, verification, and validation. IEEE, 2008.

[0166] [2] Bauer, Thomas, et al. "Combining combinatorial and model-based test approaches for highly configurable safety-critical systems." Model-based Testing in Practice (2009): 9.

Claims

Claims A test arrangement (101) for testing (210, 310) a test object (104, 410), wherein the test arrangement (101) is designed to create a model (107) (240, 330) based on an interaction (108) with the test object (104, 410), to generate one or more test cases (102, 420, 440) using the model (107) (250, 340, 430), to improve the model (107) of the test object (104, 410) based on real test results (109) based on the generated test cases (102, 420, 440). The test arrangement (101) according to claim 1, wherein the test arrangement (101) is designed to receive one, several or no response to a generated test case (102, 420, 440) from a test object (104, 410) that is unknown and / or opaque with respect to its behavior via one or more communication channels (103).The test setup (101) according to claim 1 or 2, wherein the test setup (101) is designed to verify the functionality of the test object (101) or the functionality of one or more functions of the test object (101) by means of one or more monitors (106) and / or monitoring modules (106). The test setup (101) according to claim 2 or 3, wherein the test setup (101) is designed to evaluate the received responses (105) and / or the results of the monitors (106) and to generate test results (109) based on this evaluation (220, 320). The test arrangement (101) according to one of claims 1 to 4, wherein the test arrangement (101) is designed to evaluate the quality of the test case (102, 420, 440) based on the test result (109) and / or based on a probability of a malfunction of the test object (104, 410) tested with the test case (102, 420, 440).

6. The test setup (101) according to any one of claims 1 to 5, wherein the test setup (101) is designed to predict (320) or estimate the test results (109) of the test cases (102, 420, 440) using the model (107).

7. The test arrangement (101) according to claim 6, wherein the test arrangement (101) is designed to select the test cases (102, 420, 440) in which a malfunction of the test object (104, 410) or a malfunction of one or more functions of the test object (104, 410) was predicted (320), and to test the test object (104, 410) with the selected test cases (102, 420, 440) in order to generate test cases (102, 420, 440) that produce as many malfunctions of the test object (104, 410) as possible as a real test result (109).

8. The test setup (101) according to any one of claims 1 to 7, wherein the test setup (101) is designed to determine, using the model (107), whether a real test result (109) is judged to be anomalous (230, 320).

9. The test setup (101) according to claim 8, wherein the test setup (101) is designed to assess the real test result (102, 420, 440) as abnormal (230, 320) if the real test result (109) indicates a malfunction of the test object (104, 410) or a malfunction of one or more functions of the test object (104, 410), or if the real test result (109) differs from the test result (109) predicted on the basis of the model (107) (230).

10. The test setup (101) according to one of claims 1 to 9, wherein the model (107) comprises a state machine (107).

11. The test arrangement (101) according to claim 10, wherein a state of the state machine (107) comprises at least one test case (102, 420, 440) and at least one test result (109) resulting therefrom. The test arrangement (101) according to claim 10 or 11, wherein the test arrangement (101) is designed to calculate a value for a state of the test object (104, 410) using a hash function and to identify the state with the calculated value. The test arrangement (101) according to any one of claims 1 to 12, wherein the test arrangement (101) is designed to bring the test object (104, 410) into a predetermined state and to test the test object (104, 410) in the predetermined state against one or more different test cases (102, 420, 440). The test setup (101) according to one of claims 1 to 13, wherein the test setup (101) is designed to identify (230) state-changing test cases (102, 420, 440) that lead to a state change of the test object (104, 410) and to update (240, 330) the model (107) based on this.The test arrangement (101) according to any one of claims 1 to 14, wherein the test arrangement (101) is designed to identify the state-changing test cases (102, 420, 440) using a heuristic (230), wherein the heuristic is designed to take into account a frequency of execution of a test case (102, 420, 440) and / or a frequency of identification (230) of a test case (102, 420, 440) as a state-changing test case (102, 420, 440) and / or a category or type of a test case (102, 420, 440). The test setup (101) according to any one of claims 1 to 15, wherein the test setup (101) is designed to assess whether a state is sufficiently explored and to generate new test cases (102, 420, 440) that elicit this state if the state is not sufficiently explored, or to generate new test cases (102, 420, 440) that differ significantly from the already known test cases (102, 420, 440) if the state is sufficiently explored. The test setup (101) according to any one of claims 1 to 16, wherein the test setup (101) is designed to model a decision as to whether known states should be further explored or whether new states should be discovered as a multi-armed bandit problem, and to solve the multi-armed bandit problem using a reinforcement learning method to make the decision.The test arrangement (101) according to one of claims 1 to 17, wherein the test arrangement (101) is designed to repeatedly test the test object (104, 410) with a test case (102, 420, 440) to determine whether the test result (109) of the test case (102, 420, 440) changes over time and to identify a change in state based on this. The test setup (101) according to claim 18, wherein the test setup (101) is designed to identify test cases (102, 420, 440) with which the test object (104, 410) was tested between the test case (102, 420, 440) and the repeated test case (102, 420, 440) as potentially state-changing test cases (102, 420, 440) (230).

20. The test arrangement (101) according to any one of claims 1 to 19, wherein the test arrangement (101) is designed to generate invalid and / or unexpected and / or random inputs (250, 340) when generating the test cases (102, 420, 440).

21. The test arrangement (101) according to any one of claims 1 to 20, wherein the test arrangement (101) is designed to use a mutation algorithm (430) when generating (250, 340) the test cases (102, 420, 440), which generates new test cases (102, 420, 440) based on old test cases (102, 420, 440).

22. The test setup (101) according to claim 21, wherein the test setup (101) is designed to use one or more machine learning algorithms (460) when there is a mutation (430) of the test cases (102, 420, 440).

23. The test arrangement (101) according to any one of claims 1 to 22, wherein the test arrangement (101) is designed to generate a plurality of intermediate test cases (440) starting from one or more predefined and / or random test cases (102, 420, 440), to assign a fitness value to each of the intermediate test cases (440), to select the intermediate test cases (440) depending on the respective assigned fitness value (450), and to use the selected intermediate test cases (440) as new test cases (102, 420, 440).

24. The test arrangement (101) according to claim 23, wherein the test arrangement (101) is designed to assign a fitness value to each of the intermediate test cases (440) based on a probability of a malfunction of the test object (102) tested with the intermediate test case (440).

25. The test setup (101) according to claim 23 or 24, wherein the test setup (101) is designed to use one or more machine learning algorithms (460) in determining the fitness value. The test setup (101) according to any one of claims 22 to 25, wherein the test setup (101) is designed to use one or more machine learning algorithms (460), such as support vector machine algorithms and / or random tree algorithms and / or neural network algorithms. The test setup (101) according to any one of claims 22 to 26, wherein the test setup (101) is designed to retrain the machine learning algorithm (460) or machine learning algorithms (460) based on the test results (109) of the executed test cases (102, 420, 440).The test setup (101) according to any one of claims 1 to 27, which is designed to execute a sequence of test cases (102, 420, 440) and to use the test results (109) as an observation sequence for training a hidden Markov model, and to estimate (320) or approximate (320) states of a state machine (107) with the hidden Markov model (107) based on an observation sequence. The test setup (101) according to claim 28, which is designed to bring the test object (104, 410) into the individual states of the hidden Markov model (107) and to generate multiple test cases (102, 420, 440) for each state, which bring the test object (102) from the current state into the other states of the hidden Markov model (107) in order to achieve good path coverage of the hidden Markov model (107).The test setup (101) according to claim 29, which is designed to calculate the most probable execution path of a test case in the hidden Markov model (107) using a Viterbi algorithm, and to calculate the path coverage of the test cases (102, 420, 440) based thereon. The test setup (101) according to any one of claims 1 to 30, wherein the test setup (101) is designed to continuously refine the model (107) and / or the generated test cases (102, 420, 440). The test setup (101) according to any one of claims 1 to 31, which is designed to generate one or more test cases (102, 420, 440) for a web application and to receive and / or evaluate (220, 320) the response (105) of the web application. The test setup (101) according to claim 32, which is designed to output the test case (102, 420, 440) or the test cases (102, 420, 440) in the form of Entries in one or more input fields of the web application and / or Activation of one or more buttons of the web application and / or to generate activation of one or more links of the web application. The test arrangement (101) according to claim 32 or 33, which is designed to generate the response (105) of the web application in the form of Activation or deactivation of one or more input fields of the web application and / or Activation or deactivation of one or more buttons of the web application and / or Activation or deactivation of one or more links of the web application and / or Addition or removal of one or more input fields and / or one or more buttons and / or one or more links, and / or Change of one or more texts or numbers, to receive and / or evaluate (220, 320).

35. The test arrangement (101) according to any one of claims 1 to 31, which is designed to generate one or more test cases (102, 420, 440) for a test object (104, 410) which uses a network protocol, and to receive and / or evaluate (220, 320) the response (105) of the test object (104, 410).

36. The test arrangement (101) according to claim 35, which is designed to generate the test case (102, 420, 440) or the test cases (102, 420, 440) in the form of one or more inputs in a sequence of one or more network packets (250, 340, 430).

37. The test arrangement (101) according to claim 35 or 36, which is designed to receive and / or evaluate (220, 320) the response (105) of the test object (104, 410) in the form of a sequence of one or more network packets and / or in the form of information regarding code coverage of the input.

38. The test arrangement (101) according to claim 37, which is designed to build a state machine (107) based on the test cases (102, 420, 440) and code coverages based thereon, to calculate a value for a state of the test object (104, 410) using a hash function, and to identify the state with the calculated value.

39. The test setup (101) according to claim 37 or 38, which is designed to use a sequence of communication between the test setup (101) and the test object (104, 410) as an observation sequence for training a hidden Markov model (107), and to estimate or approximate the code coverage with the Hidden Markov Model (107) based on the observation sequence. The test setup (101) according to claim 39, which is designed to bring the test object (104, 410) into the individual states of the Hidden Markov Model (107) as an initial state and to generate multiple test cases (102, 420, 440) for each state, which move the test object (104, 410) from the current state into the other states of the Hidden Markov Model (107) in order to achieve good path coverage of the Hidden Markov Model (107). The test setup (101) according to claim 40, which is designed to calculate the most probable execution path of a test case in the Hidden Markov Model (107) using a Viterbi algorithm and to calculate the path coverage of the test cases based on this.The test arrangement (101) according to any one of claims 1 to 31, which is designed to generate test cases (102, 420, 440) in the form of inputs of a network packet or a sequence of network packets for a network interface of a test object (104, 410), and to evaluate the functionality of the test object (104, 410) or the functionality of one or more functions of the test object (104, 410) by means of monitors (106), in a method for testing a test object (104, 410), comprising the following steps: Creating a model (107) based on an interaction with the test object (104, 410), Generating one or more test cases (102, 420, 440) using the model (107), Improving the model (107) of the test object based on real test results (109) based on the generated test cases (102, 420, 440).

44. A computer program for carrying out the method according to claim 43, when the computer program is executed on a computer.