Computer-implemented method for generating test scenarios for testing a technical system

Abstract

The invention relates to a computer-implemented method for generating test scenarios for testing a technical system, comprising the steps: a. providing a search space (10) (S1); wherein the search space (10) is defined by an operating design area (20), ODD; wherein the operating design area (20) comprises a plurality of elements (22); wherein the technical system is operated in the operating design area (20); b. generating at least one first test scenario based on the search space (10) (S2); wherein the at least one first test scenario comprises at least one first element of the plurality of elements of the operating design area (20), ODD; wherein the at least one first test scenario examines at least one corresponding dimension (12) of the search space (10); c. generating at least one associated first test result by executing the at least one first test scenario (S3); d.Evaluating the at least one associated first test result (S4); e. Generating at least one further test scenario based on the search space (10) taking the evaluation into account (S5); f. Repeating steps c. to d. with the at least one further test scenario and e. until the search space (10) is exhausted (S6); and g. Providing the at least one first test scenario, the at least one further test scenario and / or the corresponding test results (S7). Furthermore, the invention relates to a technical system and a corresponding computer program product.

Description

1. Technical FieldThe invention relates to a computer-implemented method for generating test scenarios for testing a technical system. The invention is also directed to a corresponding technical system and a computer program product.2. Prior ArtAutonomous driving is becoming increasingly important. Various autonomous vehicles, such as cars and trains, are known from the prior art in this context. The degree of automation also increases significantly.The autonomous vehicles are configured to be operated at least largely in a non-driver manner. In the course of the further development of autonomous vehicles, the controller is transmitted step by step by the vehicle driver to a technical system with automated control. Reliable automated obstacle detection and the introduction of appropriate countermeasures, such as rapid braking, present a great challenge to date.Furthermore, the autonomous vehicles, as well as most other safety critical technical systems, are increasingly based on machine learning. In other words, the technical systems are artificial intelligence (AI for short). According to the prior art, however, there has not yet been any guarantee that the object classification and / or the object recognition by the underlying machine learning model of the technical systems is also correct. Otherwise, the incorrect classification and consequently also control result in dangerous situations, in the worst case injured persons.For these reasons, coverage metrics are of critical importance for testing the technical systems and their software. The coverage metrics are required to determine whether the software of the technical systems is being tested sufficiently. However, conventional coverage metrics are not sufficiently applicable to the Kl-based technical systems. One challenge is to provide a coverage metric that can be used to assess whether and how fully a series of test scenarios cover an operational design domain (ODD).The present invention therefore addresses the problem of providing a computer-implemented method for generating test scenarios for testing a technical system, which is more efficient and reliable.3. SUMMARY OF THE INVENTIONThe above-mentioned object is achieved according to the invention by a computer-implemented method for generating test scenarios for testing a technical system, comprising the stepsa. providing a search space; wherein the search space is defined by an operational design area; wherein the operational design area comprises a plurality of elements; wherein the technical system is operated in the operational design area;b. generating at least one first test scenario based on the search space; wherein the at least one first test scenario comprises at least one first element of the plurality of elements of the operational design area; wherein the at least one first test scenario examines at least one corresponding dimension of the search space;c. generating at least one associated first test result by executing the at least one first test scenario;d. evaluating the at least one associated first test result;generating at least one further test scenario on the basis of the search space taking into account the evaluation;f. repeating steps c. to d. with the at least one further test scenario and e. until the search space is exhausted; andg. providing the at least one first test scenario, the at least one further test scenario and / or the corresponding test results.Accordingly, the invention is directed to a computer implemented method for generating test scenarios for testing a technical system. In other words, test scenarios are generated which are applied to the technical system in order to test the same. The term "test scenario" is to be interpreted in the conventional sense. Accordingly, a "test scenario" is a detailed description of a test condition or use case under which the technical system is to be tested. The technical system can be designed as any technical system, such as a safety-critical technical system, for example an autonomous vehicle or an industrial installation.In the first method step, the search space is provided. The search space is in this case related to a system context, namely the so-called operational design domain (ODD). The ODD typically includes a plurality of elements. The elements may also be configured as features or characteristics, such as operating states. Example operating conditions are weather conditions, visibility and time of day, etc. in autonomous vehicles.The input data may be received via one or more input interfaces. Additionally or alternatively, the output data may also be sent via one or more output interfaces. The interfaces can be designed as serial or parallel interfaces. The interfaces advantageously ensure efficient and friction-free data transmission between computing units. The data can be exchanged bidirectionally without data congestion.In the second method step, the search space is used to generate the first test scenario. The first test scenario searches the corresponding dimension of the search space. The elements of the ODD are related to the dimensions of the search space. If the first test scenario has a first element or two first elements, the first test scenario also examines the one first dimension or the two first dimensions of the search space.In a further method step, the first test scenario is executed. This results in the first test result, which is further evaluated.In a further method step, the search space is used again to generate the further test scenario. In this step, the previous evaluation is taken into account. The further test scenario can also be designed as a second test scenario.The above method steps are repeated with this further test scenario until the search space is exhausted. By depletion is meant that the search space has been fully searched, in other words each dimension of the search space has already been evaluated.In the last method step, a test scenario, a plurality of test scenarios, a test result and / or the corresponding test results are provided.In other words, the method iteratively passes through the search space that the ODD spans. In a closed loop approach, test scenarios are automatically generated, executed in a (simulated) test environment, the test results evaluated after execution, and new test scenarios are also created based on the results.The present invention thereby ensures that test scenarios are determined efficiently and reliably. With the aid of the test scenarios, the underlying technical system can thus also be tested in an efficient and reliable manner. This ensures that the technical system is operated reliably in its ODD.In other words, a coverage metric is provided that can be used to assess how fully a series of test scenarios cover an ODD.Most technical systems typically use machine learning or are Kl-based. Consequently, it can be determined which test scenarios can be managed or not with such technical systems.A further advantage of the method is that it enables the systematic coverage of the search space defined by the ODD during the test and the automated testing of the technical systems. The method is furthermore advantageously suitable for simulation-based software-in-the-loop or hardware-in-the-loop tests and functions with complex ODD definitions.Unless otherwise stated, all method steps of the computer-implemented method can be carried out by at least one arithmetic unit, which can also be referred to as a data processing device. In particular, the data processing device, which comprises at least one processing circuit which is designed or adapted to carry out a computer-implemented method according to the invention, can carry out the method steps of the computer-implemented method. For this purpose, the data processing device can store in particular a computer program comprising instructions which, when executed by the data processing device, in particular the at least one processing circuit, cause the data processing device to execute the computer-implemented method.The technical system may include, for example, a machine or plant, or multiple physically, virtually, and / or functionally connected machines and / or plants. Exemplary, but non-limiting, fields of application are systems in the field of energy technology, for example installations and / or machines for generating energy, for converting energy and / or for transporting energy. Other non-limiting areas of application are in the field of mobility, such as in railway traffic, such that the system may include, for example, train components, trains, track systems or parts thereof, passenger cars, trucks, and so forth, in the field of industrial production, such that the system may include, for example, production machines or plants, manufacturing machines or plants, test devices, monitoring plants, conveyor machines or plants, process engineering plants, and so forth. Other non-limiting areas of application are in the medical field, such that the system may include, for example, medical imaging devices, such as MRI systems, x-ray based imaging systems such as CT systems, ultrasound based imaging systems, PET systems, and so forth. The system may also include one or more robotic systems.In one embodiment, the search space has multiple dimensions and / or the number of dimensions matches the number of elements of the Operational Design Area, ODD. Accordingly, the ODD has a plurality of elements and the search space has a plurality of dimensions. If the ODD has n elements, the search space has corresponding n dimensions.In another embodiment, each dimension is defined by a enumeration, an integer, a number, a real number, or a range. Accordingly, each dimension in the search space is predefined.In a further configuration, the execution of the at least one first test scenario and / or the execution of the at least one further test scenario comprises the application of the at least one first test scenario to the technical system and / or the application of the at least one further test scenario to the technical system. The test scenario is performed by applying it to the technical system. In other words, the technical system is tested with the aid of the test scenario. The test scenario includes one or more elements of the ODD.For example, the autonomous vehicle is operated in its test scenario and thus generated ODD. The autonomous vehicle classifies, for example, data items such as image data, text data, or video data. In other words, object classification by the autonomous vehicle in the ODD is performed by machine learning. The test result in this example may include a list of classified objects.In a further configuration, the evaluation of the at least one first test result and / or of the at least one further test result further comprisescomparing the at least one first test result and / or the at least one further test result with at least one expected test result; anddetermining whether the technical system is operated correctly in the at least one first test scenario and / or at least one further test scenario taking into account the comparison.Accordingly, the test result is compared with an expected test result (also "ground trouth"). In other words, the expected test result is a reference result. The test results can coincide at least partially or completely with one another or can also deviate partially or completely from one another. The comparison is used to determine whether the technical system is operating properly in the test scenario.In a further configuration, the technical system is operated correctly in the at least one first test scenario and / or at least one further test scenario and / or the execution of the at least one test scenario was successful if the at least one first test result and / or the at least one further test result completely or at least partially matches the expected test result. If the test results correspond accordingly completely or at least partially, this indicates that the technical system is safely operated in the test scenario. In this case, the match, the execution of the test scenario was accordingly also successful. In the other case, the execution of the test scenario was not successful and thus failed.In a further configuration, the at least one further test scenario has at least one further element of the plurality of elements of the operating design area when the at least one first or further test scenario is successfully executed; wherein the further test scenario examines at least one further dimension of the search space. Accordingly, if the test scenario is successfully executed, the search space and thus also the test scenario can be extended by one or more dimensions. The extension thus results in an extended test scenario. In other words, in this case, the further test scenario is an extended test scenario which is more complex or more difficult compared to the other test scenario.In a further configuration, the at least one further test scenario has a further element of the plurality of elements of the operating design area in the event of failure to execute the at least one first or further test scenario; wherein the further test scenario examines exactly one dimension of the search space; wherein the at least one element of the first test scenario or further test scenario differs from the further element.Accordingly, if the test scenario fails to be executed, the search space and thus also the test scenario cannot be extended by dimensions. Instead, the further test scenario searches exactly one dimension of the search space. Consequently, it is ensured that exactly or exactly one dimension is considered and not several dimensions. This further test scenario is thus a simplified or simpler test scenario compared to the first test scenario. The elements differ between the first and the further test scenario.In a further embodiment, the computer-implemented method further comprises enabling the technical system for operation and / or discarding as a function of the at least one first test scenario, the at least one further test scenario and / or the corresponding test results. Accordingly, the technical system is released or rejected. The basis for this can be the test scenarios or the results thereof. If the test scenario is successfully executed, it can be enabled and also the technical system. If the test scenario fails to execute, it may not be enabled and the technical system is rejected. The technical system can then also be tested further and / or adapted if the release fails.In a further embodiment, the computer-implemented method further comprisesoutputting the at least one first test scenario, the at least one further test scenario, the corresponding test results and / or other associated data on a display unit,storing the at least one first test scenario, the at least one further test scenario, the corresponding test results and / or other associated data in a memory unit, and / ortransmitting the at least one first test scenario, the at least one further test scenario, the corresponding test results and / or other associated data to a computing unit.Accordingly, one or more measures can be initiated after the output data of the method according to the invention have been provided. The measures can be carried out simultaneously, successively or stepwise.First, the further test scenario can be displayed to a user or a user on a display unit of a computing unit. Furthermore, the further test scenario can be stored, the test scenario itself or in the form of a corresponding message or message can be transmitted to another unit, such as a terminal, a control unit or other computing unit. The receiving computing unit can likewise initiate further corresponding measures after reception. Further measures are, for example, control measures.For example, after the release has taken place, a computing unit can receive the further test scenario or a message about the release and, as a function thereof, operate the technical system.This has the advantage that any measures are also implemented reliably and promptly.The invention further relates to a technical system for carrying out the above method.The invention further relates to a computer program product comprising a computer program which has means for carrying out the method described above when the computer program is executed on a program-controlled device.A computer program product, such as a computer program means, can be provided or supplied, for example, as a storage medium, such as a memory card, USB stick, CD-ROM, DVD, or else in the form of a downloadable file from a server in a network. This can be effected, for example, in a wireless communication network by the transmission of a corresponding file with the computer program product or the computer program means. A control device, such as an industrial control PC or a programmable logic controller or a programmable logic controller, PLC for short, or a microprocessor for a smart card or the like, is particularly suitable as the program-controlled device.4. Brief Description of the DrawingsIn the following detailed description, preferred embodiments of the invention are further described herein with reference to the following figures. FIG. 1 shows a schematic flow diagram of the method according to the invention. FIG. 2 shows a schematic view of the closed loop approach according to the method according to the invention. FIG. 3 shows a schematic view of an ODD according to an embodiment of the invention. FIG. 4 is a schematic view of a search space according to an embodiment of the invention. FIG. 5 shows a schematic view of a search space of a first test scenario according to an embodiment of the invention. FIG. 6 shows a schematic view of a search space of a further test scenario according to an embodiment of the invention. FIG. 7 shows a schematic view of a search space of a further test scenario according to an embodiment of the invention.5. DESCRIPTION OF THE PREFERRED EMBODIMENTSPreferred embodiments of the present invention will be described below with reference to FIG. 1.FIG. 1 schematically illustrates a flow diagram of the method according to the invention with method steps S 1 to S 7.FIG. 2 shows a schematic view of the closed loop approach according to the method according to the invention.FIG. 3 shows a schematic view of an ODD 20 according to an embodiment of the invention. The ODD 20 spans the search space 10 shown in FIG. 4. Search space 10 is preferably n-dimensional, where n is the number of elements 12 defined in ODD 20. In other words, in a search space 10 having n dimensions 12, the ODD 20 also has n elements 22. The ODD 20 may refer to an autonomous train, as shown in FIGS. 3 and 4. In FIG. 3, the ODD 20 includes four elements 22, namely, time of day, rain, number of tracks, and visibility. Each dimension 12 (element of the ODD 22) of the search space 10 may be indicated by an enumeration, an integer, a real number, or a defined range. If the ODD 20 has four elements, the search space 10 also has four dimensions 12. While the dimensions 12 'time of day' and 'rain' may be defined by enumerations, the dimension 'number of tracks' may be defined by integers in the range [1; 8], and the dimension 'visibility' may be defined by integers in the range [1000; 10] meters.Generation of the Test Scenarios, First Test Scenario 30, S 2A test scenario includes at least one element 22 of the ODD 20 and up to all elements 22 of the ODD 20. In the case of four elements 22, the test scenario can accordingly also have one to four elements 22.In this case, the generation of the test scenarios can begin with a random scenario.FIG. 5 shows a schematic view of a search space of a first test scenario according to an embodiment of the invention. The first test scenario 30 includes the two elements 22 of the ODD 20, light rain, and a visibility of 400 meters.Test Execution S3The first test scenario 30 is executed, for example with the aid of a simulation. The first test scenario 30 may be used as input to the technical system. For example, the first test scenario 30 may be fed into object detection / classification by generated image data or lidar data.Evaluation of Test Results S 4The results of the test execution may also be called test results. The test results can be present, for example, as a list of classified objects, bounding boxes, segmentation masks, etc. They can be compared to ground truth information. The ground truth information is preferably generated together with the input data in the phase of generating the test scenarios.This comparison can be used to check whether the technical system can operate correctly in the given test scenario. This is the case, for example, when all objects are correctly recognized or classified. If this is not the case, the test scenario has failed.Generation of the test scenarios, further test scenario 40, S 5Based on the test results of the previous evaluation, further test scenarios 40 are generated.If the test execution was successful, a further test scenario 40 is generated, which further examines at least one dimension 12 of the search space 10, since the successful test shows that the technical system can handle the first test scenario. In this case, another enumeration of the dimension can be selected, the integer value or the real value can be increased. This leads to more complex or more difficult test scenarios. In FIG. 6, the amount of rain is increased from "light rain" to "medium rain".If, on the other hand, the test execution has failed, a further test scenario 40 is generated which further examines exactly one dimension 12 of the search space 10. In this case, another enumeration of the dimension can be selected, the integer value or the real value can be reduced. This leads to simplified or simpler test scenarios. In FIG. 7, the visibility is set to 600 meters.In the case of the test case generation, in the further test scenarios, at least one other dimension of the search space can be selected in each iteration, which dimension is changed during the test case generation in order to avoid oscillations. Furthermore, the steps in which the values are increased or decreased may be reduced or increased in each iteration.

Claims

A computer implemented method for generating test scenarios (30, 40) for testing a technical system, comprising the steps of a. providing a search space (10) (S1); wherein the search space (10) is defined by an operational design area (20), ODD; wherein the operational design area (20) comprises a plurality of elements (22); wherein the technical system is operated in the operational design area (20); b. generating at least one first test scenario (30) based on the search space (10) (S2); wherein the at least one first test scenario (30) comprises at least one first element of the plurality of elements of the operational design area (20), ODD; wherein the at least one first test scenario (30) examines at least one corresponding dimension (12) of the search space (10); c. generating at least one associated first test result by executing the at least one first test scenario (30)(S3); d. evaluating the at least one associated first test result (S4); e. generating at least one further test scenario (40) on the basis of the search space (10) taking into account the evaluation (S5); f. repeating steps c. to d. with the at least one further test scenario (40) and e. until the search space (10) is exhausted (S6); and g. providing the at least one first test scenario (30), the at least one further test scenario (40) and / or the corresponding test results (S7).The computer implemented method of claim 1, wherein the search space (10) has multiple dimensions (12) and / or the number of dimensions (12) matches the number of elements (22) of the operational design area (20), ODD.The computer implemented method of claim 2, wherein each dimension (12) is defined by a enumeration, an integer, a number, a real number, or a range.The computer-implemented method according to any one of the preceding claims, wherein executing the at least one first test scenario (30) and / or executing the at least one further test scenario (40) comprises respectively applying the at least one first test scenario (30) to the technical system and / or applying the at least one further test scenario (40) to the technical system.The computer-implemented method according to any one of the preceding claims, wherein evaluating the at least one first test result and / or the at least one further test result further comprises - comparing the at least one first test result and / or the at least one further test result with at least one expected test result; and - determining whether the technical system is operating correctly in the at least one first test scenario (30) and / or at least one further test scenario (40) taking into account the comparison.The computer-implemented method according to claim 5, wherein the technical system is operated correctly in the at least one first test scenario (30) and / or at least one further test scenario (40) and / or the execution of the at least one test scenario (30) was successful if the at least one first test result and / or the at least one further test result completely or at least partially matches the expected test result.The computer-implemented method according to any one of the preceding claims, wherein the at least one further test scenario (40) comprises at least one further element of the plurality of elements of the operating design area (20) upon successful execution of the at least one first (30) or further test scenario (40); wherein the further test scenario (40) examines at least one further dimension of the search space (10).The computer-implemented method according to any one of the preceding claims, wherein the at least one further test scenario (40) comprises a further element of the plurality of elements of the operating design area (20) upon failure of execution of the at least one first (30) or further test scenario (40); wherein the further test scenario (40) examines exactly one dimension of the search space (10); wherein the at least one element of the first test scenario (30) or further test scenario (40) is different from the further element.The computer-implemented method according to any one of the preceding claims, further comprising enabling the technical system for operation and / or discarding depending on the at least one first test scenario (30), the at least one further test scenario (40) and / or the corresponding test results.Computer-implemented method according to one of the preceding claims, further comprising - outputting the at least one first test scenario (30), the at least one further test scenario (40), the corresponding test results and / or other associated data on a display unit, - storing the at least one first test scenario (30), the at least one further test scenario (40), the corresponding test results and / or other associated data in a memory unit, and / or - transmitting the at least one first test scenario (30), the at least one further test scenario (40), the corresponding test results and / or other associated data to a computing unit.A technical system for performing the method of any preceding claim.Computer program product comprising a computer program which has means for carrying out the method according to one of Claims 1 to 10 when the computer program is executed on a program-controlled device.

Images