Method for a continuous integration approach of driver assistance systems

Abstract

To provide a method and test system for a continuous integration approach for driver assistance systems based on data-driven validations.SOLUTION: The method comprises performing data-driven validations repeatedly at predetermined time intervals, each validation including the steps of: loading a current version of the overall software onto a server; transferring a computer program to a hardware-in-the-loop test bench; loading a test data set onto the hardware-in-the-loop test bench; running the program; logging differences in output between the current software version and output data of the test data set; matching the output of the current software version with ground-truth data, thereby evaluating performance of the current software version; and forming performance statistics on all differences and their performance score; and evaluating changes in the overall software.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to a method for a continuous integration approach of driver assistance systems based on data-driven validation. Furthermore, a test system for a continuous integration approach of driver assistance systems is claimed. [Background technology]

[0002] The so-called continuous integration approach describes a method from software development in which the current state of a software project is compiled and automatically tested at short intervals (e.g., daily). This makes it possible to detect and correct errors very early, while their impact on a continuously evolving software project is still small. A prerequisite for this continuous integration approach is the continuous availability of the entire compilable and testable software of a software project.

[0003] For driver assistance systems (ADAS) and automated driving systems (ADS), the continuous integration approach is currently limited to simple subcomponent validation tests, where particularly simple test cases, HMI (Human Machine Interface) tests, and tests related to communication with partner control devices are tested. However, many relevant test cases or traffic scenarios are not tested, as these test cases are very complex to reproduce and are usually completely unknown or lack description.

[0004] Additionally, ADAS / ADS have a non-functional nature. This means that, in principle, all scenarios that occur in traffic must be correctly handled by the driver assistance system. However, the number of these scenarios is so large that, on the one hand, they are not all known, and, on the other hand, they cannot be described in specifications. As a result, the verification tests derived from the specifications of the continuous integration approach are also incomplete.

[0005] DE 10 2016 220 913 A1 discloses a method for generating test cases for autonomous vehicles, where the test cases are generated automatically based on data from vehicles involved in public road traffic.

[0006] A simulation system for simulating an autonomous vehicle is known from US Patent Application Publication No. 2021 / 0103283, in which the simulation system processes sensor data from a traffic scenario and errors and problems are automatically found using a continuous integration approach.

[0007] DE 102019134053 A1 describes an application method for the application of driver assistance systems, in which new application states are tested with regard to their performance in a continuous integration environment. Summary of the Invention [Problem to be solved by the invention]

[0008] In view of this, the object of the present invention is to provide a method for a continuous integration approach to driver assistance systems, in which the entire software of the driver assistance system is tested in everyday traffic situations. The test cases to be considered are not limited to simple structured test cases, but can also cover customer-specific scenarios. Furthermore, a test system is provided on which the method is performed. [Means for solving the problem]

[0009] To solve the above-mentioned task, a method for a continuous integration approach of a driver assistance system is provided, in which, during operation of a vehicle in real traffic, measurement data is measured by vehicle sensors, vehicle actuator settings are determined, and the measurement data and settings are stored in a test dataset, the test dataset comprising time series of input data and output data of the driver assistance system. The test dataset is used to test a system under test, and a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system under test. The system under test is formed or represented by an entire software, which is continuously changed. At predetermined time intervals, the following steps of data-driven verification are performed: loading into the server the current software version of the entire software forming the system under test; Compiling the current software version on the server into an executable computer program (thus simultaneously testing its readability); transferring the computer program to a hardware-in-the-loop test bench (thus simultaneously testing the transfer operation, e.g., by flashing); loading the test data set into a hardware-in-the-loop test bench; running a computer program on the hardware-in-the-loop test bench while supplying input data to a test data set; Logging each difference in output between the current software version and the output data of the test data set; simultaneously matching the output of the current software version with the ground truth data, thereby inferring an improvement or degradation in performance of the current software version, and assigning a performance score to each difference; forming performance statistics for all differences and their performance scores; evaluating the overall software changes based on performance statistics; The step of outputting the report is performed iteratively. The entire modified software is executed while controlling the actual vehicle.

[0010] The method according to the invention advantageously makes it possible to automatically verify driver assistance systems under customer-specific operation. These automatic tests can be run continuously, since they do not require a human test engineer.

[0011] In one embodiment of the method according to the invention, the test data set is generated from at least one of the following sources: customer vehicles involved in normal road traffic, vehicles taking part in testing of the system under test in normal road traffic.

[0012] In a further embodiment of the method according to the invention, the test data set is formed from at least one time series belonging to the following data categories: sensor data, map material, traffic conditions, output of driver assistance systems.

[0013] In a successive further embodiment of the method according to the invention, newly occurring system under test errors are enumerated by a report during the verification iteration.

[0014] In a still further embodiment of the method according to the invention, an open-loop ADAS / ADS is tested as the system under test.

[0015] In a still further embodiment of the method according to the invention, the system under test is formed by one of the following open-loop driver assistance systems: traffic sign recognition, night vision, egomotion locator.

[0016] In another embodiment of the method according to the invention, at least one subcomponent of a closed-loop system is tested using a functional decomposition concept. In this case, the closed-loop system comprises at least one closed-loop subcomponent and at least one open-loop subcomponent, and at least one open-loop subcomponent is selected as the at least one subcomponent to be tested. Thus, for example, in a distance control cruise control, all sensory components or lateral control can be tested as open-loop subcomponents. This represents a particularly advantageous embodiment of the method according to the invention, since a closed-loop system generally cannot be tested based on recorded and therefore immutable data, since any output of the closed-loop system will in turn affect the input data.

[0017] In yet another embodiment of the method according to the invention, multiple test benches are used in parallel by dividing the test data set into multiple periods and supplying a test data set to each test bench in each period. When using hardware-in-the-loop test benches, real-time test procedures must be performed. By partitioning the test data set according to the invention, advantageously, faster start-up can be achieved.

[0018] In another sequential embodiment of the method according to the invention, each change in the software version is discarded according to the evaluation, after which a new iterative launch is initiated.

[0019] Furthermore, a test system for a continuous integration approach of driver assistance systems is claimed, the test system comprising a server and a hardware-in-the-loop test bench having a computing unit, the computing unit being configured to execute an algorithm according to the method according to the invention.

[0020] Further advantages and embodiments of the invention will become apparent from the description and accompanying drawings.

[0021] It is understood that the features mentioned above and those to be described below may not only be used in the combinations specified, but may also be used in other combinations or alone without departing from the scope of the present invention.

[0022] In an exemplary embodiment of the method according to the present invention, a continuous integration approach for a system with a functional deficiency is presented below. As an example, the system under test is formed by traffic sign recognition software. Video data and bus signals are available from customer data or from conducted tests. Changes to the traffic sign recognition software developed throughout the day are loaded onto a server every evening. These changes may include, for example, recognition algorithms, map data, legislative data sets, and fusion algorithms. The current software version on the server is compiled and automatically launched on a computing unit of a hardware-in-the-loop test bench. Here, the first test cases for software integrity and flashability have already been performed. The recorded data is now fed into the hardware-in-the-loop test bench, so that tests are "virtually" launched again with the new software. At the same time, the software output, i.e., the traffic signs displayed to the customer, matches the ground truth data. Statistics are derived from this regarding whether the software changes have led to an improvement or deterioration in ADAS / ADS performance. The final report may include this statistics and additional information about new errors that arise during data-driven verification, such as low frequency scenarios that introduce errors into the new software. [Brief explanation of the drawings]

[0023] [Figure 1] 1 shows a flowchart of a continuous integration approach in one embodiment of the system according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] In Fig. 1, a flow chart 10 is shown for the continuous integration approach in one embodiment of the system according to the invention. Measurement data 11, measured and recorded in real traffic by vehicle sensors, form the basis of a test dataset for a system under test 12, which is implemented on a hardware-in-the-loop test bench and thus goes through various driving scenarios. The respective output data are matched in comparison 14 with the associated ground truth data 13, if an improvement or degradation in the performance of the current software version of the system under test 12 is to be inferred from the comparison 14. [Explanation of symbols]

[0025] 10. Continuous Integrated Approach Flowchart 11 Recorded Data 12 System Under Test (SUT) 13 Related Ground Truth Data 14 Matching output data with ground truth data

Claims

1. 1. A method for a continuous integration approach of a driver assistance system, wherein, during operation of a vehicle in real traffic, measurement data are measured by vehicle sensors and settings of vehicle actuators are determined, said measurement data and settings are stored in a test dataset (11), said test dataset (11) comprising time series of input and output data of said driver assistance system, a system under test (12) is tested with said test dataset (11), a driver assistance system (ADAS) or an automated driving system (ADS) is selected as said system under test (12), said system under test (12) is formed by a software entity, said software entity being continuously modified, said data-driven verification comprising the following steps: loading a current software version of the entire software forming the system under test (12) into a server; compiling the current software version on the server into an executable computer program; transferring the computer program to a hardware-in-the-loop test bench; loading said test data set (11) into said hardware-in-the-loop test bench; running the computer program on the hardware-in-the-loop test bench while supplying the input data to the test data set; logging each difference in output between the current software version and the output data of the test data set; - simultaneously matching (14) the output of the current software version with ground truth data (13), thereby inferring an improvement or degradation in performance of the current software version and assigning a performance score to the respective difference; forming performance statistics for all differences and their performance scores; evaluating the overall software changes based on the performance statistics; and outputting a report, the data-driven validation being performed repeatedly at predetermined time intervals; the entire modified software is executed while controlling an actual vehicle; The method, wherein the data-driven verification is performed by a computing unit included in the hardware-in-the-loop test bench.

2. 2. The method of claim 1, wherein the test data set (11) is generated from at least one of the following sources: customer vehicles involved in road traffic; and vehicles involved in testing the system under test in road traffic.

3. 2. The method according to claim 1, wherein the test data set (11) is formed by at least one time series for the following data categories: sensor data, map material, traffic conditions, output of the driver assistance system.

4. 2. The method of claim 1, wherein the report lists errors occurring in a new system under test (12) that occurred during a verification iteration.

5. The method of claim 1 , wherein an open-loop ADAS / ADS is tested as the system under test (12).

6. 6. The method of claim 5, wherein the system under test (12) is formed by one of the following open-loop driver assistance systems: traffic sign recognition, night vision, self-motion locator.

7. 2. The method of claim 1, wherein a functional decomposition concept is used to test at least one subcomponent of a closed-loop system, the closed-loop system comprising at least one closed-loop subcomponent and at least one open-loop subcomponent, and the at least one open-loop subcomponent is selected as the at least one subcomponent to be tested.

8. 2. The method of claim 1, wherein multiple test benches are used in parallel by dividing the test data set (11) into multiple periods and supplying a test data set (11) to each of the test benches in each of the periods.

9. The method of claim 1 , wherein, following the evaluation, each change in the software version is discarded and a new iterative launch is initiated.

10. 10. A test system for a continuous integration approach of driver assistance systems, the test system comprising: a server; and a hardware-in-the-loop test bench having a computing unit, the computing unit being configured to execute an algorithm according to the method of any one of claims 1 to 9.

Images