Method for continuously integrating strategies for driver assistance system

By collecting data in real road traffic and verifying driver assistance system software on hardware on the ring test bench, the problems of insufficient functions and incomplete verification in the existing technology are solved, and the automation of driver assistance system is achieved, all-weather and weather-oriented Verification and performance evaluation improve system reliability and efficiency.

CN119024736APending Publication Date: 2024-11-26DR ING H C F PORSCHE AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410636821.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-05-22
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing continuous integration strategies of driver assistance systems and autonomous driving systems are limited to simple sub-component verification and testing, and cannot fully cover complex traffic scenarios and be close to customers' actual use, resulting in insufficient functions and incomplete verification.

Method used

By using vehicle sensor measurement data in real road traffic, a test data set is formed, and automated verification is carried out on the hardware on the ring test bench, recording and comparing software output with ground real-time data, evaluating performance and generating reports, achieving driver assistance. Comprehensive testing and verification of the system.

Benefits of technology

It realizes automated and 24/7 verification of the driver assistance system, can detect performance improvements or deteriorations in operation close to customers, and automatically detect new errors, improving the reliability and efficiency of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024736A_ABST
    Figure CN119024736A_ABST
Patent Text Reader

Abstract

The invention relates to a method for a continuous integration strategy of a driver assistance system, a test data set comprising a time sequence of input data and output data of the driver assistance system and being formed during driving in real traffic, the driver assistance system or an automated driving system being tested as a system under test with the test data set, the system under test is formed from a continuously altered overall software, performing the repeating steps of data-driven verification at predetermined time intervals: loading the current software version onto the server; compiling into a computer program and transmitting the computer program to a hardware-in-the-loop test board; loading the test data set to a hardware-in-the-loop test board; executing a computer program; recording a corresponding output difference between the current software version and the output data of the test data set; comparing the output of the current software version with ground live data, thereby evaluating the performance of the current software version; forming performance statistics about all differences and performance evaluation thereof; and evaluating the change of the whole software.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for a continuous integration strategy for a driver assistance system based on data-driven verification. Furthermore, protection is claimed for a test system for a continuous integration strategy for a driver assistance system. Background Art

[0002] The so-called continuous integration strategy describes an approach from software development in which the current state of the software project is compiled and automatically tested at short intervals (e.g., every day). This enables early detection and repair of errors at a stage when their impact on the continuously updated software project is still small. A prerequisite for this continuous integration strategy is the continuous availability of a compilable and testable overall software of the software project.

[0003] In driver assistance systems (ADAS) and automated driving systems (ADS), continuous integration strategies are currently limited to validation tests of simple subcomponents. Here, especially simple test cases, HMI (Human Machine Interface) tests and tests on communication with partner control devices are examined. However, many relevant test cases or traffic scenarios are not examined. This is because the reproduction of these test cases is very complex and is often not known or poorly described.

[0004] In addition, ADAS / ADS have the characteristic of being under-functional. This means that, in principle, the driver assistance system must correctly handle all scenarios that occur in traffic. However, the number of these scenarios is very large, which cannot be known on the one hand, and cannot be described in the specification on the other hand. Therefore, the verification tests derived from the specification of the continuous integration strategy are also incomplete.

[0005] Document DE 10 2016 220 913 A1 discloses a method for generating test situations for an autonomous vehicle, wherein the test situations are automatically generated based on data from vehicles participating in public road traffic.

[0006] Document US 2021 / 0103283 A1 discloses a simulation system for simulating autonomous motor vehicles, wherein the simulation system processes sensor data from traffic scenarios and automatically discovers errors and problems using a continuous integration strategy.

[0007] Document DE 10 2019 134 053 A1 describes an application method for applying a driver assistance system, in which a new application state is subjected to a performance check with the aid of a continuous integration environment. Summary of the invention

[0008] In view of this, the object of the present invention is to provide a method for a continuous integration strategy for a driver assistance system, in which the overall software of the driver assistance system is tested in everyday traffic situations. The test cases to be considered should go beyond simple structured test cases and should also cover scenarios close to the customer. Furthermore, a test system for carrying out the method should be provided.

[0009] In order to solve the above-mentioned tasks, a method for a continuous integration strategy for a driver assistance system is proposed, wherein during the driving of a vehicle in real road traffic, measurement data are measured by vehicle sensors and the settings of vehicle actuators are determined, and the measurement data and settings are stored in a test data set, which includes a time series of input data and output data of the driver assistance system. The test data set is used to test the system under test, wherein a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system under test. The system under test is formed or represented by an overall software, wherein the overall software is continuously changed. The following steps of data-driven verification are repeatedly performed at predetermined time intervals:

[0010] Load the current software version of the overall software that forms the system under test onto the server;

[0011] Compile the current version of the software on the server into an executable computer program (thereby testing compilability at the same time);

[0012] Transferring a computer program to a hardware-in-the-loop test bench (thereby simultaneously testing the transfer process, such as by flashing);

[0013] Load the test data set into the hardware-in-the-loop test bench;

[0014] running the computer program on a hardware-in-the-loop test bench, supplying input data of a test data set;

[0015] Record the corresponding output differences between the current software version and the output data of the test dataset;

[0016] Compare the output of the current software version with the ground truth data to infer the performance improvement or degradation of the current software version and assign performance ratings to the corresponding differences;

[0017] · Form performance statistics about all differences and their performance evaluation;

[0018] · Evaluating changes to the overall software based on the performance statistics;

[0019] Output report.

[0020] The overall software that performs the changes during control of the real vehicle.

[0021] Advantageously, the method according to the invention can be used to automatically verify a driver assistance system in operation close to the customer. Since these automatic tests do not require a human test engineer, they can be run around the clock.

[0022] In one embodiment of the method according to the invention, the test data set is generated from at least one of the following sources: a customer vehicle participating in normal road traffic; a vehicle participating in normal road traffic for the purpose of verifying the system under test.

[0023] In a further embodiment of the method according to the invention, the test data set is formed by at least one time series of the following data categories: sensor data, map material, traffic situation, output of a driver assistance system.

[0024] In another continuation of the method according to the invention, the report lists errors of the tested system that have newly occurred during the verification iterations.

[0025] In yet another embodiment of the method according to the present invention, an open-loop ADAS / ADS is tested as the system under test.

[0026] In a further continuation of the method according to the invention, the system under test is formed by one of the following open-loop driver assistance systems: traffic sign recognition, night vision, ego-motion localizer.

[0027] In another embodiment of the method according to the invention, the concept of functional decomposition is used to test at least one subcomponent of a closed-loop system. In this case, the closed-loop system comprises at least one closed-loop subcomponent and at least one open-loop subcomponent, and at least one open-loop subcomponent is selected as the at least one subcomponent to be tested. Thus, for example, in a distance-controlled cruise control, all sensor components or lateral control devices can be tested as open-loop subcomponents. This is a particularly advantageous embodiment of the method according to the invention, since usually closed-loop systems cannot be tested based on recorded and therefore unchangeable data, since any output of the closed-loop system would in turn have an influence on the input data.

[0028] In a further embodiment of the method according to the invention, a plurality of test benches are used in parallel by dividing the test data set into a plurality of time periods and supplying the test data sets of the respective time periods to the respective test benches. When using hardware-in-the-loop test benches, real-time test procedures must be performed. By means of the division of the test data set according to the invention, faster operation can advantageously be achieved.

[0029] In a further embodiment of the method according to the invention, the corresponding changes in the software version are discarded according to the evaluation and a new iteration is then started.

[0030] Furthermore, protection is claimed for a test system for a continuous integration strategy of a driver assistance system, wherein the test system comprises a server and a hardware-in-the-loop test bench having a computing unit. The computing unit is configured to execute an algorithm according to the method according to the invention.

[0031] Other advantages and embodiments of the invention will appear from the description and the accompanying drawings.

[0032] It is to be understood that the features mentioned above and those yet to be explained below can be used not only in the respectively specified combination but also in other combinations or on their own, without departing from the scope of the present invention.

[0033] In an example embodiment of the method according to the present invention, a continuous integration strategy for a system with insufficient functionality is given below. By way of example, the system under test is formed by traffic sign recognition software. Video data and bus signals can be obtained from customer data or from tests that have been performed. Changes to the traffic sign recognition software developed during the day are loaded onto the server every night. These changes may include, for example, recognition algorithms, map data, legislative data sets, and fusion algorithms. The current software version on the server is compiled and automatically runs on a computing unit in a hardware-in-the-loop test bench. The first test case for the compilability and flashability of the software has been performed here. The recorded data is now fed to the hardware-in-the-loop test bench so that the inspection is run "virtually" again using the new software. At the same time, the output of the software (i.e., the traffic signs displayed to the customer) is compared with ground-truth data. Statistical data on the improvement or degradation of the performance of ADAS / ADS caused by software changes is thus obtained. The final report may include the statistics and additional information about new errors that appear during data-driven verification, such as infrequent scenarios that cause new software errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flow chart showing a continuous integration strategy in an embodiment of a system according to the present invention. DETAILED DESCRIPTION

[0035] exist Figure 1 In the figure, a flow chart 10 of a continuous integration strategy in an embodiment of the system according to the invention is shown. Measurement data 11 measured and recorded by vehicle sensors in real road traffic form the basis for a test data set for a system under test 12, which is executed on a hardware-in-the-loop test bench and is thus subjected to various driving scenarios. The corresponding output data are compared 14 with relevant ground truth data 13, wherein from the comparison 14, an improvement or degradation in the performance of the current software version of the system under test 12 is inferred.

[0036] Reference numerals list

[0037] 10 Flowchart of Continuous Integration Strategy

[0038] 11Recorded data

[0039] 12 System Under Test (SUT)

[0040] 13 Relevant ground truth data

[0041] 14 Compare the output data with the ground truth data

Claims

1. A method for a continuous integration strategy for a driver assistance system, wherein during the driving of a vehicle in real road traffic, measurement data are measured by vehicle sensors and settings of vehicle actuators are determined, and the measurement data and settings are stored in a test data set (11), wherein the test data set (11) includes a time series of input data and output data of the driver assistance system, wherein a system under test (12) is tested with the test data set (11), wherein a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system under test (12), wherein the system under test (12) is formed by an overall software and the overall software is continuously modified, wherein the following steps of data-driven verification are repeatedly performed at predetermined time intervals: · loading the current software version of the overall software forming the tested system (12) onto the server; Compiling the current software version on the server into an executable computer program; · transferring the computer program to a hardware-in-the-loop test bench; Loading the test data set (11) into the hardware-in-the-loop test bench; running the computer program on the hardware-in-the-loop test bench while supplying input data of the test data set; Recording the corresponding output differences between the output data of the current software version and the test data set; simultaneously comparing (14) the output of the current software version with ground truth data (13) to infer performance improvements or degradations of the current software version and assigning performance ratings to corresponding differences; · Form performance statistics about all differences and their performance evaluation; · Evaluating changes to the overall software based on the performance statistics; Output reports, And the overall software in which the changes are performed during the control of the real vehicle.

2. The method according to claim 1, wherein the test data set (11) is generated by at least one of the following sources: a customer vehicle participating in normal road traffic; a vehicle participating in normal road traffic for the purpose of verifying the system under test.

3. The method according to any of the preceding claims, wherein the test data set (11) is formed by at least one time series of the following data categories: sensor data, map material, traffic conditions, output of the driver assistance system.

4. The method according to any of the preceding claims, wherein the report lists errors of the tested system (12) that have newly occurred during verification iterations.

5. The method according to any of the preceding claims, wherein an open loop ADAS / ADS is tested as the system under test (12).

6. The method according to claim 5, wherein the tested system (12) is formed by one of the following open-loop driver assistance systems: traffic sign recognition, night vision, ego-motion localizer.

7. A method according to any of the preceding claims, wherein a functional decomposition scheme is used to test at least one subcomponent of a closed-loop system, wherein the closed-loop system comprises at least one closed-loop subcomponent and at least one open-loop subcomponent, and wherein the at least one open-loop subcomponent is selected as the at least one subcomponent to be tested.

8. The method according to any of the preceding claims, wherein a plurality of test stations are used in parallel by dividing the test data set (11) into a plurality of time periods and supplying the test data sets (11) of the respective time periods to the respective test stations. 9 . The method according to claim 1 , wherein, as a function of the evaluation, corresponding changes in the software version are discarded and a new iterative run is started. 10 . A test system for a continuous integration strategy for a driver assistance system, wherein the test system comprises a server and a hardware-in-the-loop test bench having a computing unit, and wherein the computing unit is configured to execute an algorithm according to the method according to any one of claims 1 to 9 .