Method for continuous integration approach of driver assistance systems

The method addresses the limitation of ADAS/ADS testing by using real-time sensor data and hardware-in-the-loop benches for continuous, automated validation of entire ADAS/ADS software, ensuring comprehensive and efficient testing of complex scenarios.

JP2024169320AActive Publication Date: 2024-12-05DR ING H C F PORSCHE AG
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024069229
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-04-22
Publication Date
2024-12-05
Estimated Expiration
2044-04-22

AI Technical Summary

Technical Problem

Existing continuous integration approaches for driver assistance systems (ADAS/ADS) are limited to simple subcomponent validation tests and lack comprehensive testing of complex and customer-specific traffic scenarios due to their complexity and unknown nature, leading to incomplete specification-derived validation.

Method used

A method involving real-time data collection from vehicle sensors during driving, forming a test data set, and using a hardware-in-the-loop test bench to continuously test and evaluate the entire software of ADAS/ADS, including customer-specific scenarios, without human intervention.

Benefits of technology

Enables comprehensive, automated testing of ADAS/ADS in real-world conditions, identifying performance improvements or degradations, and providing continuous, data-driven validation without human engineers, thus enhancing system reliability and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024169320000001
    Figure 2024169320000001
Patent Text Reader

Abstract

To provide a method and test system for a continuous integration approach for driver assistance systems based on data-driven validations.SOLUTION: The method comprises performing data-driven validations repeatedly at predetermined time intervals, each validation including the steps of: loading a current version of the overall software onto a server; transferring a computer program to a hardware-in-the-loop test bench; loading a test data set onto the hardware-in-the-loop test bench; running the program; logging differences in output between the current software version and output data of the test data set; matching the output of the current software version with ground-truth data, thereby evaluating performance of the current software version; and forming performance statistics on all differences and their performance score; and evaluating changes in the overall software.SELECTED DRAWING: None
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method for a continuous integration approach of driver assistance systems based on data-driven validation. Furthermore, a test system for the continuous integration approach of driver assistance systems is claimed. [Background technology]

[0002] The so-called continuous integration approach describes a method from software development where the current state of a software project is compiled and automatically tested at short intervals (e.g. daily). This allows to detect and correct errors very early, while their impact on a continuously evolving software project is still small. A prerequisite for this continuous integration approach is the continuous availability of the entire compilable and testable software of a software project.

[0003] In driver assistance systems (ADAS) and automated driving systems (ADS), the continuous integration approach is currently limited to simple subcomponent validation tests, where especially simple test cases, HMI (Human Machine Interface) tests and tests related to communication with partner control devices are tested. However, many relevant test cases or traffic scenarios are not tested, because these test cases are very complex to reproduce and usually are completely unknown or lack description.

[0004] In addition, ADAS / ADS have a non-functional nature. This means that, in principle, all scenarios occurring in traffic must be correctly handled by the driver assistance system. However, many of these scenarios are so large that, on the one hand, they are not all known and, on the other hand, they cannot be described in the specifications. As a consequence, the specification-derived validation tests of the continuous integration approach are also incomplete.

[0005] DE 10 2016 220 913 A1 discloses a method for generating test cases for autonomous vehicles, whereby the test cases are generated automatically on the basis of data from vehicles involved in public road traffic.

[0006] A simulation system for simulating an autonomous vehicle is known from US Patent Application Publication No. 2021 / 0103283, in which the simulation system processes sensor data from a traffic scenario and errors and problems are automatically found using a continuous integration approach.

[0007] DE 10 2019 134 053 A1 describes an application method for the application of driver assistance systems, in which new application situations are tested with regard to their performance by means of a continuous integration environment. Summary of the Invention [Problem to be solved by the invention]

[0008] In view of this, the object of the present invention is to provide a method for a continuous integration approach to driver assistance systems, where the entire software of the driver assistance system is tested in everyday traffic situations. The test cases to be considered are not limited to simply structured test cases, but can also cover customer-specific scenarios. Furthermore, a test system is provided on which the method is performed. [Means for solving the problem]

[0009] To solve the above mentioned task, a method is provided for a continuous integration approach of a driver assistance system, where during driving of a vehicle in real traffic, measurement data are measured by vehicle sensors, vehicle actuator settings are determined, and the measurement data and settings are stored in a test data set, the test data set comprising a time series of input data and output data of the driver assistance system. The test data set is used to test a system under test, where a driver assistance system (ADAS) or an automated driving system (ADS) is selected as the system under test. The system under test is formed or represented by an entire software, which is continuously modified. At predefined time intervals, the following steps of data-driven validation: loading into a server a current software version of the entire software forming the system under test; Compiling the current software version on the server into an executable computer program (thus simultaneously testing for readability); transferring the computer program to a hardware-in-the-loop test bench (thus simultaneously testing the transfer operation, e.g. by flashing); loading the test data set into a hardware-in-the-loop test bench; running a computer program on the hardware-in-the-loop test bench while supplying input data to a test data set; logging each difference in output between the current software version and the output data of the test data set; - simultaneously matching the output of the current software version with the ground truth data, thereby inferring an improvement or degradation in performance of the current software version and assigning a performance score to each difference; forming performance statistics for all the differences and their performance scores; evaluating the overall software changes based on performance statistics; The step of outputting the report is performed iteratively. The entire modified software is then executed while controlling the actual vehicle.

[0010] The method according to the invention advantageously makes it possible to automatically verify a driver assistance system under customer-specific operation. These automatic tests can be run continuously since they do not require a human test engineer.

[0011] In one embodiment of the method according to the invention, the test data set is generated from at least one of the following sources: customer vehicles involved in normal road traffic, vehicles taking part in a test of the system under test in normal road traffic.

[0012] In a further embodiment of the method according to the invention the test data set is formed from at least one time series belonging to the following data categories: sensor data, map material, traffic conditions, output of driver assistance systems.

[0013] In a successive further embodiment of the method according to the invention, newly occurring system under test errors are enumerated by a report during a verification iteration.

[0014] In yet a further embodiment of the method according to the invention an open-loop ADAS / ADS is tested as the system under test.

[0015] In a still further subsequent embodiment of the method according to the invention, the system under test is formed by one of the following open-loop driver assistance systems: traffic sign recognition, night vision, ego-motion locator.

[0016] In another embodiment of the method according to the invention, at least one subcomponent of a closed-loop system is tested using a functional decomposition concept. In this case, the closed-loop system comprises at least one closed-loop subcomponent and at least one open-loop subcomponent, and at least one open-loop subcomponent is selected as at least one subcomponent to be tested. Thus, for example, in a distance control cruise control, all sensory components or lateral control can be tested as open-loop subcomponents. This represents a particularly advantageous embodiment of the method according to the invention, since a closed-loop system generally cannot be tested on the basis of recorded and therefore unalterable data, since any output of the closed-loop system will then affect the input data.

[0017] In yet another embodiment of the method according to the invention, multiple test benches are used in parallel by splitting the test data set into multiple periods and feeding each test bench with a test data set in each period. When using hardware-in-the-loop test benches, real-time test procedures need to be performed. By partitioning the test data set according to the invention, advantageously faster start-up can be achieved.

[0018] In another sequential embodiment of the method according to the invention, each change in the software version is discarded according to the evaluation, after which a new iterative run is started.

[0019] Further, a test system for a continuous integration approach of a driver assistance system is claimed, the test system comprising a server and a hardware-in-the-loop test bench having a computing unit, the computing unit being configured to execute an algorithm according to the method according to the invention.

[0020] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0021] It is understood that the features mentioned above and those which will be described below can not only be used in the respective specified combinations, but can also be used in other combinations or alone without departing from the scope of the invention.

[0022] In an exemplary embodiment of the method according to the invention, a continuous integration approach for a system with a functional deficiency is presented below. As an example, the system under test is formed by a traffic sign recognition software. Video data and bus signals are available from customer data or from the performed tests. Changes to the traffic sign recognition software developed throughout the day are loaded to the server every evening. These changes may include, for example, recognition algorithms, map data, legislative data sets and fusion algorithms. The current software version on the server is compiled and automatically launched on a computing unit of a hardware-in-the-loop test bench. Here, the first test cases for software integrity and flashability have already been performed. The recorded data is now fed to the hardware-in-the-loop test bench, so that the tests are launched again "virtually" with the new software. At the same time, the output of the software, i.e. the traffic signs displayed to the customer, coincides with the ground truth data. Statistics are derived from this as to whether the software changes have led to an improvement or a deterioration of the performance of the ADAS / ADS. The final report may include this statistics and additional information about new errors that arise during the data-driven validation, such as low frequency scenarios that introduce errors into the new software. [Brief description of the drawings]

[0023] [Figure 1] 1 shows a flowchart of a continuous integration approach in one embodiment of the system according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In Fig. 1 a flow chart 10 is shown for the continuous integration approach in one embodiment of the system according to the invention. Measurement data 11 measured and recorded in real traffic by vehicle sensors form the basis of a test data set of a system under test 12, implemented on a hardware-in-the-loop test bench and thus going through different driving scenarios. The respective output data are matched in comparison 14 with the associated ground truth data 13, if an improvement or deterioration in the performance of the current software version of the system under test 12 is to be inferred from the comparison 14. [Explanation of symbols]

[0025] 10. Continuous Integration Approach Flowchart 11 Recorded Data 12 System Under Test (SUT) 13 Related ground truth data 14 Matching output data with ground truth data

Claims

1. A method for a continuous integration approach of a driver assistance system, in which during driving of a vehicle in real traffic, measurement data are measured by vehicle sensors and settings of vehicle actuators are determined, said measurement data and settings are stored in a test data set (11), said test data set (11) comprising time series of input data and output data of said driver assistance system, a system under test (12) is tested with said test data set (11), a driver assistance system (ADAS) or an automated driving system (ADS) is selected as said system under test (12), said system under test (12) is formed by a whole software, said whole software being continuously modified, said data-driven validation comprising the steps of: loading a current software version of the entire software forming the system under test (12) into a server; compiling said current software version on said server into an executable computer program; transferring said computer program to a hardware-in-the-loop test bench; loading said test data set (11) into said hardware-in-the-loop test bench; running the computer program on the hardware-in-the-loop test bench while supplying the input data to the test data set; logging each difference in output between the current software version and the output data of the test data set; - simultaneously matching (14) the output of the current software version with ground truth data (13), thereby inferring improvements or degradations in performance of the current software version and assigning a performance score to each of the differences; forming performance statistics for all the differences and their performance scores; evaluating the overall software changes based on the performance statistics; and outputting a report, the data-driven validation being performed repeatedly at predetermined time intervals; The method wherein the entire modified software is executed while controlling an actual vehicle.

2. 2. The method of claim 1, wherein the test data set (11) is generated from at least one of the following sources: customer vehicles involved in normal road traffic and vehicles involved in testing the system under test in normal road traffic.

3. 2. The method according to claim 1, wherein the test data set (11) is formed by at least one time series for data categories of sensor data, map material, traffic conditions, output of the driver assistance system.

4. The method of claim 1 , wherein the report lists any new system under test (12) errors that occurred during a validation iteration.

5. The method of claim 1 , wherein an open loop ADAS / ADS is tested as the system under test (12).

6. 6. The method according to claim 5, wherein the system under test (12) is formed by one of the following open-loop driver assistance systems: traffic sign recognition, night vision, self-motion locator.

7. 2. The method of claim 1, wherein a functional decomposition concept is used to test at least one subcomponent of a closed-loop system, the closed-loop system comprising at least one closed-loop subcomponent and at least one open-loop subcomponent, and the at least one open-loop subcomponent is selected as the at least one subcomponent to be tested.

8. 2. The method of claim 1, wherein multiple test benches are used in parallel by dividing the test data set (11) into multiple periods and supplying a test data set (11) to each of the test benches in each of the periods.

9. The method of claim 1 , wherein, following said evaluation, each change in said software version is discarded and a new iterative launch is initiated.

10. A test system for a continuous integration approach of driver assistance systems, the test system comprising a server and a hardware-in-the-loop test bench having a computing unit, the computing unit being configured to execute an algorithm according to the method of any one of claims 1 to 9.

Citation Information

Patent Citations

  • Application installation control method and device

    CN105653303A

  • Software performance testing method and device, storage medium and electronic equipment

    CN114880224A

  • Software performance evaluation method and device, storage medium and computer equipment

    CN115878493A

  • Image processing device and program

    JP2019057133A

  • Enterprise performance and capacity testing

    US20200099773A1