Method and system for enhancing the detection of remote-controlled dummies for the development and validation of ADAS

Abstract

Method of enhancing detection of remote-controlled dummies 10 in the development and validation of vehicle advanced driver assistance system (ADAS) through visual augmentation of the remote-controlled

Description

FIELD OF THE INVENTION

[0001] The invention relates to the field of automobiles. More specifically, the present invention relates to a method for enhancing the detection of remote-controlled dummies for the development and validation of ADAS according to claim 1. Furthermore, the present invention relates to a system, a corresponding computer program product, and a corresponding non-transitory computer-readable storage medium. BACKGROUND INFORMATION

[0002] Safety-critical systems, like Advanced Driver Assistance Systems (ADAS), must pass a suite of validation tests before the can be publicly released. This testing is nontrivial for many safety-critical systems. It is often difficult to maintain the realism of the test while minimizing the risk to human participants.

[0003] Therefore, automotive pedestrian crash avoidance mitigation (PCAM) systems are not tested with real pedestrians. Instead, soft targets are often emulated by remote controlled dummies. These dummies can be precisely controlled to induce an imminent collision. They are designed to realistically reflect the motion of human pedestrians and yield similar measurements from many ADAS sensors (e.g. camera, RADAR, LIDAR, ultrasonic, etc.). In all, these dummies are essential for validating many current safety-critical systems that involve pedestrians.

[0004] Unfortunately, however, these dummies are quickly becoming insufficient to validate the next generation of ADAS. This is because the dummies have two major limitations: fidelity and diversity.

[0005] Pedestrian detection and tracking algorithms are rapidly advancing, steadily increasing their true positive rates and decreasing their false positive rates. Therefore, if an algorithm can become precise enough to differentiate between a real and dummy pedestrian, the dummy-based validation may fail to test the system’s true performance. This is particularly true as ADAS algorithms begin to take advantage of finer details not provided from these dummies, e.g. the gaze of a pedestrian’s face as a sign of their intent, field-of-view, and awareness.

[0006] Furthermore, tests are typically performed with almost identical dummy models. However, this does not reflect reality. The ADAS may interact with a human population that is incredibly diverse in appearance. Unfortunately, this may result in a system that was insufficiently validated for pedestrians that have different visual features than the test dummy. Even more concerning, if the dummy was used to generate data for algorithm training, the algorithm may perform worse for pedestrians that do not resemble the dummy. In all, this contributes to the well-known issue of algorithmic bias for self-driving cars.

[0007] Today’s remote-controlled dummies are already very expensive. Therefore, it would be unrealistic to demand dummies with higher fidelity. It would also be unrealistic to buy and test enough dummies to reflect the diversity of the human population.

[0008] The objective of the invention is to provide an algorithmic solution that requires little to no physical modifications to dummies used by validation testers.

[0009] This objective is achieved by means of a method with the features of claim 1 and by means of a system according to the invention. Advantageous embodiments and further developments can be inferred from the dependent claims and the description. SUMMARY OF THE INVENTION

[0010] A first aspect of the invention relates to a method for enhancing the detection of remote-controlled dummies for development and validation of an Advanced Driver Assistance System - ADAS - of a vehicle, where sensor data from at least one sensor device is used to identify the different dummies or same dummies with different appearance based on visual characteristics and / or positions and / or movements. The sensor data is processed using image processing and / or sensor data fusion techniques to detect the dummies, with corresponding measures for vehicle control through the ADAS being initiated depending on the detected dummies.

[0011] To achieve the objective of the invention, the sensor data is linked with artificial realistic representations of the detected dummies using existing video processing methods involving a corresponding algorithm, and the measures for vehicle control are adjusted accordingly. Therefore, it is intended, that artificial visual enhancements of remote-controlled dummies will be used for ADAS development and validation.

[0012] This approach allows connected downstream modules to undergo comprehensive testing to determine if they can effectively recognize various pedestrian appearances. This includes different dummies, each displaying distinct external features such as clothing, size, posture, and other visually distinguishable characteristics. The modules e.g. are integrated into the vehicle and connected to the driver assistance system, facilitating a direct link to the sensors for data acquisition. This configuration ensures efficient data transmission between the sensors and the modules, which as sensors—like cameras, radar, and Lidar—continuously collect environmental data necessary for detecting pedestrians, their appearances, positions, and movements (or the remote-controlled dummies).

[0013] The collected data are relayed in real-time to the downstream modules responsible for processing and analyzing this information. Advanced image processing techniques and data fusion algorithms are employed to merge data from various sources into a coherent image. This enables the method (or system used to implement the method) to make precise decisions regarding vehicle control, such as speed adjustments or triggering warning signals, based on the recognized situation and the actions of the pedestrians.

[0014] This (system) architecture and the direct connection between sensors and modules allow the driver assistance systems to effectively handle complex and dynamic road scenarios, significantly enhancing vehicle operation safety and efficiency by detecting the dummies in a more sophisticated method. Additionally, this method enables the evaluation of modules concerning their ability to perceive finer details, such as the gaze angle of a pedestrian's face. This is particularly important for the functionality of driver assistance systems that rely on interpreting the attention and direction of gaze of individuals in traffic to accordingly adjust the vehicle's behavior.Therefore these tests are used for developing algorithms used in ADAS systems. They enable developers to optimize algorithms not only to recognize basic human behaviors and appearances but also more complex and subtle ones in real-time and under various environmental conditions. This significantly contributes to increasing road safety and enhancing the efficiency of autonomous and semi-autonomous driving systems.In other words, the method proposed for a more sophisticated detecting of dummies for ADAS in a vehicle utilizes sensor data from at least one sensor device, such as cameras, RADAR, LIDAR, and / or ultrasonic, among others, to identify dummies based on visual characteristics and / or positions and / or movements. Additionally, other parameters or data can be captured by the sensor devices. The sensor data, particularly camera images, are then processed using image processing and / or sensor data fusion techniques to detect the dummies, with corresponding measures for vehicle control being initiated depending on the detected dummies. This method is characterized by linking the sensor data, especially the camera images, with artificial realistic representations of the detected dummies using existing video processing methods before they are fed into the ADAS system. These artificial realistic representations can be provided, for example, by Generative Al computer vision algorithms or similar techniques. Furthermore, realistic movements can be simulated, for example, with legs and arms moving according to the velocity and size of the dummy, and the platform, tracks, and other test equipment can be removed from the image to create a more realistic scene. Special care must be taken to ensure that the measurements remain internally consistent after processing replacements.

[0015] "Artificial realistic assets" refers e.g. to digitally created objects or elements that are designed to look and behave like real-world objects, but are entirely generated through software. These assets are used in various fields such as virtual reality, simulations, video games, and film production. In the context of ADAS development and validation, artificial realistic assets might be used to enhance the representation of the pedestrians or other elements in a simulated environment.

[0016] The purpose of using these assets is to enhance the realism of test scenarios without the need for physical objects, which can be costly and less versatile for testing. In ADAS, for instance, artificial realistic assets allow for detailed and controlled testing of how systems perceive and react to different road situations and behaviors. These assets can be programmed with specific behaviors and appearances that might not be easily reproducible with real objects, providing a robust platform for validating the effectiveness and safety of assistive driving technologies.

[0017] Furthermore, the inputs of the algorithm are implementation-specific. One implementation may only use the camera images as inputs, while another implementation may augment the camera image inputs with corresponding measurements from LIDAR, RADAR, ultrasonic, etc. This method can also be applied to other test targets such as motorcycles, bicycles, children, etc.

[0018] Particularly advantageous is the ability to artificially improve the fidelity and diversity of the remote-controlled human dummies used in developing, testing, and validating safety-critical automotive systems. Additionally, conducting more realistic testing of perception algorithms with test dummies is advantageous. Furthermore, it is advantageous to combine simulation and real-world data to achieve the best of both worlds, including realistic distance and material measurements from sensor recordings (e.g., LIDAR, RADAR, ultrasonic) and simulation capability for camera images. Finally, it is advantageous to save money on creating multiple scenarios by using one recording for machine learning training and evaluation datasets, with ground truth provided by the test dummies. For example, pedestrian crosswalk scenarios can be modified to have the pedestrian look at the driver or away from the car, and the appearance of the assets can be changed accordingly.

[0019] In an advantageous embodiment of the invention, it is provided that the dummy has an outfit marked with visual fiducial markers to aid the aforementioned replacement algorithms. Thus, it is possible to improve the accuracy of the replacement algorithms and depict them more precisely by incorporating visual fiducial markers. These fiducial markers can be placed at predefined areas on the dummies, which can then be specifically detected, for example. Alternatively, they can also be randomly placed on the dummy or depending on its configuration. It is particularly envisaged to detect and capture these markers exceptionally well to enhance the safety of the system.

[0020] In another advantageous embodiment of the invention, it is provided that dummy and the base are specifically colored with unique hues to support scene segmentation and replacement algorithms. The use of different colors has the advantage of creating contrasts that are particularly easy to capture, such as green, blue, etc., to aid in scene segmentation and replacement algorithms (e.g., "green screen video systems"). This also simplifies the detection of the dummy, especially in situations where the surrounding environment may be affected by weather conditions.

[0021] In another advantageous embodiment of the invention, it is provided that the algorithm for generating data for algorithm development, machine learning training, etc., is executed offline. One benefit of this is for example the increased flexibility in data generation, as executing the algorithm offline allows for greater control over the process and can be performed at a convenient time without requiring real-time processing resources.

[0022] In another advantageous embodiment of the invention, it is provided that the algorithm for generating data for unit tests, Model-in-the-Loop (MIL) tests, Software-in-the-Loop (SIL) tests, Hardware-in-the-Loop (HIL) tests, and Vehicle-in-the-Loop (VIL) tests is executed offline. This embodiments also allows for greater control over the process and can also be performed at a convenient time without requiring real-time processing resources.

[0023] In another advantageous embodiment of the invention, it is provided that the algorithm is executed online as an ad-hoc modification for validation testing. An advantage of this embodiment is a real-time validation testing capability, allowing for example for immediate feedback and adjustment during testing procedures.

[0024] In another advantageous embodiment of the invention, it is provided that the algorithm is deployed on other measurement modalities instead of or in addition to the sensor data (camera images).A benefit of this may be the potential for enhanced accuracy and robustness by leveraging multiple data sources for analysis and decisionmaking.

[0025] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawing. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figure and / or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWING

[0026] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0027] The drawing shows in:

[0028] Fig. 1 a dummy 10 that reflects a real human’s basic shapes and colors.

[0029] In the figure the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0030] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0031] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0032] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0033] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0034] Fig. 1 shows a dummy 10 that reflects a real human’s basic shapes and colors. The dummy 10 exhibits a humanoid form with basic facial features resembling those of a real person, adjusted in size and appearance. It is marked with visual fiducial markers 12 and possibly colored with unique hues 14 such as green or blue to support scene or environment 16 segmentation. Furthermore, its movements can be simulated, with legs and arms moving according to the velocity and size of the dummy 10. The environment 16 and other test equipment may have been removed from the image to create a more realistic scene. It is envisaged that a vehicle, not shown here, captures this dummy 10 using the inventive method and is at least partially autonomously controlled depending on the detected dummy 10.

[0035] The method for enhancing the detection of remote-controlled dummies (10) for the development and validation of ADAS of a vehicle involves utilizing sensor data from at least one sensor device to identify dummies 10 based on visual characteristics, positions, and movements. This sensor data undergoes processing using image processing and sensor data fusion techniques to detect the dummies 10, initiating corresponding measures for vehicle control based on the detected dummies. The sensor data is integrated with artificial realistic assets of the detected dummies 10 using existing video processing methods and a corresponding algorithm, with vehicle control measures adjusted accordingly. Additionally, the dummy 10 may have an outfit marked with visual fiducial markers 12 to aid replacement algorithms. The dummy 10 and a base 11 are specifically colored with unique hues 14 to support scene segmentation and replacement algorithms. The algorithm for generating data for algorithm development, machine learning training, etc., is executed offline, as well as for unit tests, Model-in-the-Loop (MIL) tests, Software-in-the-Loop (SIL) tests, Hardware-in-the-Loop (HIL) tests, and Vehicle-in-the-Loop (VIL) tests. Furthermore, the algorithm can be executed online as an ad-hoc modification for validation testing and deployed on other measurement modalities instead of or in addition to sensor data especially camera images.

[0036] Accordingly, the invention proposes artificial visual enhancements of remote-controlled dummies 10 for ADAS development and validation especially in vehicles. Reference signs 10 11 12 14 dummy base fiducial markers unique hues

Claims

1. Method for enhancing the detection of remote-controlled dummies (10) for the development and validation of ADAS of a vehicle, wherein sensor data from at least one sensor device is utilized to identify the dummies (10) based on visual characteristics and / or positions and / or movements, and wherein the sensor data is processed using image processing and / or sensor data fusion techniques to detect the dummies, wherein corresponding measures for vehicle control through the ADAS are initiated depending on the detected dummies (10), characterized in that the sensor data are linked with artificial realistic assets of the detected dummies (10) using existing video processing methods comprising a corresponding algorithm, and the measures for vehicle control are adjusted accordingly.

2. Method according to claim 1, characterized in that the dummy (10) has an outfit marked with visual fiducial markers (12) to aid the aforementioned replacement algorithms.

3. Method according to claim 1 or 2, characterized in that the dummy (10) and / or a base (11) for the dummy (10) are specifically colored with unique hues (14) to support scene segmentation and replacement algorithms.

4. Method according to any one of claims 1 to 3, characterized in thatthe algorithm for generating data for algorithm development, machine learning training, etc., is executed offline.

5. Method according to any one of the preceding claims, characterized in thatthe algorithm for generating data for unit tests, Model-in-the-Loop (MIL) tests, Software-in-the-Loop (SIL) tests, Hardware-in-the-Loop (HIL) tests, and Vehicle-in-the-Loop (VIL) tests is executed offline.

6. Method according to any one of the preceding claims, characterized in thatthe algorithm is executed online as an ad-hoc modification for validation testing.

7. Method according to any one of the preceding claims, characterized in thatthe algorithm is deployed on other measurement modalities instead of or in addition to the sensor data.

8. System to execute a method for enhancing the detection of remote-controlled dummies (10) for the development and validation of ADAS of a vehicle according to any of the preceding claims.

9. A computer program product comprising program code means for performing a method according to claim 1 to 7.

10. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 9.

Images