METHOD AND SYSTEM FOR IMPROVING THE DETECTION OF REMOTELY CONTROLLED DUMMYS FOR THE DEVELOPMENT AND VALIDATION OF ADAS

Abstract

The invention relates to a method for improving the detection of remotely controlled dummies (10) for the development and validation of ADAS of a vehicle, wherein sensor data from at least one sensor device are used to identify the dummies (10) based on visual features and / or positions and / or movements, and wherein the sensor data are processed using image processing and / or sensor data fusion techniques to detect the dummies, wherein, depending on the detected dummies (10), appropriate measures for vehicle control are initiated by the ADAS, characterized in that the sensor data are linked with artificially realistic objects of the detected dummies (10) using existing video processing methods with a corresponding algorithm and the measures for vehicle control are adapted accordingly.Furthermore, a system, a computer program product, and a non-volatile, computer-readable storage medium.

Description

AREA OF INVENTION

[0001] The invention relates to the field of motor vehicles. In particular, the present invention relates to a method for improving the detection of remote-controlled dummies for the development and validation of ADAS according to claim 1. Furthermore, the present invention relates to a system, a corresponding computer program product, and a corresponding non-volatile, computer-readable storage medium. BACKGROUND INFORMATION

[0002] Safety-critical systems, such as advanced driver assistance systems (ADAS), must pass a series of validation tests before they can be released to the public. These tests are not trivial for many safety-critical systems. It is often difficult to maintain the realism of the test while simultaneously minimizing the risk to human participants.

[0003] Therefore, pedestrian collision avoidance systems (PCAMs) are not tested with real pedestrians. Instead, soft targets are often simulated using remotely controlled dummies. These dummies can be precisely controlled to create an imminent collision. They are designed to realistically mimic the movements of human pedestrians and provide similar measurements to many ADAS sensors (e.g., camera, radar, lidar, ultrasound, etc.). Overall, these dummies are essential for validating many current safety-critical systems involving pedestrians.

[0004] Unfortunately, these dummies quickly become inadequate for validating the next generation of ADAS. This is because the dummies have two major limitations: fidelity and diversity.

[0005] Algorithms for pedestrian detection and tracking are advancing rapidly, with the rate of true positives steadily increasing and the rate of false positives decreasing. Therefore, as an algorithm becomes precise enough to distinguish between a real pedestrian and a dummy, dummy-based validation may not be testing the system's true performance. This is especially true as ADAS algorithms begin to utilize finer details not provided by these dummies, such as a pedestrian's facial expression as an indicator of their intention, field of vision, and attention.

[0006] Furthermore, the tests are typically conducted with nearly identical dummy models. However, this does not reflect reality. ADAS can interact with a human population that is incredibly diverse in appearance. Unfortunately, this can lead to a system that has not been sufficiently validated for pedestrians with different visual characteristics than the test dummy. Even more concerning is that the algorithm may perform worse with pedestrians who do not resemble the dummy if the dummy was used to generate data for algorithm training. Overall, this contributes to the well-known problem of algorithmic bias in self-driving cars.

[0007] Today's remote-controlled dummies are already very expensive. Therefore, it would be unrealistic to demand dummies with higher accuracy. It would also be unrealistic to buy and test enough dummies to reflect the diversity of the human population.

[0008] The object of the invention is to provide an algorithmic solution that requires little to no physical modification to the dummies used by the validation testers.

[0009] This objective is achieved by a method having the features of claim 1 and by a system according to the invention. Advantageous embodiments and further developments are described in the dependent claims and the description. SUMMARY OF THE INVENTION

[0010] A first aspect of the invention relates to a method for improving the detection of remote-controlled dummies for the development and validation of an advanced driver assistance system (ADAS) of a vehicle, in which sensor data from at least one sensor device are used to identify the various dummies or identical ADAS with different appearances based on optical features and / or positions and / or movements. The sensor data are processed using image processing and / or sensor data fusion techniques to detect the ADAS, and depending on the detected ADAS, appropriate measures for vehicle control are initiated by the ADAS.

[0011] To achieve the objective of the invention, the sensor data are combined with artificial, realistic representations of the detected Advanced Driver Assistance Systems (ADAS) using existing video processing methods and a suitable algorithm, and the vehicle control measures are adjusted accordingly. It is therefore intended that artificial visual enhancements of remotely controlled ADAS can be used for ADAS development and validation.

[0012] This approach allows the connected downstream modules to undergo comprehensive testing to determine their ability to effectively recognize various pedestrian appearances. These include different dummies, each exhibiting distinct external characteristics such as clothing, size, posture, and other visually distinguishable features. The modules are integrated into the vehicle and connected to the driver assistance system, enabling a direct link to the data acquisition sensors. This configuration ensures efficient data transmission between the sensors and the modules, which—such as cameras, radar, and lidar—continuously collect environmental data necessary for detecting pedestrians, their appearance, positions, and movements (or the remotely controlled dummies).

[0013] The collected data is forwarded in real time to downstream modules responsible for processing and analyzing this information. Modern image processing techniques and data fusion algorithms are used to combine data from various sources into a coherent picture. In this way, the system (or the system used to implement the system) can make precise decisions regarding vehicle control based on the detected situation and the actions of pedestrians, such as adjusting speed or triggering warning signals.

[0014] This (system) architecture and the direct connection between sensors and modules enable driver assistance systems to effectively manage complex and dynamic road scenarios and significantly improve the safety and efficiency of vehicle operation through more sophisticated dummy detection. Furthermore, this method allows for the evaluation of modules regarding their ability to perceive finer details, such as the angle of a pedestrian's face. This is particularly important for the functionality of driver assistance systems that rely on interpreting the attention and gaze direction of road users in order to adjust the vehicle's behavior accordingly. These tests are therefore used for the development of algorithms employed in ADAS systems.They enable developers to optimize algorithms to recognize not only basic human behaviors and appearances, but also more complex and subtle behaviors in real time and under various environmental conditions. In other words, the proposed method for more sophisticated dummy detection for ADAS in a vehicle uses sensor data from at least one sensor device, such as cameras, radar, lidar, and / or ultrasound, to identify dummies based on visual features and / or positions and / or movements. Additional parameters or data can also be acquired from the sensor devices. The sensor data, particularly camera images, are then processed using image processing and / or sensor data fusion techniques to detect the dummies, and appropriate vehicle control measures are initiated depending on the detected dummies.The method is characterized by the fact that the sensor data, particularly the camera images, are combined with artificial, realistic representations of the detected dummies using existing video processing techniques before being fed into the ADAS system. These artificial, realistic representations can be generated, for example, by generative AI computer vision algorithms or similar techniques. Furthermore, realistic movements can be simulated, such as legs and arms moving according to the dummy's speed and size, and the platform, rails, and other testing equipment can be removed from the image to create a more realistic scene. Particular attention must be paid to ensuring that the measurements remain consistent after the substitutions have been processed.

[0015] The term "artificially realistic objects" refers, for example, to digitally created objects or elements designed to look and behave like real-world objects, but generated entirely by software. These objects are used in various fields such as virtual reality, simulations, video games, and film production. In the context of ADAS development and validation, artificially realistic objects can be used to improve the representation of pedestrians or other elements in a simulated environment.

[0016] The purpose of using these objects is to increase the realism of test scenarios without requiring physical objects, which can be costly and less versatile for testing. In ADAS, for example, artificial realistic objects enable detailed and controlled testing of how systems perceive and react to various road situations and behaviors. These objects can be programmed with specific behaviors and appearances that are not readily reproducible with real objects, thus providing a robust platform for validating the effectiveness and safety of driver assistance systems.

[0017] Furthermore, the algorithm's inputs are implementation-specific. One implementation might only use camera images as input, while another might supplement the camera image inputs with corresponding measurements from LiDAR, radar, ultrasound, etc. This method can also be applied to other test objects such as motorcycles, bicycles, children, etc.

[0018] A particularly advantageous feature is the ability to artificially enhance the fidelity and diversity of remotely controlled human dummies used in the development, testing, and validation of safety-critical automotive systems. Furthermore, it is beneficial to conduct more realistic tests of perception algorithms using test dummies. Additionally, combining simulation and real-world data to achieve the best of both worlds is advantageous, including realistic distance and material measurements from sensor recordings (e.g., LiDAR, radar, ultrasound) and simulation capabilities for camera images. Finally, it is beneficial to save money on creating multiple scenarios by using a single recording for the training and evaluation datasets of the machine learning system, with the test dummies providing the baseline reality.For example, pedestrian crossing scenarios can be modified so that the pedestrian looks towards the driver or away from the car, and the appearance of the objects can be changed accordingly.

[0019] In an advantageous embodiment of the invention, the dummy is equipped with visual markers to support the aforementioned replacement algorithms. This makes it possible to improve and more accurately represent the accuracy of the replacement algorithms by incorporating visual markers. These markers can be placed at predefined locations on the dummies, which can then be specifically identified. Alternatively, they can be placed randomly or depending on the dummy's configuration. It is particularly important that these markers are easily recognizable and detectable to enhance the system's security.

[0020] In a further advantageous embodiment of the invention, the dummy and the base are specifically colored with distinct hues to support scene segmentation and replacement algorithms. The use of different colors has the advantage of creating contrasts that are particularly easy to detect, such as green, blue, etc., to support scene segmentation and replacement algorithms (e.g., "greenscreen video systems"). This also simplifies the detection of the dummy, especially in situations where the environment may be affected by weather conditions.

[0021] In a further advantageous embodiment of the invention, the algorithm for generating data for algorithm development, machine learning, etc., is executed offline. One advantage is, for example, the increased flexibility in data generation, since offline execution of the algorithm allows for better control over the process and can be performed at a convenient time without requiring real-time processing resources.

[0022] In a further advantageous embodiment of the invention, the algorithm for generating data for unit tests, model-in-the-loop (MIL) tests, software-in-the-loop (SIL) tests, hardware-in-the-loop (HIL) tests, and vehicle-in-the-loop (VIL) tests is executed offline. This allows for better control over the process and can be performed at a convenient time without requiring real-time processing resources.

[0023] In a further advantageous embodiment of the invention, the algorithm is executed online as an ad-hoc modification for validation tests. An advantage of this embodiment is the possibility of real-time validation testing, which, for example, allows for immediate feedback and adjustments during the test procedures.

[0024] In a further advantageous embodiment of the invention, it is provided that the algorithm is applied to other measurement modalities instead of or in addition to the sensor data (camera images), which has the advantage that higher accuracy and robustness can be achieved by using multiple data sources for analysis and decision-making.

[0025] Further advantages, features, and details of the invention will become apparent from the following description of preferred embodiments and from the drawing. The features and combinations of features mentioned above in the description, as well as those mentioned in the following description of the figure and / or illustrated in the figure alone, can be used not only in the combinations specified, but also in any other combination or on their own, without departing from the scope of the invention. BRIEF DESCRIPTION OF THE DRAWING

[0026] The new features and properties of the disclosure are set forth in the accompanying claims. The accompanying drawings, which form part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. In the figures, the leftmost digit(s) of a reference number indicates the figure in which the reference number first appears. The same numbers are used in the figures to refer to identical features and components. Some embodiments of systems and / or methods in accordance with embodiments of the present subject matter are described below only by way of example and with reference to the accompanying figures.

[0027] The drawing shows in: Fig. 1 a dummy 10 that reproduces the basic shapes and colors of a real human being.

[0028] In the figure, identical elements or elements with the same function are identified by the same reference symbols. DETAILED DESCRIPTION

[0029] In this document, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Each embodiment or implementation of the subject matter described herein as "exemplary" is not necessarily to be construed as preferable or advantageous over other embodiments.

[0030] While the disclosure is open to various modifications and alternative forms, specific embodiments are illustrated by way of example in the drawing and are described in detail below. It should be understood, however, that the disclosure is not intended to be limited to these particular forms, but rather encompasses all modifications, equivalents, and alternatives that fall within the scope of the disclosure.

[0031] The terms “includes,” “contains,” or other variations thereof are intended to cover non-exclusive inclusion, so that a system, device, or process that includes a list of components or steps may contain not only those components or steps but may also contain other components or steps not expressly listed or belonging to such system, device, or process. In other words, one or more elements in a system or device preceded by the expression “includes” or “include” do not, without further limitations, preclude the presence of other elements or additional elements in the system or process.

[0032] The following detailed description of the embodiment of the disclosure refers to the accompanying drawing, which forms part of the disclosure and illustrates a specific embodiment in which the disclosure can be implemented. This embodiment is described in sufficient detail to enable the person skilled in the art to apply the disclosure, and it is understood that other embodiments may be used and modifications may be made without departing from the scope of the present disclosure. The following description is therefore not to be understood in a restrictive sense.

[0033] Fig.Figure 1 shows a dummy 10 that reproduces the basic shapes and colors of a real human. The dummy 10 has a humanoid shape with basic facial features similar to those of a real human, adapted in size and appearance. It is provided with visual markers 12 and may be colored with unique shades 14, such as green or blue, to aid in the segmentation of the scene or environment 16. Furthermore, its movements can be simulated, with the legs and arms moving according to the speed and size of the dummy 10. The environment 16 and other testing equipment may have been removed from the image to create a more realistic scene. It is envisaged that a vehicle (not shown here) detects this dummy 10 using the method according to the invention and is controlled at least partially autonomously depending on the detected dummy 10.

[0034] The method for improving the detection of remotely controlled dummies (10) for the development and validation of a vehicle's ADAS uses sensor data from at least one sensor device to identify dummies 10 based on visual features, positions, and movements. This sensor data is processed using image processing and sensor data fusion techniques to detect the dummies 10 and initiate appropriate vehicle control measures based on the detected dummies. The sensor data is integrated with artificial, realistic representations of the detected dummies 10 using existing video processing methods and a corresponding algorithm, and the vehicle control measures are adjusted accordingly. Additionally, the dummy 10 can be equipped with visual markers 12 to support the replacement algorithms.The dummy 10 and a base 11 are specially colored with unique shades 14 to support scene segmentation and replacement algorithms. The algorithm for generating data for algorithm development, machine learning training, etc., is executed offline, as well as for unit tests, model-in-the-loop (MIL) tests, software-in-the-loop (SIL) tests, hardware-in-the-loop (HIL) tests, and vehicle-in-the-loop (VIL) tests. Furthermore, the algorithm can be run online as an ad-hoc modification for validation tests and can be used instead of or in addition to sensor data, particularly camera images, on other measurement modalities.

[0035] Accordingly, the invention proposes artificial visual enhancements of remotely controlled dummies 10 for ADAS development and validation, particularly in vehicles. Reference sign 10 dummies 11 Basic 12 markers 14 unique shades

Claims

[1] Method for improving the detection of remotely controlled dummies (10) for the development and validation of ADAS of a vehicle, wherein sensor data from at least one sensor device are used to identify the dummies (10) based on visual features and / or positions and / or movements, and wherein the sensor data are processed using image processing and / or sensor data fusion techniques to detect the dummies, and wherein, depending on the detected dummies (10), appropriate measures for vehicle control are initiated by the ADAS, characterized by , that the sensor data are linked with artificially realistic objects of the detected dummies (10) using existing video processing methods with a suitable algorithm and the measures for vehicle control are adjusted accordingly. [2] Method according to claim 1, characterized by, that the dummy (10) has an outfit with visual markings (12) to support the aforementioned replacement algorithms. [3] Method according to claim 1 or 2, characterized by , that the dummy (10) and / or a base (11) for the dummies (10) are specifically colored with unique shades (14) to support scene segmentation and replacement algorithms. [4] Method according to any one of claims 1 to 3, characterized by that the algorithm for generating data for algorithm development, machine learning training, etc., is executed offline. [5] Method according to any one of the preceding claims, characterized by , that the algorithm for generating data for unit tests, model-in-the-loop (MIL) tests, software-in-the-loop (SIL) tests, hardware-in-the-loop (HIL) tests and vehicle-in-the-loop (VIL) tests is executed offline. [6] Method according to any one of the preceding claims, characterized by that the algorithm is run online as an ad-hoc modification for validation tests. [7] Method according to any one of the preceding claims, characterized by that the algorithm is applied to other measurement modalities instead of or in addition to the sensor data. [8] System for carrying out a method for improving the detection of remotely controlled dummies (10) for the development and validation of ADAS of a vehicle according to one of the preceding claims. [9] Computer program product comprising program code means for carrying out a method according to any one of claims 1 to 7. [10] Non-volatile computer-readable storage medium containing at least the computer program product according to claim 9.

Images