Method and system for the automated validation of functions of a driver assistance system

Abstract

Methods for the automated validation of functions of an advanced driver assistance system (ADAS system); comprehensive: - Recording (S10) of initial sensor data (270) by at least one sensor such as a camera, lidar system and / or radar system of a first sensor device (250) of a test vehicle (200) while driving along a route to test at least one function of an ADAS system (230), wherein the camera of the first sensor device (250) is a digital camera with full HD resolution; - Processing (S20) the first sensor data (270) from the ADAS system (230) and generating test data (280) using the processed first sensor data (270); - Transfer (S30) the test data (280) to an evaluation module (500) and / or to a first database (290); - Recording (S40) of second sensor data (270) by at least one sensor such as a camera, lidar system and / or radar system of a second sensor device (350) of a ground truth module (300) while driving along a route, wherein the camera of the second sensor device (350) is a digital camera with 4K resolution, wherein the sensors of the second sensor device (350) have higher performance in terms of resolution, frame rate, range and computing resources than the sensors of the first sensor device (250); - Processing (S50) the second sensor data (370) by a software application (330) and generating ground truth data (380) using the processed second sensor data (370); - Transfer (S60) the ground truth data (380) to the evaluation module (500) and / or to a second database (390); - Comparing (S70) the test data (280) with the ground truth data (380) by a software application (550) that includes comparison algorithms, wherein the comparison algorithms use convolutional neural networks (CNN) and / or histogram-based algorithms; - Generating (S80) evaluation results (580) for validating at least one function of the ADAS system (230).

Description

[0001] The invention relates to a method, a system and a computer program product for the automated validation of functions of a driver assistance system.

[0002] Modern Advanced Driver Assistance Systems (ADAS) encompass a range of technologies designed to improve vehicle safety and driving comfort. ADAS systems utilize various sensors, cameras, and algorithms to analyze the vehicle's surroundings and provide real-time assistance to the driver. To this end, the ADAS system includes a wide range of features that are continually being developed to further improve vehicle safety and driving comfort.

[0003] A typical feature is adaptive cruise control (ACC), which uses radar or laser sensors to maintain a safe following distance from the vehicle ahead. It automatically adjusts the vehicle's speed to the flow of traffic, eliminating the need for the driver to constantly adjust their speed. Another feature is lane departure warning (LDW), which uses cameras or sensors to detect unintentional lane departures. The driver is warned by visual, audible, or haptic signals and prompted to correct their lane. Another feature is automatic emergency braking (AEB), which automatically applies the brakes to avoid a collision with a vehicle or pedestrian.

[0004] Another important function is traffic sign recognition (TSR), which uses cameras and image recognition algorithms to detect and interpret traffic signs such as speed limits, stop signs, and information signs. This information can be displayed in the vehicle cockpit or on the head-up display, reminding the driver of important traffic rules. Traffic sign recognition can also activate other functions of an ADAS system. For example, when an ADAS-equipped vehicle approaches a stop sign, the traffic sign algorithm detects the sign and transmits the relevant information to the vehicle's control system. The control system can then use this information to automatically apply the brakes and bring the vehicle to a stop.Similarly, when the ADAS system detects a speed limit sign, it can adjust the vehicle's speed to ensure that the posted speed limit is observed.

[0005] The traffic sign algorithm is a machine vision algorithm for detecting and classifying traffic signs in images or video recordings. The algorithm uses a combination of image processing techniques and machine learning algorithms to recognize the shape, color, and content of traffic signs. The algorithm essentially consists of the following procedural steps: a) Preprocessing: The image or video image is first preprocessed to improve the image quality, reduce noise and increase the contrast and brightness of the image. b) Feature extraction: The algorithm extracts relevant features from the image, e.g. color, shape and texture, which can be used to distinguish different traffic sign types. c) Classification: The extracted features are fed into a machine learning algorithm, e.g., a neural network trained on a dataset of traffic sign images to detect and classify the traffic signs in the input image. d) Post-processing: After traffic signs are detected and classified, post-processing techniques can be used to remove false positives, track traffic signs over time, and improve recognition accuracy.

[0006] The traffic sign algorithm is therefore an essential component of a driver assistance system, as it is capable of recognizing a wide variety of traffic signs and triggering a response from the driver assistance system, especially warning, regulatory, and information signs. The algorithm must be accurate and robust, functioning under various lighting and weather conditions to ensure the correct functioning of the driver assistance system. A well-functioning traffic sign algorithm is therefore crucial for improving the safety and efficiency of vehicles on the road.

[0007] Validation is performed to test the performance, functionality, and reliability of an ADAS system and / or individual functions of an ADAS system. The validation of an ADAS system involves a complex process for testing and verifying its performance, functionality, and safety. The goal is to ensure that the ADAS system functions as intended, meets legal requirements, and reliably supports the driver. Each individual system component, such as adaptive cruise control, lane departure warning, or collision warning, is tested both individually and in combination. Functional tests verify whether the ADAS system operates correctly, provides precise feedback to the driver, and responds appropriately to various scenarios.An important aspect is the calibration of the sensors to check their accuracy and alignment and to ensure that they provide consistent and reliable data.

[0008] To validate the various functions of an advanced driver assistance system, such as the traffic sign algorithm, test drives are conducted regularly. Various test cases are used to ensure that the ADAS system functions correctly under various conditions. These test cases can be simulated or performed in real-world environments.

[0009] In the example of the traffic sign algorithm, a test driver evaluates it from the customer's perspective. Based on the test driver's evaluation, a statistical analysis is carried out which allows conclusions to be drawn about the performance and the type of errors. For example, it is checked whether a displayed traffic sign, which was detected via the vehicle camera or retrieved from map data, is actually valid. This data analysis is carried out manually and is therefore time- and labor-intensive. This is due to both the volume and complexity of data. ADAS systems generate a large amount of data during the test drive, including sensor measurements, camera images, vehicle parameters and driver inputs. They also record complex and multidimensional data, including measurements from various imaging sensors such as radar, lidar and cameras, which differ in their resolution and recording quality.In addition, human errors can occur during manual evaluation by overlooking critical details or misinterpreting information, leading to inaccuracies and inconsistencies in the evaluation process.

[0010] US 2022 / 90995 A1 discloses a system for validating road object data using ground truth data, wherein the ground truth data is determined from information from sensor systems of different vehicles.

[0011] US 2022 / 048533 A1 discloses a method and system for validating automatic control software for an autonomously driving vehicle, wherein the validation is performed using ground truth data provided in real time by other vehicles participating in traffic.

[0012] DE 10 2020 200 650 A1 discloses a method for operating an automated lane guidance system for a motor vehicle to guide the motor vehicle in a lane. The automated lane guidance is performed taking into account image data acquired by a first sensor system. An evaluation of an implementation of the automated lane guidance is performed by a second sensor system, and the determined sensor data from the second sensor system are used to assess whether the lane guidance is being executed correctly.

[0013] DE 10 2021 124 913 A1 discloses a method for assessing the performance of subsystems of a system, wherein the system is a software stack of an autonomous vehicle (AV). The method comprises perturbing a first attribute of a first subsystem and determining a change in a second attribute of a second subsystem of the system, wherein the second attribute is responsive to the perturbation of the first attribute and at least one output of the first subsystem is transmitted to the second subsystem, and determining a value for a performance metric of the system based on a correlation between the first and second attributes, wherein the performance metric is an objective function output that measures a quality of the AV's drivability.

[0014] DE 10 2019 115 809 A1 discloses a method for controlling an autonomous vehicle. A preprocessor processes image data acquired by sensors at a current time step to generate preprocessed image data, which is concatenated with additional inputs to produce a dynamic scene output. A convolutional neural network (CNN) processes the dynamic scene output to generate a spatial context feature vector used to calculate control commands (e.g., steering angle, acceleration rate, and / or braking rate control command).

[0015] The invention is based on the object of demonstrating possibilities how the validation of an ADAS system can be carried out automatically in order to significantly reduce the time required for manual evaluation and to increase the reliability and accuracy of the validation and thus of the ADAS system.

[0016] This object is achieved according to the invention with respect to a method by the features of patent claim 1, with respect to a system by the features of patent claim 4, with respect to an ADAS system by the features of patent claim 5, and with respect to a computer program product by the features of patent claim 6. The further claims relate to preferred embodiments of the invention.

[0017] The present invention enables accurate and reliable validation of a modern advanced driver assistance system (ADAS) through the automated comparison of ground-truth data with ADAS system test data using suitable comparison algorithms. Accurate validation ensures that the ADAS system functions as intended, thereby increasing overall road safety. Furthermore, the performance of the ADAS system can be evaluated. By comparing the test data with the ground-truth data, accuracy, reliability, and other performance indicators can be measured. This evaluation helps identify opportunities for improvement and fine-tune the ADAS system for optimal performance.

[0018] According to a first aspect, the invention provides a method for the automated validation of functions of a driver assistance system (ADAS system). The method comprises the following steps: - Recording first sensor data by at least one sensor such as a camera, a lidar system and / or a radar system of a first sensor device of a test vehicle while driving along a route for testing at least one function of an ADAS system, wherein the camera of the first sensor device is a digital camera with full HD resolution; - processing the first sensor data from the ADAS system and generating test data using the processed first sensor data; - Transferring the test data to an evaluation module and / or to a first database; - Recording second sensor data by at least one sensor such as a camera, a lidar system and / or a radar system of a second sensor device of a ground truth module when driving along a route, wherein the camera of the second sensor device is a digital camera with 4K resolution, wherein the sensors of the second sensor device have a higher performance in terms of resolution, frame rate, range and computing resources than the sensors of the first sensor device; - processing the second sensor data by a software application and generating ground truth data using the processed second sensor data; - Transferring the ground truth data to the evaluation module and / or to a second database; - comparing the test data with the ground truth data by a software application, wherein the software application comprises comparison algorithms, wherein the comparison algorithms use convolutional neural networks (CNN) and / or histogram-based algorithms; - Generating evaluation results to validate at least one function of the ADAS system.

[0019] In a further development, one function of the ADAS system is planned to be a traffic sign algorithm.

[0020] Advantageously, the first database and / or the second database and / or the evaluation module is / are integrated in a cloud computing infrastructure.

[0021] According to a second aspect, the invention provides a system for the automated validation of functions of a driver assistance system (ADAS system).

[0022] The system comprises a test vehicle with an ADAS system and a first sensor device with at least one sensor such as a camera, a lidar system, and / or a radar system, wherein the camera of the first sensor device is a digital camera with full HD resolution, a ground truth module with a software application and a second sensor device, and an evaluation module with a software application. The first sensor device is configured to record first sensor data as the test vehicle travels a route to test at least one function of an ADAS system. The ADAS system is configured to process the first sensor data to generate test data and to transmit the test data to the evaluation module.The second sensor device is configured to record second sensor data using at least one sensor, such as a camera, a lidar system, and / or a radar system, while traveling along a route. The camera of the second sensor device is a digital camera with 4K resolution, and the sensors of the second sensor device have higher performance in terms of resolution, frame rate, range, and computing resources than the sensors of the first sensor device. The software application is configured to process the second sensor data to generate ground truth data and to transmit the ground truth data to the evaluation module.The software application comprises comparison algorithms and is designed to compare the test data with the ground truth data and to generate evaluation results for validating the at least one function of the ADAS system, and wherein the comparison algorithms use convolutional neural networks (CNN) and / or histogram-based algorithms.

[0023] According to a third aspect, the invention provides an ADAS system for a motor vehicle which uses the method according to the first aspect for recognizing traffic signs comprising a main traffic sign and an additional sign.

[0024] According to a fourth aspect, the invention provides a computer program product comprising executable program code configured to carry out the method according to the first aspect when executed.

[0025] The invention is explained in more detail below with reference to embodiments shown in the drawing.

[0026] It shows: Fig. 1 is a block diagram illustrating an embodiment of a system according to the invention; Fig. 2 a flow chart to explain the individual method steps of a method according to the invention; Fig. 3 is a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

[0027] Further features, aspects and advantages of the invention and its embodiments are explained in the following description in conjunction with the patent claims.

[0028] Fig. 1 shows a system 100 according to the invention for the automated validation of a driver assistance system (ADAS system). The system 100 according to the invention comprises a test vehicle 200, a first database 290, a ground truth module 300, a second database 390, an evaluation module 500, and an output module 700. The test vehicle 100 comprises an ADAS system 230 with at least one driving function, such as a traffic sign algorithm, and a first sensor device 250. The ground truth module 300 comprises a high-resolution second sensor device 350 for generating ground truth data 380.

[0029] In the context of the invention, the term "ground truth" refers to the objectively known and correct values ​​or images associated with a specific data set. The ground truth data 380 represents accurate and reliable information with which the results of the ADAS system 230 are compared and evaluated when the test vehicle 100 is driving on a test track.

[0030] The ground truth data 380 serves as a reference or benchmark to evaluate the performance and accuracy of an ADAS system under test (SUT). For a traffic sign algorithm that must perform various image classification tasks, the ground truth data 380 provides the correct class or category for each image in a dataset and thus also predicts the correct traffic sign. The detection of traffic signs by the ADAS system 230 while driving on a test track can be compared with the ground truth data 380 to measure the accuracy and precision of the traffic sign detection or other performance indicators of the traffic sign algorithm.

[0031] Within the scope of the present invention, the ground truth data 380 is generated in particular by the high-resolution second sensor device 350. However, it is also possible for the ground truth data 380 to be obtained from other relevant sources.

[0032] The ADAS system 230, the first sensor device 250, the first database 290, the ground truth module 300, the second sensor device 300, the second database 390, the evaluation module 500, and the output module 700 can each be equipped with a processor and / or a memory unit. However, they can also be integrated into a common computing unit, in particular in the test vehicle 200.

[0033] In the context of the invention, a "module" can be understood, for example, as a processor and / or a processor unit and / or a memory unit for storing program instructions. The processor is specifically configured to execute the program instructions in order to carry out or implement the method according to the invention or a step of the method according to the invention. In particular, a module can be integrated into a cloud computing infrastructure. However, it can also be provided that a computing unit, such as a processor, comprises several modules.

[0034] In the context of the invention, a "processor" can be understood, for example, as a machine or an electronic circuit. A processor can be, in particular, a central processing unit (CPU), a microprocessor, or a microcontroller, e.g., an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc. A processor can also be a virtualized processor, a virtual machine, or a soft CPU.

[0035] In the context of the invention, a "storage unit" or "storage module" can be understood as, for example, a volatile memory in the form of random access memory (RAM), a permanent memory such as a hard drive or a data storage device, or, for example, a replaceable memory module. The memory module can also be designed as a cloud storage solution.

[0036] In the context of the invention, “data” refers to both raw data and already processed data from measurement results from sensors such as cameras and from other data sources.

[0037] In particular, the first database 290, the second database 390, and the evaluation module 500 can be integrated into a cloud computing infrastructure. Likewise, components of the ground truth module 300, such as a computing unit for generating the ground truth data 380, can be housed in the cloud computing infrastructure. This allows for a fast calculation result, since cloud-based solutions offer the advantage of high and thus fast computing power.

[0038] The term cloud computing infrastructure refers to the entire set of hardware, software, network, and storage resources used to support cloud computing services. Cloud computing infrastructure provides a platform for delivering a wide range of services, including data storage and management, application development and deployment, and virtual computing resources. It includes services for providing computing resources such as processing power and storage to support cloud-based applications. Security technologies such as firewalls, encryption, and identity and access management tools are used to protect data and applications from unauthorized access. Cloud computing infrastructure can significantly increase computing power and data processing speed.

[0039] Furthermore, communication connections are provided for the exchange and transmission of data between the individual modules, which are designed in particular as a wireless communication connection, e.g., a mobile radio connection, and / or as a near-field communication connection, e.g., Bluetooth®, Ethernet, NFC (near field communication), or Wi-Fi®. In particular, 5G or 6G radio modules are provided for the communication connection between the vehicle 200 and a cloud computing infrastructure. This also applies to the communication connection between the ground truth module 300 and a cloud computing infrastructure. To protect the connection to the cloud computing infrastructure via a mobile radio connection, cryptographic encryption methods can also be provided.

[0040] The test vehicle 200 includes the first sensor device 250, which provides data for the ADAS system 230. The first sensor device 250 includes one or more cameras arranged in the vehicle 200. These cameras typically use different imaging technologies and are arranged at various locations on the vehicle to provide a comprehensive view of the surroundings. The most common type of camera used in advanced driver assistance systems is a forward-facing camera, which is mounted on the front of the vehicle and detects and records other vehicles, pedestrians, and obstacles in the vehicle's path. It is often used for functions such as lane departure warning, automatic emergency braking, and adaptive cruise control. There is also a rear-facing camera, typically mounted on the rear of the vehicle, which records the area behind the vehicle, including other vehicles, pedestrians, and obstacles.It is often used for functions such as rearview cameras and rear cross-traffic alert. Side-view cameras can also be used. These cameras are usually mounted on the sides of the vehicle and capture areas that the driver cannot see through the mirrors. They are often used for functions such as hidden area detection and lane change assistance. Surround-view cameras can also be used. These cameras are usually mounted on multiple sides of the vehicle and provide a 360-degree view of the vehicle's surroundings. They are often used for functions such as parking assistance and collision avoidance.

[0041] The cameras of the first sensor device 250 are typically implemented as RGB cameras. An RGB camera is a digital camera that captures color information in the visible range of the electromagnetic spectrum and divides it into three channels: red, green, and blue (RGB). These three channels are then combined to create the entire color spectrum of the image. The cameras operate in the visible spectral range, which encompasses the wavelength range visible to the human eye (approximately 400 to 700 nanometers). However, some cameras can also operate in the infrared or ultraviolet spectrum, which can be useful for applications such as night vision or the detection of certain types of hazards. IR cameras use infrared imaging technology to capture the environment in low or no-light conditions, such as at night or in tunnels. They are often used for functions such as night vision and pedestrian detection.

[0042] Furthermore, the first sensor equipment 250 can include a lidar system and / or a radar system. A lidar system (Light Detection and Ranging) is a remote sensing technology that measures distances and creates high-resolution 3D maps of the surroundings. A lidar system typically consists of a laser transmitter that emits laser beams, usually in the infrared spectrum. These laser beams are directed at the vehicle's surroundings. A scanner, e.g., a rotating mirror, directs the laser beams from the laser transmitter in different directions. This allows the lidar system to cover a large field of view. The laser beams reflected by the objects in the surroundings are received by a photodetector. The signals received by the photodetector are processed by a special signal processing unit.It analyzes the time-of-flight information, the intensity of the reflected laser beams, and other data to create a detailed 3D map of the environment, including the positions and shapes of objects. Lidar systems provide high-resolution and accurate 3D maps and enable precise detection, localization, and tracking of objects. They are effective at detecting and identifying objects such as vehicles, pedestrians, and obstacles under various driving conditions.

[0043] A radar system (radio detection and ranging, or radar) is a sensing technology that uses radio waves to detect and locate objects. A radar system has an antenna for transmitting and receiving radio waves. It sends the radio waves from the radar transmitter and receives the waves returned by objects in the surrounding area. A signal processor analyzes the received signals and extracts relevant information such as time delay, frequency shift, and the intensity of the reflected waves. By analyzing the frequency shift of the received waves (Doppler effect) and the time delay, the radar system can estimate the relative speed and distance of objects. This information helps determine the presence, location, and movement of objects such as vehicles or pedestrians. Radar systems provide accurate distance measurements and reliable object detection.They are particularly useful in adverse weather conditions that can limit visibility, as radio waves are less affected by factors such as rain, fog, or dust than other sensor technologies.

[0044] Cameras, lidar systems, and radar systems all have their strengths and limitations, and the combination of information from multiple sensing modalities is used in ADAS systems to improve overall perception and increase the accuracy and reliability of object detection and tracking. Furthermore, other sensor types, such as ultrasonic sensors, can be used within the scope of the invention.

[0045] The second sensor system 350 essentially contains the same sensor types, such as cameras and / or a lidar system and / or a radar system. However, the resolution of the individual sensors is significantly higher, and the number of sensors can also be larger.

[0046] A camera's resolution refers to the number of pixels that make up the image captured by the camera. A camera with a higher resolution captures more detailed and sharper images than one with a lower resolution. A camera's resolution is usually expressed by the number of horizontal and vertical pixels in the image, for example, 1920 x 1080 (also known as Full HD) or 3840 x 2160 (also known as 4K). Full HD cameras are typically used for the first sensor device 250 because the processing power and storage capacity required for image processing from a 4K camera in the vehicle are too large.For the second sensor device 350 of the ground truth module 300, however, 4K cameras are used because a significantly higher resolution is desired here and the ground truth module 300 has greater computing resources, either directly through corresponding processors or through integration into a cloud computing infrastructure.

[0047] The frame rate of a vehicle camera for an ADAS system 230 refers to the rate at which the camera captures and stores images per second. This parameter is also referred to as frame rate or frames per second (FPS) and is an important criterion when selecting a camera for ADAS applications. A higher frame rate means the camera can capture and store more images per second, enabling a smoother and more detailed representation of the surroundings. Again, the cameras of the second sensor device 350 have a higher frame rate than the cameras of the first sensor device 250. Typical frame rates for cameras of the first sensor device 250 for ADAS applications are between 30 and 60 frames per second, while the more powerful cameras of the second sensor device 350 can capture up to 120 frames per second.Thanks to the higher frame rate, the cameras of the second sensor device 350 can take factors such as vehicle speed into account much better, thus depicting the complexity of the environment more accurately and with higher resolution.

[0048] Higher frame rates increase the demands on storing and processing the captured image data. As already mentioned, the ground truth module 300 has such computing resources, whereas a conventional ADAS system 230 is not equipped with such computing resources, as this would significantly increase the overall cost and complexity of the ADAS system 230. When selecting a camera for a vehicle-based ADAS system 230, a balance must therefore be found between image capture frequency, image quality, and system requirements to ensure optimal performance and efficiency. However, for the validation of the ADAS system 230, this means that the design of the camera system installed in the vehicle is not sufficient to make a satisfactory assessment of the reliability of the ADAS system 230 based solely on the data provided by the first sensor device 250.

[0049] These considerations apply equally to lidar and radar systems. The main differences between high-resolution lidar and radar systems and low-resolution systems are the level of detail and range. For example, high-resolution lidar systems capture a more detailed image of the surroundings. They emit a larger number of laser beams and have a higher point density in the resulting point cloud, enabling more detailed object detection, e.g., distinguishing small objects or detecting complex object shapes. In contrast, low-resolution lidar systems have fewer laser beams and produce point clouds with fewer data points, resulting in a lower level of detail. High-resolution lidar systems therefore generally offer higher accuracy in ranging and object localization.They can measure shorter distances and detect objects with greater accuracy, resulting in a more accurate representation of the surrounding environment. Lower-resolution lidar systems may have lower accuracy due to longer measurement intervals and limited point density. However, the range of high-resolution lidar systems may be shorter compared to lower-resolution systems. While they are capable of capturing detailed information within a limited range, their ability to detect objects at greater distances may be compromised. Low-resolution lidar systems, on the other hand, can cover a larger area but with lower resolution and detail.

[0050] Radar systems generally have lower resolution than lidar systems. However, high-resolution radar systems use advanced signal processing techniques and more sophisticated antenna arrays to improve resolution. They can provide more detailed information about objects, such as their size, shape, and movement patterns. Lower-resolution radar systems have fewer antenna elements and cannot capture finer details as well. High-resolution radar systems therefore offer greater accuracy in detecting and tracking objects. They can provide accurate measurements of relative speeds, angles, and positions, thus enabling a better estimation of objects' trajectories. Lower-resolution radar systems may have lower accuracy and are subject to certain limitations, such as lower accuracy in speed measurement.In terms of range, high-resolution radar systems can maintain better resolution and accuracy over longer distances, while lower-resolution systems may have poorer detection and tracking performance.

[0051] High-resolution lidar and radar systems offer finer detail, higher accuracy, and better localization capabilities compared to low-resolution systems. However, low-resolution systems often offer greater coverage and a wider detection range, making them suitable for certain applications where high detail is not strictly necessary or feasible. The choice between high-resolution and low-resolution systems depends on the specific requirements of the ADAS application and the trade-off between resolution, accuracy, range, and cost.

[0052] To test the functions of the ADAS system 230, the first sensor device 250 records first sensor data 270 from the surroundings of the vehicle 200 while driving along a test track with a variety of traffic scenarios and sends it to the ADAS system 230, which reacts to this first sensor data 270 and generates test data 280 therefrom. As described, the first sensor device 230 comprises at least one camera and / or a lidar system and / or a radar system and / or another sensor for perception and data acquisition. The first sensor data 270 can include images, point clouds, sensor measurements, or other relevant information. The recorded sensor data 270 is transmitted or sent to the ADAS system 230. The first sensor data 270 is typically transmitted via a wired or wireless connection so that the ADAS system 230 can access and analyze the information.The algorithms and processing components of the ADAS system 230 analyze the received first sensor data 270 to interpret the environment, identify objects, and make decisions based on the sensor information. Based on the analysis of the first sensor data 270, the ADAS system 230 generates test data 280. This test data 280 may include outputs such as detected objects, calculated distances, vehicle trajectories, lane information, or other relevant information that the ADAS system 230 provides as part of its functionality. The generated test data 280 serves as the result or response of the ADAS system 230 and reflects its behavior and performance in a specific traffic scenario.

[0053] The algorithms of the ADAS system 230 include, in particular, extraction and classification algorithms for extracting and classifying essential features from the first sensor data 270 acquired by the first sensor device 250. The extraction and classification algorithms particularly utilize artificial intelligence algorithms such as neural networks.

[0054] A neural network consists of neurons arranged in several layers and connected to one another in various ways. A neuron is able to receive information at its input from outside or from another neuron, evaluate this information in a specific way and pass it on in a modified form at the neuron's output to another neuron or output it as the final result. The hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, there can be several layers of hidden neurons. They ensure that the information is forwarded and processed. Output neurons ultimately provide a result and pass it on to the outside world. The different arrangement and interconnection of the neurons give rise to different types of neural networks, such as a feedforward network (FNN), a backward network (FNN), and a multi-level network (MNN).Recurrent Network (RNN) or Convolutional Neural Network (CNN). These networks can be trained using unsupervised or supervised learning.

[0055] The extraction algorithm detects, for example, whether the first recorded sensor data 270 contains a traffic sign or other information. From this extracted information, test data 280 is created, which is transmitted according to the invention to the evaluation module 500. However, the test data 280 can also be stored in the database 290 to be available for evaluation at a later time.

[0056] The second sensor device 350 also detects the surroundings of the vehicle 200. In particular, it is provided that the ground truth module 300 or the second sensor device 350 is also installed in the vehicle 200, so that the surroundings detected by the first sensor device 250 and the second sensor device 350 are identical in each case. Since the sensors of the second sensor device 350 have a significantly higher resolution than the first sensor device 250, the surroundings are detected with significantly greater accuracy. The second sensor data 370 recorded by the second sensor device 350 are further processed by a software application 330 to create ground truth data 380.The software application 330 comprises a variety of perception algorithms, such as image processing algorithms, extraction algorithms, and classification algorithms, which in particular use artificial intelligence algorithms, such as neural networks. Since the recorded second sensor data 370 is significantly more complex than the first sensor data 270, the neural networks used also have a higher complexity in order to be able to process this second sensor data 370. In particular, these neural networks are trained with correspondingly high-resolution data sets. In addition to the recorded second sensor data 370

[0057] 0, additional data sources such as map data, historical data, weather data, traffic data, etc. can be used. This data is stored in particular in database 390. In addition, target specifications, key figures, etc. can be taken into account.

[0058] In particular, the map data contains various types of information that cannot be collected by vehicle sensors alone. This data is collected from various sources, such as GPS satellites, road markings, and other sources. The data typically includes information about road geometry such as curvature, slope, and width, as well as the location of intersections, traffic lights, and pedestrian crossings. In addition, the map data may contain information about traffic conditions, such as traffic flow, collected in real time from various sources, such as roadside sensors and GPS-enabled devices used by other vehicles.

[0059] The ground truth data 380 is then transmitted to the evaluation module 500. The evaluation module 500 has a software application 550 that contains comparison algorithms, which are based in particular on artificial intelligence algorithms. Comparison algorithms play a crucial role in image analysis because they enable the quantification of the similarity or dissimilarity between images. According to the invention, a variety of comparison algorithms can be used, whereby the selection of the suitable algorithm depends on the type of data, the desired properties of the comparison, and the specific question regarding the performance of the ADAS system 230.

[0060] An example of comparison algorithms are histogram-based algorithms that compare the intensity distributions of pixels in two images. These include histogram intersection, histogram correlation, and histogram intersection. Histogram comparison methods are useful for tasks such as image matching, object detection, and content-based image search.

[0061] Deep learning-based approaches, such as convolutional neural networks (CNNs), extract features from images and enable comparison based on learned data representations. The images are inserted into a feature space where similarity can be calculated using metrics such as cosine similarity or Euclidean distance.

[0062] The software application 550 now compares the test data 280 with the ground truth data 380. By comparing the test data 280 with the ground truth data 380, it can be determined whether the test data 280 exhibits the required accuracy and reliability with regard to the performance of the ADAS system 230. By comparing the highly accurate ground truth data 380 with the vehicle-based test data 280, the accuracy and reliability of the ADAS system 230 can now be verified. This allows the algorithms of the ADAS system 230 to be automatically validated with regard to their ability to detect objects in the environment of the vehicle 200 and to respond to them appropriately and safely.

[0063] According to the invention, the precise ground truth data 380 thus serves to evaluate and validate the performance and reliability of the ADAS system 230 and to identify areas requiring improvement. Since the ground truth data 380 represents the reference point for evaluating the performance of the ADAS system 230, artificial intelligence-based comparison algorithms can be used, which enable automated analysis of the test data 280 and thus also automated validation of the ADAS system 230.

[0064] According to the invention, the evaluation of the test data 280 can be carried out by comparison with the ground truth data 380 both in real time while the test vehicle 200 is driving along a route and at a later time using the recorded test data 280 and the stored ground truth data 380. For a real-time evaluation, the use of a cloud computing infrastructure and the use of 5G or 6G radio modules for mobile data transmission are particularly suitable.

[0065] Evaluation results 580 are created from the comparison. These evaluation results 580 can be forwarded to an output module 700, which, using a software application 750, creates output data 780 from the evaluation results 580, particularly in the form of reports, histograms, diagrams, text messages, software codes, PowerPoint presentations, etc., which can be used for validating the ADAS system 230.

[0066] In Fig. 2 shows the procedural steps for the automated validation of functions of a driver assistance system (ADAS system).

[0067] In a step S10, first sensor data 270 are recorded by at least one sensor such as a camera, a lidar system and / or a radar system of a first sensor device 250 of a test vehicle 200 while driving along a route for testing at least one function of an ADAS system 230.

[0068] In a step S20, the first sensor data 270 are processed by the ADAS system 230 and test data 280 are generated using the processed first sensor data 270.

[0069] In a step S30, the test data 280 are transmitted to an evaluation module 500 and / or to a first database 290.

[0070] In a step S40, second sensor data 270 are recorded by at least one sensor such as a camera, a lidar system and / or a radar system of a second sensor device 350 of a ground truth module 300 while traveling along a route, wherein the sensors of the second sensor device 350 have a higher performance in terms of resolution, frame rate, range, and computing resources than the sensors of the first sensor device 250.

[0071] In a step S50, the second sensor data 370 are processed by a software application 330 and ground truth data 380 are generated using the processed second sensor data 370.

[0072] In a step S60, the ground truth data 380 are transmitted to the evaluation module 500 and / or to a second database 390.

[0073] In a step S70, the test data 280 are compared with the ground truth data 380 by a software application 550, wherein the software application 550 comprises comparison algorithms.

[0074] In a step S80, evaluation results 580 are generated to validate the at least one function of the ADAS system 230.

[0075] Fig. 3 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention.

[0076] The present invention enables accurate and reliable validation of a modern advanced driver assistance system (ADAS) through the automated comparison of ground-truth data with ADAS system test data using suitable comparison algorithms. Accurate validation ensures that the ADAS system functions as intended, thereby increasing overall road safety. Furthermore, the performance of the ADAS system can be evaluated. By comparing the test data with the ground-truth data, accuracy, reliability, and other performance indicators can be measured. This evaluation helps identify opportunities for improvement and fine-tune the ADAS system for optimal performance. Reference symbol 100 systems 200 test vehicles 230 ADAS system 250 first sensor device 270 first sensor data 280 test data 290 first database 300 Ground Truth Module 330 Software application 350 second sensor device 370 second sensor data 380 ground truth data 390 second database 500 evaluation module 550 software application 580 evaluation results 700 output module 750 software applications 780 output data 900 Computer program product 950 program code

Claims

[1] Method for the automated validation of functions of a driver assistance system (ADAS system); comprising: - recording (S10) first sensor data (270) by at least one sensor such as a camera, a lidar system and / or a radar system of a first sensor device (250) of a test vehicle (200) when driving along a route for testing at least one function of an ADAS system (230), wherein the camera of the first sensor device (250) is a digital camera with full HD resolution; - processing (S20) the first sensor data (270) from the ADAS system (230) and generating test data (280) using the processed first sensor data (270); - transmitting (S30) the test data (280) to an evaluation module (500) and / or to a first database (290); - recording (S40) second sensor data (270) by at least one sensor such as a camera, a lidar system and / or a radar system of a second sensor device (350) of a ground truth module (300) when driving along a route, wherein the camera of the second sensor device (350) is a digital camera with 4K resolution, wherein the sensors of the second sensor device (350) have a higher performance in terms of resolution, frame rate, range and computing resources than the sensors of the first sensor device (250); - processing (S50) the second sensor data (370) by a software application (330) and generating ground truth data (380) using the processed second sensor data (370); - transmitting (S60) the ground truth data (380) to the evaluation module (500) and / or to a second database (390); - comparing (S70) the test data (280) with the ground truth data (380) by a software application (550) comprising comparison algorithms, wherein the comparison algorithms use convolutional neural networks (CNN) and / or histogram-based algorithms; - generating (S80) evaluation results (580) for validating the at least one function of the ADAS system (230). [2] The method of claim 1, wherein a function of the ADAS system (230) is a traffic sign algorithm. [3] Method according to claim 1 or 2, wherein the first database (290) and / or the second database (390) and / or the evaluation module (500) is / are integrated in a cloud computing infrastructure. [4] A system (100) for the automated validation of functions of a driver assistance system (ADAS system), comprising a test vehicle (200) with an ADAS system (230) and a first sensor device (230) with at least one sensor such as a camera, a lidar system and / or a radar system, wherein the camera of the first sensor device (250) is a digital camera with full HD resolution, a ground truth module (300) with a software application (330) and a second sensor device (350), and an evaluation module (500) with a software application (550); wherein the first sensor device (250) is designed to record first sensor data (270) when the test vehicle (200) travels a route for testing at least one function of an ADAS system (230); wherein the ADAS system (230) is configured to process the first sensor data (270) to generate test data (280) and to transmit the test data (280) to the evaluation module (500);wherein the second sensor device (350) is designed to record second sensor data (270) by at least one sensor such as a camera, a lidar system and / or a radar system when driving along a route, wherein the camera of the second sensor device (350) is a digital camera with 4K resolution, wherein the sensors of the second sensor device (350) have a higher performance in terms of resolution, frame rate, range, and computing resources than the sensors of the first sensor device (250); wherein the software application is designed to process the second sensor data (370) to generate ground truth data (380) and to transmit the ground truth data (380) to the evaluation module (500);wherein the software application (550) comprises comparison algorithms and is designed to compare the test data (280) with the ground truth data (380) and to generate evaluation results (580) for validating the at least one function of the ADAS system (230), and wherein the comparison algorithms use convolutional neural networks (CNN) and / or histogram-based algorithms; [5] ADAS system (230) for a motor vehicle, characterized by that the method according to one of claims 1 to 3 is used to validate at least one function of the ADAS system (230). [6] A computer program product (900) comprising an executable program code (950) configured to carry out the method according to any one of claims 1 to 3 when executed.

Images