Method and system for improving the recognition of road users for an ADAS / ADS system

Abstract

The invention relates to a method for improving the detection of road users who are outside the detection range of a sensor device of a vehicle for controlling an ADAS / ADS system, comprising the following steps: - detecting sensor data (220) of a traffic scene and surroundings of a vehicle (10) with an ADAS / ADS system (750) by sensors of a sensor device (220) of the vehicle (10); - Preprocessing the sensor data (220) and / or the fused sensor data (250) by an input layer (430) of an AI model (410) of the evaluation module (400), in particular for noise reduction and generating preprocessed sensor data (435); - Processing the preprocessed sensor data (435) by a first task layer (440) with an algorithm for detecting and tracking other vehicles, a second task layer (450) with an algorithm for traffic sign recognition, a third task layer (460) with an algorithm for predicting the trajectories of other vehicles, and a fourth task layer (470) with an algorithm for detecting road users such as pedestrians or animals to create processed input data (510, 520, 530, 540) for a decision model (500) - Merging processed input data (510, 520, 530, 540) into a fused context.

Description

[0001] The invention relates to a method, a system, an ADAS / ADS system and a computer program product for improving the detection of road users who are outside the detection range of a sensor device of a vehicle for controlling a driver assistance system (ADAS) and / or an automated driving system (ADS).

[0002] Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) are two distinct types of systems used in modern vehicles to enhance safety and driving comfort. The main difference is that driver assistance functions support the driver while maintaining control of the vehicle, while automated driving systems are capable of completely taking over the driving task under certain conditions, thus removing the driver's responsibility for driving the vehicle.

[0003] ADAS / ADS systems incorporate a range of technologies such as various sensors, cameras, and algorithms to analyze the vehicle's surroundings and provide real-time driver assistance. To this end, the ADAS / ADS system includes a variety of features that are continually being developed to further improve vehicle safety and driving comfort.

[0004] A typical function is adaptive cruise control (ACC), which uses radar or laser sensors to maintain a safe distance from the vehicle in front. It automatically adjusts the vehicle's speed to the flow of traffic, eliminating the need for the driver to constantly adjust their speed. Another function is lane departure warning (LDW), which uses cameras or sensors to detect unintentional lane departures. The driver is warned by visual, audible, or haptic signals and prompted to correct their lane. Another function is automatic emergency braking (AEB), which automatically applies the brakes to avoid a collision with a vehicle or pedestrian.

[0005] In a typical urban traffic scene, a vehicle equipped with an ADAS / ADS system, at least partially automated, moves along a multi-lane road with one or more lanes in its direction of travel and one or more lanes in the opposite direction. It is surrounded by other vehicles traveling either in front of or behind the partially automated vehicle, or in one or more adjacent lanes, or in one or more of the oncoming lanes. These vehicles can be cars, trucks, buses, or motorcycles, traveling at different speeds. The road may be lined with tall buildings whose facades reflect sunlight in appropriate weather conditions.

[0006] The lanes of the multi-lane road are clearly marked by road markings, and traffic signs and traffic lights regulate traffic flow. Pedestrian crossings, recognizable by their characteristic zebra stripes, allow pedestrians to cross the street safely. Different vehicles move at different speeds, change lanes, and negotiate intersections.

[0007] A partially automated vehicle equipped with an ADAS / ADS system must continuously respond to changing conditions to ensure safe and efficient driving. To achieve this, the partially automated vehicle uses a variety of sensors to perceive its surroundings, including cameras, radar sensors, lidar, and ultrasound. These sensors continuously scan the partially automated vehicle's surroundings and collect data on other vehicles, pedestrians, bicycles, and road infrastructure. However, the partially automated vehicle's view is limited by surrounding vehicles, especially if they are driving directly alongside the partially automated vehicle or are in close proximity.

[0008] In this situation, it is especially important that the partially automated vehicle closely monitors the movements of surrounding vehicles. If another vehicle approaches a pedestrian crossing and slows down or stops, the partially automated vehicle must react accordingly. This means it must also slow down and possibly stop to ensure the safety of pedestrians attempting to cross the pedestrian crossing.

[0009] US 2021 / 0031766 discloses a cruise control system with a distance sensor, a camera, and a control unit. The distance sensor detects the forward speed of a vehicle traveling in front of a host vehicle. The camera detects an object within the detection zone. The control unit is connected to the distance sensor and the camera. The control unit uses the distance sensor to determine a change in the forward speed. The control unit reduces the speed of the host vehicle if the forward speed of the vehicle traveling in front decreases, no object is detected by the camera, and part of the detection zone is obscured by the host vehicle.

[0010] The invention is based on the object of developing possibilities to predict the presence of road users, in particular pedestrians, outside the detection range of a sensor device of an at least partially automated vehicle equipped with an ADAS / ADS system with very high accuracy and prediction quality and thus to increase the safety and reliability of driver assistance functions and / or driving functions.

[0011] This object is achieved according to the invention with respect to a method by the features of patent claim 1, with respect to a system by the features of patent claim 7, with respect to an ADAS / ADS system by the features of patent claim 13, and with respect to a computer program product by the features of patent claim 14. The further claims relate to preferred embodiments of the invention.

[0012] According to the invention, specialized task layers can be used to more precisely determine the respective input data for a decision model, so that the decisions of the decision model are based on reliable data processing. In this way, the presence of road users who are not within the detection range of the sensors of the sensor device of a partially automated vehicle with an ADAS / ADS system can be predicted. These predictions about the current vehicle environment are used in the ADAS / ADS module to control the vehicle, to warn the driver, or to trigger other safety-relevant actions. Overall, the present invention achieves improved collision detection and avoidance, allowing the partially automated vehicle to navigate safely.

[0013] According to a first aspect, the invention provides a method for improving the detection of road users who are outside the detection range of a sensor device of a vehicle for controlling a driver assistance system (ADAS) and / or an automated driving system (ADS). The method comprises the following steps: - Collecting sensor data of a traffic scene and surroundings of a vehicle with an ADAS / ADS system by sensors of a sensor device of the vehicle; - Forwarding the sensor data and / or fused sensor data to an evaluation module; - Preprocessing of the sensor data and / or the fused sensor data by an input layer of an AI model of the evaluation module, in particular for noise reduction and generation of preprocessed sensor data; - processing the pre-processed sensor data by a first task layer with an algorithm for detecting and tracking other vehicles to generate first input data for a decision model, wherein the first input data includes detected and tracked vehicles; - processing the preprocessed sensor data by a second task layer with a traffic sign recognition algorithm to generate second input data for a decision model, wherein the second input data includes recognized traffic signs; - processing the preprocessed sensor data by a third task layer with an algorithm for predicting the trajectories of other vehicles to generate third input data for a decision model, wherein the third input data contains predicted trajectories of other vehicles; - processing the pre-processed sensor data by a fourth task layer with an algorithm for detecting road users such as pedestrians or animals to generate fourth input data for a decision model, wherein the fourth input data contains predicted trajectories of the road users; - merging the first input data, the second input data, the third input data and the fourth input data into a fused context for detecting critical traffic situations such as collisions, obstacles, nearby pedestrians, unclear traffic signs; - Determining an indicator value for predicting whether road users such as pedestrians or animals are outside the detection range of the sensor device; - Selecting appropriate vehicle behavior such as acceleration, braking, lane changing, evasive maneuvering, stopping or warning based on the indicator value; - Implementation of the decisions made about the vehicle's behavior by an ADAS / ADS module by transmitting specific control commands to the vehicle's actuators for controlling the accelerator pedal, brakes, steering and other components.

[0014] In a further development, it is provided that the indicator value is a binary number, where the number 1 corresponds to the presence of a road user and the number 0 corresponds to the absence of a road user.

[0015] In an advantageous embodiment, it is provided that the input layer and the task layers each comprise neural networks, in particular convolutional neural networks (CNNs).

[0016] In a further embodiment, it is provided that the sensor device comprises a plurality of sensors such as cameras, lidar and ultrasound, which record sensor data in real time to detect the surroundings of the vehicle.

[0017] In a further embodiment, it is provided that additional task layers can be integrated into the AI ​​model.

[0018] In particular, the input layer, the task layers and the decision model will each be trained with suitable training data, which in particular consists of historical sensor data and corresponding target specifications.

[0019] According to a second aspect, the invention provides a system for improving the detection of road users who are outside the detection range of a sensor device of a vehicle for controlling a driver assistance system (ADAS) and / or an automated driving system (ADS). The system comprises a sensor device, an evaluation module, a decision model, and an ADAS / ADS module with an ADAS / ADS system. The system is designed to carry out the method according to the first aspect.

[0020] In a further development, it is provided that the indicator value is a binary number, where the number 1 corresponds to the presence of a road user and the number 0 corresponds to the absence of a road user.

[0021] In an advantageous embodiment, it is provided that the input layer and the task layers each comprise neural networks, in particular convolutional neural networks (CNNs).

[0022] In a further embodiment, it is provided that the sensor device comprises a plurality of sensors such as cameras, lidar and ultrasound, which record sensor data in real time to detect the surroundings of the vehicle.

[0023] Advantageously, additional task layers can be integrated into the AI ​​model.

[0024] In particular, the input layer, the task layers and the decision model will each be trained with suitable training data, which in particular consists of historical sensor data and corresponding target specifications.

[0025] According to a third aspect, the invention provides an ADAS / ADS system for a vehicle, wherein the method according to the first aspect is used to control the ADAS / ADS system.

[0026] According to a fourth aspect, the invention provides a computer program product comprising executable program code configured to carry out the method according to the first aspect when executed.

[0027] The invention is explained in more detail below with reference to embodiments shown in the drawing.

[0028] It shows: Fig. 1 a schematic representation of a multi-lane carriageway with a pedestrian crossing; Fig. 2 is a block diagram illustrating an embodiment of a system according to the invention; Fig. 3 a flow chart to explain the individual method steps of a method according to the invention; Fig. 4 a schematic representation of a computer program product.

[0029] Additional features, aspects and advantages of the invention or embodiments thereof will become apparent from the detailed description taken in conjunction with the claims.

[0030] An ADAS / ADS system is a complex collection of software and hardware components designed to improve driving safety and comfort. The software components of an ADAS / ADS system play a crucial role in collecting, processing, and evaluating data to enable various driver assistance and driving functions. Sensors include radar sensors, lidar sensors, cameras, ultrasonic sensors, and GPS (Global Positioning System). Radar sensors use electromagnetic waves to scan the environment and detect the position and movement of objects. Lidar sensors use laser pulses to measure distances and create a detailed 3D image of the surroundings. Cameras capture visual information and enable image recognition, as well as the detection of traffic signs, lanes, and pedestrians. Ultrasonic sensors detect obstacles in the immediate vicinity of the vehicle.GPS sensors determine the vehicle's precise position. Other sensors include speed sensors, steering angle sensors, accelerometers that measure the vehicle's acceleration in various directions, gyro sensors that measure the vehicle's rotational movement, humidity sensors, and temperature sensors.

[0031] The data collected by the various sensors is combined by data fusion or sensor fusion software. This enables a holistic perception of the environment and increases the accuracy of detecting objects and obstacles. The software creates a model of the vehicle's environment, including the position and movement of objects, road conditions, and other relevant information. The sensor and image processing software components analyze the sensor data, detect and classify objects, determine distances and speeds, and extract relevant information. Based on the environmental modeling and the acquired data, the system makes decisions such as controlling the vehicle, activating warnings, or initiating emergency braking. In an ADAS system, information and warnings are displayed to the driver via a user interface.This can be a head-up display, an infotainment system, or warning messages on the dashboard. In addition, an ADAS / ADS system can include communication functions that enable data exchange with other vehicles (V2V) or the infrastructure (V2I).

[0032] In the Fig. 1a - 1c depict the temporal movement sequence of an at least partially automated vehicle 10 equipped with an ADAS / ADS system 750, traveling in a lane 43 of a multi-lane roadway 40 with two lanes 43, 45 in the direction of travel and two lanes 47, 49 in the opposite direction. Furthermore, other vehicles 20 are located either in front of or behind the partially automated vehicle 10, or in the adjacent lane 45 next to the partially automated vehicle 10, or in the two opposite lanes 47, 49. These vehicles 20 can be passenger vehicles, trucks, buses, or motorcycles traveling at different speeds.

[0033] The partially automated vehicle 10 is equipped with a sensor device 200, which may include cameras, radar sensors, lidar sensors, and / or ultrasonic sensors. However, the detection range 15 of the sensor device is limited by the vehicles 70 traveling in the adjacent lane 45. In this case, it is important to observe how the vehicles 70 move. For example, if the vehicle 72 next to the partially automated vehicle 10 in the adjacent lane 45 approaches a pedestrian crossing 50 and either stops or reduces its speed, the partially automated vehicle 10 must also reduce its speed and stop at the pedestrian crossing 50.

[0034] In Fig. 1a, the partially automated vehicle 10 approaches pedestrian crossing 50. Vehicles 72 and 74 restrict the detection range 15 of the sensors of the partially automated vehicle 10. Vehicle 72 decelerates in the adjacent lane 45. From this point on, the partially automated vehicle 10 must also decelerate, as there is a high probability that a pedestrian 55 is on the pedestrian crossing 50.

[0035] In Fig. 1b, vehicle 72 has stopped in front of pedestrian crossing 50, and the partially automated vehicle 10 will also soon reach pedestrian crossing 50. Pedestrian 55 is still outside the detection range of the sensors. Starting at this time t1 at the latest, the partially automated vehicle 10 must also slow down to ensure the safety of pedestrian 55 and the driving safety of vehicle 10.

[0036] In Fig. 1c, the sensors of the partially automated vehicle 10 have a clear detection zone 15 and can detect the pedestrian 55 on the pedestrian crossing 50. If the driver of the vehicle 10 has not reacted beforehand, the emergency braking system or another ADAS function will react at that moment and initiate an emergency braking maneuver. However, such an emergency braking maneuver always involves potential risks.

[0037] Fig. 2 shows a system 100 according to the invention for predicting the presence of road users, in particular pedestrians, outside the detection range of a sensor device of an at least partially automated vehicle 10 equipped with an ADAS / ADS system. The system 100 comprises a sensor device 200, a database 300, an evaluation module 400, a decision model 500 and an ADAS / ADS module 700 with an ADAS / ADS system 750. The decision model 500 is typically integrated in the evaluation module 400.

[0038] The sensor device 200, the database 300, the evaluation module 400, and the ADAS / ADS module 700 can each be equipped with a processor and / or a memory unit. In particular, the database 300, the evaluation module 400, and the decision model 500 can be integrated into a cloud computing infrastructure. The connection between the sensor device 200 and the cloud computing infrastructure is established via interfaces. These interfaces can comprise different technologies, such as, in particular, wireless communication protocols or APIs (Application Programming Interfaces). The cloud computing infrastructure includes servers, storage, networks, and databases required to provide cloud services such as data storage, computing power, data analysis, and other services. The use of a cloud computing infrastructure is particularly advantageous with regard to computing speed.Cloud computing services utilize high-performance servers and networks to perform complex calculations quickly and efficiently. Furthermore, cloud computing platforms utilize a distributed architecture, where computing resources are spread across multiple physical locations. This enables parallel processing and load balancing, increasing overall computing speed.

[0039] In the context of the invention, a "processor" can be, for example, a machine or an electronic circuit. A processor can in particular be a main processor unit ("Central Processing Unit", CPU), a microprocessor or a microcontroller, e.g., an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions. A processor can also be a virtualized processor, a virtual machine or a soft CPU. It can also be, for example, a programmable processor that is equipped with configuration steps for executing the aforementioned method according to the invention or is configured with configuration steps such that the programmable processor implements the inventive features of the method, the component, the modules or other aspects and / or sub-aspects of the invention.In particular, the processor can contain highly parallel computing units and powerful graphics modules.

[0040] In the context of the invention, a "storage unit" or "storage module" and the like can be understood as, for example, a volatile memory in the form of random access memory (RAM), a permanent memory such as a hard disk or a data storage device, or, for example, a replaceable storage module. The storage module can also be a cloud-based storage solution.

[0041] In the context of the invention, a "module" can be understood, for example, as a processor and / or a memory unit for storing program instructions. For example, the processor is specifically configured to execute or implement the program instructions of the method according to the invention or a step of the method according to the invention.

[0042] In the context of the invention, “data” refers to both raw data and already processed data, e.g. from measurement results from sensors or from simulation results.

[0043] In addition, communication connections are provided for the exchange and transmission of data between the individual modules, which are designed in particular as wireless communication connections, e.g., as a mobile radio connection (e.g., 4G LTE, 5G, 6G) and / or as near-field communication connections, e.g., Bluetooth®, Ethernet, NFC (Near Field Communication) or Wi-Fi®.

[0044] The sensor device 200 of the vehicle 10 typically includes a plurality of sensors such as cameras, lidar and ultrasound, which record sensor data 220 for detecting the environment.

[0045] Cameras are used to capture visual information about the vehicle's surroundings. They can be used for traffic sign recognition, lane detection, object classification, and traffic sign recognition. Depending on the application and intended use, the cameras can have different specifications.

[0046] A camera's resolution determines the clarity and detail of the captured images. High-resolution cameras are often used for ADAS / ADS systems to enable precise object detection and tracking. Typical resolutions range from Full HD (1920x1080 pixels) to 4K (3840x2160 pixels), depending on the application requirements.

[0047] The frame rate indicates how many images the camera can capture per second. A higher frame rate enables faster image updates and more accurate detection of fast-moving objects. Typical frame rates for cameras for an ADAS / ADS system 750 are between 30 and 60 frames per second (fps), but in some cases, an even higher frame rate may be required.

[0048] Cameras can capture various wavelengths of the electromagnetic spectrum, including visible light, infrared (IR), and ultraviolet (UV). For an ADAS / ADS system, visible-light (RGB) cameras are typically used to capture visual information about the vehicle's surroundings. However, in some cases, infrared cameras can also be used to provide additional information in low-light conditions or for special applications such as night vision support.

[0049] The size of the image sensor and the quality of the optics affect the camera's light sensitivity and image quality. Larger image sensors and high-quality lenses can help produce sharper images with less noise and distortion. This is especially important for object detection and tracking in different lighting conditions and environments. HDR (High Dynamic Range) cameras capture a wider dynamic range of brightness, meaning they can capture both dark and bright areas of an image with high accuracy, allowing for clear details even in high-contrast lighting conditions such as shadows and sunlight.

[0050] Typically, an ADAS / ADS system 750 often uses multiple cameras with different specifications and viewing angles to ensure precise perception.

[0051] Lidar (Light Detection and Ranging) sensors use laser beams to precisely map the vehicle's surroundings and create 3D models of objects. They can be used for obstacle detection, navigation, and precise vehicle positioning.

[0052] LRR sensors (Range Radar Sensors) are designed to detect distant objects and determine their position and speed. They can be used, for example, to detect other vehicles on the highway or to identify obstacles at a great distance.

[0053] Medium-range radar (MRR) sensors are specialized for detecting objects at medium distances from the vehicle. They can be used for detecting vehicles in adjacent lanes, for collision warning, or for adaptive cruise control.

[0054] NR sensors (nano-radar) are smaller and more precise radar sensors that can detect fine details in the surroundings of the vehicle 10. They can be used, for example, to detect pedestrians, cyclists, or other objects near the vehicle 10.

[0055] The sensor data 220 is continuously collected as a time series and combined using sensor data fusion. The information from the various sensors is combined to create a comprehensive and accurate picture of the vehicle's surroundings. By integrating various sensors and fusing their data, the vehicle 10 can develop a detailed understanding of its surroundings and perform complex driving tasks such as lane keeping, collision avoidance, and autonomous navigation.

[0056] The fused sensor data 250 is made available via a FlexRay / CAN bus, a communication bus used in the vehicle 10 to exchange data between various components.

[0057] FlexRay is a communication protocol developed in the automotive industry for bidirectional data communication between various electronic control units (ECUs) in a vehicle, such as sensors, actuators, engine control units, transmission control units, brake control units, etc. FlexRay supports a wide range of communication requirements, including time-critical applications that require data to be transmitted with very low latency. It was specifically developed for applications with high demands on real-time transmission and reliability. FlexRay enables real-time data transmission with predefined schedules and guaranteed time frames. This is especially important for safety-critical applications such as braking and steering controls. FlexRay supports high bandwidth for data transmission, allowing large amounts of sensor data and control commands to be exchanged between different components in the vehicle.FlexRay is flexibly configurable and can be adapted to the specific requirements of different vehicle manufacturers and applications. Furthermore, FlexRay is designed to operate reliably even under extreme conditions, such as environments with strong electromagnetic interference or high temperatures. In modern vehicles, FlexRay is often used in conjunction with other communication protocols such as CAN (Controller Area Network) and Ethernet to create a comprehensive and reliable communications infrastructure.

[0058] The sensor data 220 and / or the fused sensor data 250 are transmitted to the evaluation module 400. The evaluation module 400 comprises a machine AI model 410 (artificial intelligence) comprising several layers. The layers of the AI ​​model 410 consist of an input layer 430, a first task layer 440, a second task layer 450, a third task layer 460, and a fourth task layer 470. According to the invention, additional task layers 480 can be provided to expand the AI ​​model 410.

[0059] The task layers 440, 450, 460, 470, and 480 are specialized for different tasks. The task of the first task layer 440 is the detection of vehicles 70 in the environment of the partially automated vehicle 10. The task of the second task layer 450 is the recognition of traffic signs. The task of the third task layer 460 is the prediction of trajectories of the vehicles 70 in the environment of the automated vehicle 10. The task of the fourth task layer 470 is the detection and prediction of trajectories of other road users such as pedestrians 55. However, these road users can also be cyclists or even animals on the roadway. The fifth task layer 480 can, for example,be specialized in processing other environmental data such as buildings, trees, special road markings, special weather conditions that influence the probability of the occurrence of road users who are not visible in the detection area 15 of the sensors of the sensor device.

[0060] The input layer 430 of the AI ​​machine model 410 is used to preprocess the sensor data 220 and / or the already fused sensor data 250. The sensor data 220 includes both images and digitized values ​​from the sensors of the sensor device 200. The sensor data 220 and / or the already fused sensor data 250 are processed by the input layer 430 so that they can be further processed by the subsequent task layers 440, 450, 460, and 470. Preprocessing particularly includes noise suppression. Noise can take various forms in the raw sensor data and be caused by various sources, including electrical noise, environmental disturbances, sensor deficiencies, and other disturbances.Reducing noise in the raw data is crucial to improving data quality and ensuring that the subsequent task layers 440, 450, 460, and 470 of the AI ​​model 410 can operate accurately and reliably. For example, low-pass filters are used to attenuate high-frequency signal components while allowing lower frequencies to pass. This is particularly useful for reducing high-frequency noise that can degrade signal quality. Furthermore, calibration and correction may be required to improve data accuracy.

[0061] The thus preprocessed sensor data 435 is now simultaneously transferred to the first task layer 440, the second task layer 450, the third task layer 460, and the fourth task layer 470. If a further task layer 480 is provided, the preprocessed sensor data 435 is also forwarded to this task layer 480.

[0062] Task layers 440, 450, 460, and 470 each comprise extraction algorithms for feature extraction and feature selection from the preprocessed sensor data 435 in order to extract and select the features relevant to the respective task layer. Depending on the task layer, this refers to the identification of objects such as vehicles, pedestrians, bicycles, traffic signs, etc., as well as their position, speed, and direction.

[0063] The task layers 440, 450, 460, and 470 of the AI ​​model 410 learn through training to extract the relevant features from the preprocessed sensor data 435. To do this, the task layers 440, 450, 460, and 470 each use neural networks, in particular convolutional neural networks (CNNs), as these are particularly suitable for image processing. However, classical methods can also be used for numerical data.

[0064] Neural networks are a fundamental concept in machine learning, inspired by the functioning of the human brain. They are often referred to as the prototype of AI algorithms. A neural network consists of neurons arranged in multiple layers and connected to each other in various ways. A neuron is capable of receiving information at its input from outside or from another neuron, evaluating it in a specific way, and passing it on in a modified form at the neuron's output to another neuron or outputting it as the final result. Hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, there can be several layers of hidden neurons. They ensure the forwarding and processing of information. Output neurons ultimately provide a result and pass it on to the outside world.Different types of neural networks are created by the arrangement and connection of neurons. The simplest form is feed-forward neural networks, in which data flows through layers of neurons without feedback. Convolutional neural networks (CNNs) are specialized neural networks for processing images and other two-dimensional data. Recurrent neural networks (RNNs) have a feedback loop and are therefore suitable for processing sequential data such as text or time series. Neural networks can be combined to increase their learning capacity. Such coupled neural networks are also known as deep learning.

[0065] A convolutional neural network (CNN) consists of several layers specifically designed for processing images or other multidimensional data structures. The input layer of the CNN receives the input image, which is encoded as a matrix of pixel values. Each pixel represents the brightness or color of a specific image point. The size of the input layer is usually the same as the size of the input image. Convolutional layers are the core of a CNN. They consist of several filters convolved over the input image to extract features such as edges, shapes, or textures. Each filter learns to respond to specific patterns in the image. The number of filters and their size are hyperparameters determined during the network design. After convolution with the filters, an activation function is applied to the results.This feature introduces nonlinearity into the network and helps model complex relationships between features. Pooling layers are used to reduce the dimensionality of the feature maps and lower computational costs. Typical pooling operations are max-pooling and average-pooling, which select the maximum and average values, respectively, within a specific region of the feature map. Several convolutional and pooling layers are typically followed by one or more fully connected layers. These layers function like conventional neural networks, learning complex nonlinear relationships between the extracted features. The output layer of the CNN provides the final prediction or extraction based on the features learned in the previous layers.This basic structure of a CNN can vary depending on the application and complexity of the problem, and additional layers or special architectures can be used to meet specific requirements.

[0066] In addition, task layers 440, 450, 460, and 470 each contain classification algorithms to classify the extracted features. For example, the first task layer 440 for vehicle detection can distinguish between different vehicle types, such as passenger cars or trucks.

[0067] The task layers 440, 450, 460, 470 are each trained with training data consisting of historical sensor data and the corresponding target specifications. This training data is typically stored in the database 300. This training can be performed iteratively, with the task layers 440, 450, 460, 470 each being optimized based on feedback about their performance. This can include adjusting hyperparameters, changing the network architecture, or selecting appropriate loss functions. This also applies to submodels that may be included in the respective task layers 440, 450, 460, 470.

[0068] After training, task layers 440, 450, 460, and 470 are tested with separate validation data to ensure they generalize well and are not overfitted. This is important to ensure that task layers 440, 450, 460, and 470 perform well with unknown data, not just the training data. The trained task layers 440, 450, 460, and 470 are then used on new sensor data 220, 250 collected in real time from the vehicle sensors.

[0069] The first task layer 440 contains a vehicle detection and tracking algorithm to detect and track vehicles in a traffic scene, and potentially predict their movements over time. The first step is to identify and locate possible vehicles in an image. This can be done using techniques such as convolutional neural networks (CNNs), histogram-based approaches, or other object detection algorithms. The goal is to identify the regions in the image where vehicles may be located. After detection, the identified vehicles are tracked over multiple frames to track their movements over time. This can be achieved by matching vehicles between consecutive frames, tracking motion paths, or fitting vehicle motion models.Techniques such as Kalman or particle filters are commonly used to estimate and update the position and speed of vehicles. During tracking, various attributes of the vehicles are extracted to characterize and classify them. These include properties such as size, shape, color, direction of movement, speed, acceleration, etc. These attributes can be used to identify and distinguish vehicles, as well as to analyze behavior patterns.

[0070] Based on the extracted attributes and the movement paths of the vehicles, predictions can be made about their future movements and positions. This can be done by applying predictive models such as Kalman filters, dynamic movement models, or AI models trained with historical movement data. This prediction of the trajectories can already be performed in the first task layer 440, or more specifically in the third task layer 460. Within the scope of the invention, it can also be provided that the first task layer 440 and the third task layer 460 interact.

[0071] Finally, the vehicles detected and tracked by the first task layer 440 are passed to the decision model 500 as first input data 510.

[0072] The second task layer 450 contains an algorithm for traffic sign recognition using camera images. Either the camera images have already been preprocessed in the input layer 430 to improve image quality by applying filters for noise reduction, contrast adjustment, and color space transformations, or this is performed in the second task layer 450. Possible traffic signs are then located and segmented in the image. This can be achieved using techniques such as edge detection, region growing, thresholding, and morphological functions. The goal is to identify the regions in the image that may contain traffic signs. After segmentation, features are extracted from the identified regions to prepare them for classification. These features can represent various properties of the traffic signs, such as shape, color, texture, symmetry, etc.The extracted features are then fed to a classifier trained to detect and classify different types of traffic signs. This can be an artificial neural network, a support vector machine (SVM), a decision tree, or another classification model. The second task layer 450 is previously trained with a training dataset containing images of various traffic signs and specifying the corresponding traffic sign classes. After classification, the detected traffic signs can be subjected to further post-processing techniques to correct false positive or false negative detections. Additionally, a detection reliability assessment can be performed to ensure that only reliable detections are considered.

[0073] Finally, the traffic signs recognized by the second task layer 450 are passed to the decision model 500 as second input data 520.

[0074] The third task layer 460 contains an algorithm for predicting the trajectories of other vehicles. The vehicles detected by the sensors are tracked over several time steps to determine their trajectories. This can be achieved using techniques such as Kalman filters, particle filters, or other tracking algorithms. The goal is to estimate the position, speed, and acceleration of each vehicle over time. During tracking, various attributes of the vehicles are extracted to characterize their movement patterns. These include features such as current position, speed, acceleration, steering angle, lane, traffic density, and other contextual information. Based on the extracted vehicle attributes, future vehicle movements are predicted.

[0075] For this purpose, a prediction model is used, which employs methods such as physical models, motion models, machine learning, or AI techniques. The prediction model is trained with historical data to learn the future movement patterns of the vehicles and predict where they will move. The prediction model is then used to predict the future movement paths of the detected vehicles. This involves estimating the likely positions of the vehicles at different times in the future, based on their current movement states and the predicted movement patterns.

[0076] Finally, the trajectories of other vehicles predicted by the third task layer 460 are passed to the decision model 500 as third input data 530.

[0077] The fourth task layer 470 contains an algorithm for detecting road users such as pedestrians or animals and predicting their trajectories. The preprocessed sensor data 435, along with images or videos from cameras and / or other sensors, is used to identify possible road users within the detection range of the sensors. This can be done using techniques such as convolutional neural networks (CNNs) or other object detection algorithms. The goal is to find regions in the image that contain pedestrians or animals. After detection, the identified road users are tracked across multiple image sequences to determine their movement paths. This can be done by matching objects between consecutive images, by tracking movement paths, or by adapting models for object movement.Methods such as Kalman filters, particle filters, or even deep learning-based tracking algorithms can be used to track the movement of detected objects. During tracking, various attributes of the road users are extracted to characterize their movement patterns. These include features such as current position, speed, acceleration, direction of movement, size, shape, color, and other contextual information. Based on the extracted attributes of the road users, a prediction model is created that predicts the future movements of the road users. This can be done by applying motion models, machine learning, or AI techniques. The prediction model is trained with historical data to learn the future movement patterns of the road users and make predictions about where they will move.

[0078] The prediction model is then used to predict the future movement paths of road users. This involves estimating the likely positions of road users at different times in the future, based on their current movement states and the predicted movement patterns.

[0079] Finally, the trajectories of the road users predicted by the fourth task layer 470 are passed to the decision model 500 as fourth input data 540.

[0080] The decision model 500 is used to control the ADAS / ADS module 700. The various input data 510, 520, 530, 540 such as detected and tracked vehicles, predicted trajectories of vehicles and other road users, and detected traffic signs are fused and placed in a fused context.

[0081] The fused context refers to an integrated data set or representation that combines various information and data sources to provide a holistic view of the environment and the current situation of the vehicle 10. In addition to the input data 510, 530, 530, 540, the fused context may contain additional data about the current state of the vehicle, including speed, acceleration, steering angle, position, operating states of the vehicle systems, energy consumption, and other parameters relevant to vehicle control. Furthermore, data about the vehicle's environment, such as weather conditions, lighting conditions, road conditions, traffic density, traffic signs, lane markings, pedestrian crossings, and other factors that may influence driving safety and behavior, may be included.Map and navigation data can include information about road layouts, speed limits, traffic flow, elevation profiles, and other geographical features. Furthermore, historical data about previous trips, traffic conditions, traffic patterns, and other relevant events can be combined to better understand the current situation and support future decisions.

[0082] Based on the fused context, potential hazardous situations and risks in traffic are identified. This can include the detection of critical traffic situations such as collisions, obstacles, nearby pedestrians, unclear traffic signs, etc. Each detected situation is assessed for its risk to determine the urgency of a response. According to the invention, the prediction that road users such as pedestrians or animals are outside the detection range of the sensor device 200 is evaluated with an indicator value 570, which is decisive for the decision-making process. The indicator value 570 can be a binary number, where the number 1 corresponds to the presence of a road user and the number 0 to the absence of a road user.

[0083] The decision model 500 can include various algorithms and techniques, including neural networks. For example, the decision model 500 uses the indicator value 570 to make decisions about the operation of individual functions of the ADAS / ADS system 750. Furthermore, various machine learning techniques can be used to make decisions based on historical data or experience. The decision model can be trained using labeled data sets to learn decision rules, e.g., the classification of traffic situations. Additionally, unsupervised learning can be used to recognize structures in the data and derive decisions from them, e.g., clustering of traffic situations. Other algorithms include reinforcement learning or evolutionary algorithms.

[0084] Based on the detected situations and risks, the decision model 500 makes decisions about how the vehicle 10 should react and forwards these decisions to the ADAS / ADS module 700. The decisions may include selecting appropriate behaviors such as accelerating, braking, changing lanes, swerving, stopping, or warning, which are implemented by the ADAS / ADS module 700. The decisions are made taking into account safety aspects, traffic regulations, passenger comfort, and other factors. The decisions made are then translated by the ADAS / ADS module 700 into concrete control commands for the actuators of the vehicle 10. This includes controlling the accelerator pedal, brakes, steering, and other components to steer the vehicle 10 according to the defined behavior plans. This ensures that the actions are executed safely and smoothly.

[0085] The decision model 500 continuously monitors the environment of the vehicle 10 using the sensor device 200 and dynamically adapts its decisions and behavior plans to changing conditions. This may include re-evaluating situations, updating risk assessments, and adapting behavior plans in real time to ensure that the vehicle 10 responds appropriately and safely to its environment.

[0086] The modular design of the AI ​​model 410 with multiple layers 430, 440, 450, 460, 470, each of which is trained separately, enables it to be more accurate than a single AI model. Each layer 430, 440, 450, 460, 470 of the modular AI model 410 is tailored to a specific task or aspect of the problem. This allows the individual layers 430, 440, 450, 460, 470 to respond more effectively to the respective requirements and develop specialized capabilities. Because each layer 430, 440, 450, 460, 470 is trained with specific training data, this data can be specifically selected and optimized to improve the performance of each layer 430, 440, 450, 460, 470. This allows for more precise adaptation to the specific features and properties of the training data.Training each layer 430, 440, 450, 460, and 470 with specific training data also reduces the risk of overfitting, as each layer specializes only in the features and patterns in the training data relevant to its task. This contributes to improved generalization capabilities of the overall model. In particular, the AI ​​model 410 is flexible and scalable, as new layers or modules can be easily added or removed to adapt or improve the performance or functionality of the AI ​​model 410.

[0087] According to the invention, the specialized task layers 440, 450, 460, 470 can thus determine the respective input data 510, 520, 530, 540 more precisely, so that the decisions of the decision model 500 are based on reliable data processing. In this way, the presence of road users who are not within the detection range of the sensors of the sensor device 200 of the vehicle 10 can be predicted. These predictions about the current vehicle environment are used in the ADAS / ADS module 700 to control the vehicle 10, to warn the driver or trigger other safety-relevant actions. Overall, the present invention achieves improved collision detection and avoidance, so that the partially automated vehicle 10 navigates safely.

[0088] In Fig. 3 shows the method steps of a method for improving the detection of road users who are outside the detection range of a sensor device of a vehicle for controlling a driver assistance system (ADAS) and / or an automated driving system (ADS). The method comprises the following steps: - Acquiring (S10) sensor data 220 of a traffic scene and surroundings of a vehicle 10 with an ADAS / ADS system 750 by sensors of a sensor device 220 of the vehicle 10. - Forwarding (S20) the sensor data 220 and / or fused sensor data 250 to an evaluation module 400. - Preprocessing (S30) of the sensor data 220 and / or the fused sensor data 250 by an input layer 430 of an AI model 410 of the evaluation module 400, in particular for noise reduction and generating preprocessed sensor data 435. - Processing (S40) the preprocessed sensor data 435 by a first task layer 440 with an algorithm for detecting and tracking other vehicles to generate first input data 510 for a decision model 500, wherein the first input data 510 contains detected and tracked vehicles. - Processing (S50) the preprocessed sensor data 435 by a second task layer 450 with a traffic sign recognition algorithm to generate second input data 520 for a decision model 500, wherein the second input data 520 contains recognized traffic signs. - Processing (S60) the preprocessed sensor data 435 by a third task layer 460 with an algorithm for predicting the trajectories of other vehicles to generate third input data 530 for a decision model 500, wherein the third input data 530 contains predicted trajectories of other vehicles. - Processing (S70) the preprocessed sensor data 435 by a fourth task layer 470 with an algorithm for detecting road users such as pedestrians or animals to generate fourth input data 540 for a decision model 500, wherein the fourth input data 540 contains predicted trajectories of the road users. - fusing (S80) the first input data 510, the second input data 520, the third input data 530 and the fourth input data 540 into a fused context for detecting critical traffic situations such as collisions, obstacles, pedestrians in the vicinity, unclear traffic signs. - Determining S90 an indicator value 570 regarding the prediction of whether road users such as pedestrians or animals are outside the detection range of the sensor device 200. - Selecting S100 an appropriate vehicle behavior such as acceleration, braking, lane change, evasive maneuvering, stopping or warning based on the indicator value 570. - Implementing (S110) the decisions made regarding the behavior of the vehicle 10 by an ADAS / ADS module (700) by transmitting concrete control commands to the actuators of the vehicle 10 for controlling the accelerator pedal, the brakes, the steering and other components.

[0089] Fig. 4 schematically illustrates a computer program product 900 comprising executable program code 950 configured to perform the method according to the first aspect of the present invention when executed. Reference symbol 10 partially automated vehicle 15 Detection range 20 lanes 30 secondary lanes 40 lane 43 lanes 45 lanes 47 lanes 49 lanes 50 Pedestrian crossing 55 pedestrians 70 vehicles 72 vehicles 74 vehicles 100 systems 200 sensor device 220 sensor data 250 fused sensor data 300 database 400 evaluation module 410 AI model 430 Input layer 435 preprocessed sensor data 440 first task shift 450 second task shift 460 third task layer 470 fourth task layer 480 further task layer 500 decision model 510 first input data 520 second input data 530 third input data 540 fourth input data 570 indicator value 700 ADAS / ADS module 750 ADAS / ADS system 900 Computer program product 950 program code QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2021 / 0031766

[0009]

Claims

[1] Method for improving the detection of road users who are outside the detection range of a vehicle's sensor device for controlling an Advanced Driver Assistance System (ADAS) and / or an Automated Driving System (ADS), comprising the following steps: - Acquisition (S10) of sensor data (220) of a traffic scene and environment of a vehicle (10) with an ADAS / ADS system (750) by sensors of a sensor device (220) of the vehicle (10); - Forwarding (S20) the sensor data (220) and / or fused sensor data (250) to an evaluation module (400); - Preprocessing (S30) of the sensor data (220) and / or the fused sensor data (250) by an input layer (430) of an AI model (410) of the evaluation module (400) in particular for noise reduction and generation of preprocessed sensor data (435); - Processing (S40) the preprocessed sensor data (435) by a first task layer (440) with an algorithm for detecting and tracking other vehicles to generate first input data (510) for a decision model (500), wherein the first input data (510) includes detected and tracked vehicles; - Processing (S50) the preprocessed sensor data (435) by a second task layer (450) with a traffic sign recognition algorithm to generate second input data (520) for a decision model (500), wherein the second input data (520) contain recognized traffic signs; - Processing (S60) the preprocessed sensor data (435) by a third task layer (460) with an algorithm for predicting the trajectories of other vehicles to generate third input data (530) for a decision model (500), wherein the third input data (530) contain predicted trajectories of other vehicles; - Processing (S70) the preprocessed sensor data (435) by a fourth task layer (470) with an algorithm for detecting road users such as pedestrians or animals to generate fourth input data (540) for a decision model (500), wherein the fourth input data (540) contains predicted trajectories of the road users; - Fusion (S80) of the first input data (510), the second input data (520), the third input data (530) and the fourth input data (540) into a fused context for the detection of critical traffic situations such as collisions, obstacles, pedestrians nearby, unclear traffic signs; - Determining (S90) an indicator value (570) regarding the prediction of whether road users such as pedestrians or animals are outside the detection range of the sensor device (200); - Selecting (S100) an appropriate vehicle behavior such as accelerating, braking, changing lanes, swerving, stopping or warning based on the indicator value (570); - Implementation (S110) of the decisions made regarding the behavior of the vehicle (10) by an ADAS / ADS module (700) by transmitting specific control commands to the actuators of the vehicle (10) to control the accelerator pedal, brakes, steering and other components. [2] Method according to claim 1, wherein the indicator value (570) is a binary number, and wherein the number 1 corresponds to the presence of a road user and the number 0 corresponds to the absence of a road user. [3] Method according to claim 1 or 2, wherein the input layer (430) and the task layers (440, 450, 460, 470) each comprise neural networks, in particular Convolutional Neural Networks (CNNs). [4] Method according to any one of claims 1 to 3, wherein the sensor device (200) comprises a plurality of sensors such as cameras, lidar and ultrasound which record sensor data (220) in real time to detect the environment of the vehicle (10). [5] Method according to any one of claims 1 to 4, wherein further task layers (480) can be integrated into the AI ​​model (410). [6] Method according to any one of claims 1 to 5, wherein the input layer (430), the task layers (440, 450, 460, 470) and the decision model (500) are each trained with suitable training data, which in particular consists of historical sensor data and corresponding target specifications. [7] System (100) for improving the detection of road users who are outside the detection range of a sensor device of a vehicle, for controlling an ADAS and / or an automated driving system (ADS), comprising a sensor device (200), an evaluation module (400), a decision model (500) and an ADAS / ADS module (700) with an ADAS / ADS system (750), wherein the system (100) is configured to carry out the method according to claim 1. [8] System (100) according to claim 7, wherein the indicator value (570) is a binary number, and wherein the number 1 corresponds to the presence of a road user and the number 0 corresponds to the absence of a road user. [9] System (100) according to claim 7 or 8, wherein the input layer (430) and the task layers (440, 450, 460, 470) each comprise neural networks, in particular Convolutional Neural Networks (CNNs). [10] System (100) according to any one of claims 7 to 9, wherein the sensor device (200) comprises a plurality of sensors such as cameras, lidar and ultrasound, which record sensor data (220) in real time to detect the environment of the vehicle (10). [11] System (100) according to one of claims 7 to 10, wherein further task layers (480) can be integrated into the AI ​​model (410). [12] System (100) according to one of claims 7 to 11, wherein the input layer (430), the task layers (440, 450, 460, 470) and the decision model (500) are each trained with suitable training data, which in particular consists of historical sensor data and corresponding target specifications. [13] ADAS / ADS system (750) for a vehicle (10), characterized by , that the method according to one of claims 1 to 6 is used to control the ADAS / ADS system (750). [14] Computer program product (900) comprising an executable program code (950) configured to perform the method according to any one of claims 1 to 6 when executed.

Images