Driver assistance system and driver assistance procedures for automated driving of a vehicle
The driving assistance system uses a neural network trained on sensor data to estimate human-like driving paths, reducing the need for high computational power by processing pre-processed information, thus addressing the resource-intensive challenges of existing systems.
Patent Information
- Application Number
- DE102023113531
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-28
AI Technical Summary
Existing driving assistance systems for automated vehicles require large amounts of training data and high computational power to estimate driving paths that mimic human behavior, making them complex, expensive, and resource-intensive, particularly due to the reliance on GPUs for processing raw pixel data from cameras.
A driving assistance system using a neural network trained on real and simulated data from environmental sensors, such as LiDAR and radar, processes pre-processed sensor information like lane polynomials and odometry to estimate driving paths, allowing the use of CPUs instead of GPUs, thereby reducing computational requirements while mimicking human driving behavior.
The system effectively estimates driving paths that imitate human behavior with reduced computational resources, enabling efficient and cost-effective automated driving across various road conditions, including lane changes and handling unstable sensor data.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The present disclosure relates to a driver assistance system for automated driving of a vehicle, a vehicle having such a driver assistance system, a driver assistance method for automated driving of a vehicle, a method for training neural networks to output driving paths for automated driving, and a system for training neural networks to output driving paths for automated driving. In particular, the present disclosure relates to a data-driven estimation of a driving path. State of the art
[0002] Driver assistance systems for automated driving are becoming increasingly important. Automated driving can be achieved with various levels of automation. Examples of automation levels include assisted, partially automated, conditionally automated, highly automated, or fully automated driving. The five levels of automation mentioned above correspond to SAE Levels 1 to 5 of the SAE J3016 standard (SAE - Society of Automotive Engineering) as of April 30, 2021. With fully automated driving (SAE Level 5), the system can perform all aspects of the dynamic driving task under any road and environmental conditions that can also be mastered by a human driver.
[0003] The automated vehicle uses sensors that visually perceive the environment, both in the range visible to humans and invisible to humans. These sensors can be, for example, a camera, radar, and / or LiDAR. Along with highly accurate maps, these are the primary signal sources for driver assistance systems for automated driving.
[0004] The driver assistance system typically uses an environment model component to estimate a virtual lane centerline based on the sensor information. This centerline can be used by downstream control components to keep the vehicle in the center of the lane while driving (trajectory controller). It can also determine whether there are any objects in the lane the vehicle is traveling in.
[0005] The centerline can be estimated using a Kalman filter. For example, a camera records the area in front of the vehicle, and camera software detects lane markings in the recorded pixel data. The detected lane markings are transmitted to the control system of the automated vehicle, for example, as polynomials in a top-down coordinate system with the origin at the rear axle of the vehicle (the so-called ego coordinate system). Due to sensor noise, the shape of the polynomials changes in each time step. The Kalman filter filters out this noise and determines a virtual line midway between the polynomials for the left and right lane markings of the lane in which the vehicle is located.
[0006] However, it is not always desirable for the vehicle to drive exactly in the middle between the detected lane markings, for example in the case of lane widening, as in the Fig. 1 is shown as an example. Here, the human driver would drive along a straight line GL, but the center line BL between the lane markings SM is curved. This results in an unnatural driving path for the automated vehicle. It is therefore desirable to provide a driving path for the automated vehicle that approximates human behavior.
[0007] Instead of a Kalman filter, a neural network can be used to determine a driving path that mimics human behavior (so-called behavioral cloning). For this purpose, driving trajectories of real drivers are recorded (odometry) and, in addition to sensor data, are presented to the neural network during training as the target behavior in this situation. The neural network thus learns to output a driving path in corresponding situations that corresponds to the odometry contained in the training data.
[0008] However, such an approach requires a large amount of training data from real-world driving situations in a wide variety of situations, making it not only complex but also expensive to implement. Furthermore, behavioral cloning uses pixel data directly from the camera to estimate a driving path. However, processing pixel data in real time in an automated vehicle requires high computing power, which only a GPU (graphical processing unit), for example, can provide. However, a GPU is expensive and is therefore generally not installed in vehicles.
[0009] Therefore, it is desirable to estimate a driving path that mimics human behavior in a reliable manner and with little computational power. Disclosure of the invention
[0010] It is an object of the present disclosure to provide a driver assistance system for the automated driving of a vehicle, a vehicle with such a driver assistance system, a driver assistance method for the automated driving of a vehicle, a method for training neural networks to output driving paths for automated driving, and a system for training neural networks to output driving paths for automated driving, which enable a reliable estimation of a driving path that mimics human behavior. Furthermore, it is an object of the present disclosure to enable an estimation of a driving path with low computing resources.
[0011] This object is achieved by the subject matter of the independent claims. Advantageous embodiments are specified in the subclaims.
[0012] According to an independent aspect of the present disclosure, a driver assistance system for automated driving of a vehicle, in particular a motor vehicle, is provided. The driver assistance system comprises a driving path determination module with at least one neural network and a control module.
[0013] The at least one neural network is a network based on (i) recorded training data from real manual rides and / or (ii) neural network trained on simulated training data from driving simulations.
[0014] The travel path determination module is configured to input detections of an environmental sensor system (directly) into the at least one neural network, wherein the detections indicate two or more lane boundaries in an environmental area of the vehicle, and wherein the at least one neural network is configured to determine the travel path based on the detections and output it to the control module.
[0015] The control module is configured to perform automated driving of the vehicle on an ego lane of the vehicle based on the driving path determined by the driving path determination module.
[0016] According to the invention, the driver assistance system uses at least one trained neural network to estimate the driving path, which mimics human behavior. The at least one neural network uses direct detections, i.e., preprocessed sensor information, and optionally further information, such as odometry information and information from a navigation map, to estimate the driving path. The detections can, for example, be lane polynomials. This means that no raw sensor data, such as pixel data, is input into the at least one neural network, so that fewer computing resources are required for estimating the driving path. This allows, for example, a CPU to be used instead of an expensive GPU to estimate the driving path.
[0017] In this document, the term "automated driving" refers to driving with automated longitudinal and / or lateral guidance. Automated driving can, for example, involve extended driving on the highway or limited-time driving while parking. The term "automated driving" encompasses automated driving with any degree of automation. Examples of levels of automation include assisted, partially automated, conditionally automated, highly automated, and fully automated driving (each with an increasing degree of automation). The five levels of automation mentioned above correspond to SAE Levels 1 to 5 of the SAE J3016 standard (SAE - Society of Automotive Engineering) as of April 30, 2021.
[0018] In assisted driving (SAE Level 1), the system performs longitudinal or lateral guidance in certain driving situations. In partially automated driving (SAE Level 2), the system assumes longitudinal and lateral guidance in certain driving situations, although the driver must continuously monitor the system, as with assisted driving. In conditionally automated driving (SAE Level 3), the system assumes longitudinal and lateral guidance in certain driving situations without the driver having to continuously monitor the system; however, the driver must be able to assume control of the vehicle within a certain period of time upon request from the system. In highly automated driving (SAE Level 4), the system assumes control of the vehicle in certain driving situations, even if the driver does not respond to a request for intervention, thus eliminating the driver as a fallback.In fully automated driving (SAE Level 5), the system can perform all aspects of the dynamic driving task under any road and environmental conditions that can also be mastered by a human driver.
[0019] Furthermore, the term "at least partially automated driving or maneuvering" is also understood in this document to include partially automated, conditionally automated, highly automated, and fully automated driving. In other words, the term "at least partially automated driving" refers to a level of automation up to and including SAE Level 2.
[0020] The vehicle preferably comprises an environmental sensor system configured to capture environmental data. The environmental sensor system preferably comprises at least one LiDAR system and / or at least one radar system and / or at least one camera and / or at least one ultrasound system. The environmental sensor system can provide the environmental data (also referred to as "surrounding data") that map the surrounding area of the vehicle.
[0021] The environmental data from the environmental sensors are (pre-)processed within or outside the driver assistance system to generate the detections, such as lane polynomials.
[0022] The term "detections" as used in the present disclosure refers to (pre-)processed sensor data, not raw sensor data. In some embodiments, the (pre-)processing may be performed by the environmental sensor system, such as camera firmware (e.g., a GPU of the camera firmware).
[0023] Preferably, the path determination module is a CPU (Central Processing Unit) or is implemented by a CPU. The CPU differs from a GPU (Graphics Processing Unit) in terms of architecture.
[0024] Preferably, the control module is a trajectory controller.
[0025] Preferably, the travel path determination module is configured to input map information into the at least one neural network to be used in determining the travel path.
[0026] Preferably, the at least one neural network is configured to further determine the travel path based on odometry information of the vehicle. The odometry information may be or may be a position and orientation (attitude estimate) of the vehicle.
[0027] Preferably, the two or more lane markings include: - at least one lane marking; and / or - at least one physical boundary (e.g. curbs, crash barriers, concrete dividers, parked vehicles, grass edges, snow edges, etc.).
[0028] Preferably, the first lane is delimited on a first side thereof by a first lane boundary of the two or more lane boundaries, and on a second side opposite the first side, by a second lane boundary of the two or more lane boundaries. The first lane boundary can, for example, be a first lane marking. Additionally or alternatively, the second lane boundary can be a second lane marking.
[0029] Preferably, the detections comprise or are polynomials corresponding to the two or more lane boundaries. The polynomials may, for example, be lane marking polynomials.
[0030] In some embodiments, the travel path determination module may be configured to input coefficients of the polynomials into the at least one neural network.
[0031] In some embodiments, the travel path determination module may be configured to input distance validity information of the polynomials into the at least one neural network. The distance validity information indicates the distances in the direction of travel for which the polynomials are valid (so-called "view range").
[0032] In some embodiments, the detections corresponding to the two or more lane boundaries are polylines.
[0033] Preferably, the at least one neural network is configured to output the travel path as a polyline. However, the present disclosure is not limited to this, and other output representations are possible.
[0034] Preferably, the at least one neural network is configured to further determine a curvature of the travel path and to output it to the control module and / or at least one other control unit of the vehicle.
[0035] Preferably, the at least one neural network is configured to further determine and output a curvature of the travel path. In other words, the neural network can not only estimate the x- and y-coordinates of the points of a polyline for a travel path, but also output the curvature for the points of this polyline.
[0036] Preferably, the at least one neural network is configured to determine the driving path longer than the input detections. Since the odometry can be selected arbitrarily long in front of the vehicle during training, the neural network can learn to estimate a driving path that is longer than incoming lane detections. It can thus predict the further driving course, e.g., in the case where vehicle markings are obscured by preceding vehicles.
[0037] Preferably, the at least one neural network is configured to output multiple travel paths. This is particularly advantageous when the road topology allows for multiple possible travel paths, e.g., in the case of lane splitting.
[0038] Preferably, the at least one neural network is configured to output reference paths for neighboring lanes, in particular to enable a lane change.
[0039] Preferably, the at least one neural network is configured to determine and output an uncertainty with respect to the specific travel path. In safety-critical applications, such as an automated driving function, it is necessary for a detection function to always provide an indication of the uncertainty of these values in addition to the output values.
[0040] Preferably, the at least one neural network is trained to determine a reference path in the center of the ego lane during off-center driving. The control module can be configured to guide the vehicle to the center of the ego lane using the reference path. For this purpose, the training data, in particular the sensor detections and odometry, can be modified such that the neural network outputs the desired reference path for the trajectory controller during off-center driving.
[0041] Preferably, the path determination module is configured to input historical information into the at least one neural network, wherein the historical information comprises at least sensor information from the environmental sensor system from previous time steps. Taking the historical information into account makes it possible to smooth the detections over time and to identify and filter out incorrect measurements. In particular, the at least one neural network can learn during training to recognize and filter out certain temporal patterns in the sensor data (e.g., noise, sporadic incorrect measurements, etc.). This enables the provision of temporally stable paths despite unstable detections, e.g., in snow or heavy rain.
[0042] Preferably, the at least one neural network comprises an encoder and a decoder, wherein the decoder uses a history as input, and wherein the history comprises the information encoded by the encoder from sensor information from previous time steps. In particular, the encoded information calculated from the original sensor information from previous time steps in the first layers is used for the history.
[0043] In some embodiments, the decoder is configured to estimate lane markings, and the driver assistance system is configured to compare the estimated lane markings with lane marking detections. Through the comparison during training, the network can learn to better understand the input data. Through the comparison during real-world use, the network's functionality can be verified.
[0044] Preferably, the driver assistance system is configured to monitor outputs of the at least one neural network, comprising the driving path, using threshold values. If the threshold values are exceeded, automated driving can be aborted and the driver can be prompted to take over control of the vehicle.
[0045] According to a further independent aspect of the present disclosure, a vehicle, in particular a motor vehicle, is specified. The vehicle comprises the driver assistance system for automated driving according to the embodiments of the present disclosure.
[0046] The term "vehicle" includes cars, trucks, buses, mobile homes, motorcycles, etc., used to transport people, goods, etc. In particular, the term includes motor vehicles used to transport people.
[0047] According to a further independent aspect of the present disclosure, a driver assistance method for automated driving of a vehicle, in particular a motor vehicle, is provided. The driver assistance method comprises inputting detections from an environmental sensor system into at least one neural network, wherein the detections indicate two or more lane boundaries in a surrounding area of the vehicle, and wherein the at least one neural network is a neural network trained based on recorded training data from real manual journeys and / or simulated training data from driving simulations; determining, by the at least one neural network, a driving path based on the detections; and performing automated driving of the vehicle in an ego lane of the vehicle based on the driving path determined by the at least one neural network.
[0048] The driver assistance procedure can implement the aspects of the driver assistance system described in this document.
[0049] According to a further independent aspect of the present disclosure, a software (SW) program is provided. The SW program can be configured to be executed on one or more processors and thereby to carry out the driver assistance method described in this document for the automated driving of a vehicle.
[0050] According to a further independent aspect of the present disclosure, a storage medium is provided. The storage medium can comprise a software program configured to be executed on one or more processors and thereby to execute the driver assistance method described in this document for the automated driving of a vehicle.
[0051] According to a further independent aspect of the present disclosure, software with program code is provided. The software is configured to carry out the driver assistance method for automated driving of a vehicle when the software runs on one or more software-controlled devices.
[0052] According to a further independent aspect of the present disclosure, a driver assistance system for automated driving of a vehicle is provided. The system comprises one or more processors; and at least one memory connected to the one or more processors and containing instructions executable by the one or more processors to carry out the driver assistance method for automated driving of a vehicle described in this document.
[0053] According to a further independent aspect of the present disclosure, a method for training a neural network to output driving paths for automated driving is provided. The method comprises providing training data, wherein the training data comprises recorded training data from real manual drives and / or simulated training data from driving simulations, wherein the training data comprises sensor detections of lane boundaries and odometry information, and wherein the sensor detections indicate two or more lane boundaries; and training the neural network using the training data.
[0054] Preferably, providing the training data comprises providing the sensor detections as polynomials corresponding to the two or more lane boundaries. The polynomials may, for example, be lane marking polynomials.
[0055] Preferably, providing the training data comprises sampling the polynomials at equidistant intervals to generate a sequence of sample points that define a polyline. The points of the polylines can be specified in the vehicle's local ego coordinate system.
[0056] Alternatively, the sensor detections can be provided as polylines. In this case, providing the training data involves interpolating the polylines at equidistant intervals to generate a sequence of sample points.
[0057] Preferably, the odometry information includes coordinate values (e.g. y-values) and a curvature.
[0058] Preferably, a section of odometry information is converted from a global coordinate system into a local ego coordinate system using a timestamp comparison and a coordinate transformation.
[0059] Preferably, providing the training data comprises generating augmented training data from the recorded training data and / or simulated training data, wherein the augmented training data indicates off-center driving in a lane. This allows the vehicle to be returned to the center of the ego lane during off-center driving.
[0060] In some embodiments, generating the augmented training data includes generating lateral shifts and / or rotations and / or reflections and / or track width changes and / or shortenings of the sensor detections in the recorded training data and / or simulated training data. For example, the input values for the neural network can be changed in constant steps within a certain range. However, a certain number of randomly selected values can also be used within such a range.
[0061] In some embodiments, generating the augmented training data includes adding artificial noise to the recorded training data and / or simulated training data, such as a normal distribution with a certain standard deviation. This can, for example, simulate adverse effects on the camera(s) caused by bad weather with heavy precipitation.
[0062] In some embodiments, generating the augmented training data comprises generating artificial false detections in the recorded training data and / or simulated training data. The artificial false detections can be generated, for example, by altering values of sensor detections in a sufficiently significant manner. For example, it can be simulated that the camera incorrectly detects a tar seam, which is located to the side of the correct lane marking, as a lane marking for a certain period of time instead of the correct lane marking. This can be achieved by shifting the detection of a lane marking by a certain amount (e.g., 40 cm) within a certain time range (e.g., 1 second).
[0063] Preferably, the neural network is further trained using digital map data. For example, a distance to the next intersection, lane split, or lane merge can be determined from a navigation map and fed to the neural network. From this information, the neural network can learn not to use certain sensor detections near intersections, lane splits, or lane merges when estimating the path, as they may be inaccurate. In another example, elevation information can be used for training. The neural network can thus learn to better determine the driving paths, for example, if the camera's sensor detections are deformed due to perspective distortions when driving over hills and valleys and do not correspond to the actual course of the lane markings.
[0064] Preferably, the selection of training data is automated through a target-actual comparison and a similarity search.
[0065] According to a further independent aspect of the present disclosure, a software (SW) program is provided. The SW program can be configured to be executed on one or more processors and thereby to carry out the method described in this document for training a neural network to output driving paths for automated driving.
[0066] According to a further independent aspect of the present disclosure, a storage medium is provided. The storage medium can comprise a software program configured to be executed on one or more processors and thereby to execute the method described in this document for training a neural network to output driving paths for automated driving.
[0067] According to a further independent aspect of the present disclosure, software with program code is provided. The software is configured to carry out the method for training a neural network to output driving paths for automated driving when the software runs on one or more software-controlled devices.
[0068] According to a further independent aspect of the present disclosure, a system for training neural networks to output driving paths for automated driving is provided. The system comprises one or more processors; and at least one memory connected to the one or more processors and containing instructions executable by the one or more processors to carry out the method described in this document for training a neural network to output driving paths for automated driving.
[0069] A processor or processor module is a programmable computing unit, i.e. a machine or an electronic circuit that controls other elements according to given instructions and thereby drives an algorithm (process). Short description of the drawings
[0070] Embodiments of the disclosure are illustrated in the figures and are described in more detail below. They show: Fig. 1 schematically shows a track widening; Fig. 2 schematically shows a vehicle with a driver assistance system for automated driving according to embodiments of the present disclosure, Fig. 3 schematically shows a vehicle with a driver assistance system for automated driving according to further embodiments of the present disclosure, Fig. 4 schematically shows lane markings and an odometry according to embodiments of the present disclosure, Fig. 5 schematically shows lane markings and an odometry according to further embodiments of the present disclosure, Fig. 6 schematically shows a learned off-center travel path and a reference travel path for returning the vehicle to the center of the lane, Fig. 7 schematically recorded training data from real manual rides, Fig. 8 schematically augmented data generated from recorded training data from real manual rides, Fig. 9 schematically shows inputs to a neural network and an output of the neural network according to embodiments of the present disclosure, Fig. 10 schematically shows a neural network with encoder and decoder, Fig. 11 schematically shows a neural network with encoder and decoder according to embodiments of the present disclosure, and Fig. 12 is a flowchart of a driver assistance method for automated driving of a vehicle according to embodiments of the present disclosure. Embodiments of the disclosure
[0071] In the following, unless otherwise stated, the same reference symbols are used for identical and equivalent elements.
[0072] Fig. 2 schematically shows a vehicle 10 with a driver assistance system 200 for automated driving according to embodiments of the present disclosure.
[0073] In automated driving, the longitudinal and / or lateral guidance of the vehicle 10 is automatic. The driver assistance system 200 thus assumes vehicle guidance. To this end, the driver assistance system 200 controls the drive 20, the transmission 22, the (e.g., hydraulic) service brake 24, and the steering 26 via intermediate units (not shown).
[0074] To plan and implement automated driving, the driver assistance system 200 receives environmental information from an environmental sensor system 12 that monitors the vehicle's surroundings. In particular, the environmental sensor system 12 can comprise at least one environmental sensor configured to record environmental data indicating the vehicle's surroundings. The at least one environmental sensor can comprise, for example, one or more LiDAR systems, one or more radar systems, one or more ultrasonic sensors, and / or one or more cameras.
[0075] Automated driving occurs along a driving path that is continuously determined by the driver assistance system 200 based on the environmental data from the environmental sensor system 12 and other information, such as digital map data. The driving path (also referred to as a "driving trajectory") can be understood as a prediction of the vehicle's ego-travel path. The driving path corresponds to (or is a) computationally defined driving corridor or driving path, the boundaries of which define a space through which the vehicle can move safely, e.g., along a lane.
[0076] According to the invention, the driver assistance system 200 uses at least one trained neural network to estimate the driving path, which mimics human behavior. This is described in detail below.
[0077] Fig. 3 schematically shows a vehicle 10 with a driver assistance system 200 for automated driving according to embodiments of the present disclosure.
[0078] The driver assistance system 200 includes a driving path determination module 210 configured to determine a driving path FAP. The driver assistance system 200 further includes a control module 220 configured to perform automated driving of the vehicle 10 in an ego lane of the vehicle 10 based on the driving path FAP determined by the driving path determination module 210.
[0079] The travel path determination module 210 and the control module 220 can be implemented in a common software and / or hardware module. Alternatively, the travel path determination module 210 and the control module 220 can each be implemented in separate software and / or hardware modules.
[0080] In a preferred embodiment, the travel path determination module 210 is a CPU (Central Processing Unit) or is implemented by a CPU. The CPU differs from a GPU (Graphics Processing Unit) in terms of architecture.
[0081] In a preferred embodiment, the control module 220 is a trajectory controller.
[0082] The travel path determination module 210 comprises (or implements) at least one neural network for estimating the travel path, for example in a corresponding software component or a corresponding software module.
[0083] Neural networks consist of a series of processing units connected to each other via communication channels, called neurons. The neurons process input data and pass the resulting processing results on to other neurons via the communication channels. Parallel data processing, enabled by the interconnection of neurons and their processing functions, allows complex, nonlinear dependencies in the input data to be mapped. Neural networks are trained using a suitable learning technique, such as machine learning, to learn these dependencies. Neural networks can be trained using empirical data, known as training data.
[0084] The driving path determination module 210 or the at least one neural network uses detections DET from an environmental sensor system 30 of the vehicle 10, such as detections from image data from a camera, as input data. The detections DET indicate two or more lane boundaries in an area surrounding the vehicle 10. The at least one neural network processes the detections DET and further information (e.g., odometry information ODO, digital map data, etc.) to determine the driving path FAP and output it to the control module 120.
[0085] Detections generally refer to (pre-)processed sensor data, thus, they are to be understood as distinct from sensor data or raw sensor data. In some embodiments, the (pre-)processing can be performed by the environmental sensor system, such as camera firmware (e.g., a GPU of the camera firmware). The detections DET can be, for example, lane polynomials, but are not limited to them.
[0086] In the following, the terms “detections” and “sensor detections” are used synonymously.
[0087] The at least one neural network is trained based on recorded training data from real manual driving and / or simulated training data from driving simulations. This enables imitation of human behavior, so that, for example, vehicle guidance along the Fig. The straight line GL shown in Figure 1 occurs instead of the curved center line BL.
[0088] The training data recorded during real manual (or manually driven) journeys may, in some embodiments, include or be sensor detections of lane markings and human-driven paths (in particular, odometry ODO). This data, collected during journeys of test vehicles and / or a vehicle fleet, may be available, for example, in automobile manufacturers' data centers.
[0089] In a further embodiment, the training data may include or be generated or simulated training data. Here, the sensor detections and odometry can be obtained from a driving simulation and used to train the at least one neural network of the driver assistance system 200.
[0090] In some embodiments, the training data may also include or be a combination of recorded training data and simulated training data.
[0091] According to the invention, the driver assistance system 200 uses at least one trained neural network to estimate the driving path FAP, which mimics human behavior. The at least one neural network uses detections DET, i.e., preprocessed sensor information, and optionally further information, such as odometry information ODO and information from a navigation map, to estimate the driving path FAP. Thus, no sensor data, such as pixel data, are input into the at least one neural network, so that fewer computing resources are required for estimating the driving path FAP. This allows, for example, a CPU to be used instead of an expensive GPU for estimating the driving path FAP.
[0092] Detailed embodiments of the general inventive concept described above are explained below.
[0093] Suitable training data are provided for training the at least one neural network. Referring to the Fig. 4, the training data includes, for example, recorded sensor detections of lane markings (e.g., lane marking polynomials) and paths driven by humans (odometry). These data, recorded during journeys of test vehicles and / or a vehicle fleet, are available, for example, in data centers of automobile manufacturers.
[0094] In one embodiment, only the sensor detections from the nearest left and right lane markings are used. Optionally, sensor detections from other and / or additional lane markings may be used, such as from adjacent lanes and / or camera-detected road boundaries such as curbs, guardrails, concrete dividers, parked vehicles, grass edges, and / or snow edges.
[0095] The training data can come not only from recorded data in vehicles but also be generated entirely artificially. In this case, sensor detections and odometry can be obtained from a driving simulation. The training data can also consist of a combination of recorded data and simulated data.
[0096] The neural network is now trained so that, upon receiving sensor detections (e.g., lane marking polynomials), it outputs a driving path that corresponds to the path driven by a human in a similar situation (odometry). The vehicle's trajectory controller then uses this driving path as a reference path to control the vehicle.
[0097] The data of the lane marking polynomials, such as the coefficients, can be fed directly to the neural network, for example together with information about the distances in the direction of travel for which the polynomials are valid (so-called “view range”).
[0098] In some embodiments, during training, each received polynomial of the detected lane markers may be sampled at equidistant intervals. This results in polylines (ie, a sequence of points). Fig. 4 are sample points SP-SM of the lane markings. The points of the polylines are specified in the local ego coordinate system of vehicle 10 (e.g., x, y in ego coordinates).
[0099] Using a timestamp comparison, the location and direction of the vehicle at the time of sensor detection can be determined from the traveled path (odometry). Then, a section of the traveled path, which consists, for example, of xy coordinates in a global coordinate system, is converted into the local ego coordinate system of the vehicle 10. This results in a polyline of the odometry in the ego coordinate system. In one implementation, sample points SP-ODO of the odometry can be taken at equidistant intervals along this transformed polyline, as shown in the Fig. 4 is shown as an example.
[0100] The sample points SP-SM of the lane markings and the interpolated sample points SP-ODO of the odometry are then fed to the neural network as training data.
[0101] Referring to Table 1, the training data can be divided into two categories: first data, which is also available for later use, and second data, which is only available during training. The first data represents the so-called inputs, in this example, the points LSM of the left lane marking and the points RSM of the right lane marking (specified as the y-coordinate with a fixed x-coordinate). The second data represents the so-called labels, in this example, the odometry (y-value ODO and odometry curvature ODO-K).
[0102] The task of the neural network is to estimate the missing label at the time of use based on the available inputs, i.e., the output. In this case, the future odometry is estimated based on the lane detections, i.e., the driving path FAP and optionally the curvature KR, as shown in Table 1 below: Table 1 INPUT LABEL OUTPUT LSM y EX RSM y EX ODO y ODO-K FAP y KR 2 1 -2,2 1 -0,1 0,01 -0,1 0,01 2,1 1 -2,1 1 0 0 0 0 2,2 1 -2 1 0,1 -0,01 0,1 -0,01 2 1 0 0 0,1 0 0,1 0 0 0 0 0 0,1 -0,01 0,1 -0,01 0 0 0 0 0,1 -0,01 0,1 -0,01 0 0 0 0 0,1 0 0,1 0 0 0 0 0 0,2 0,01 0,2 0,01 0 0 0 0 0,2 0,01 0,2 0,01 0 0 0 0 0,2 0,01 0,2 0,01
[0103] The output of the neural network may be a polyline describing a driving path. However, the present disclosure is not limited to this, and other output representations are possible.
[0104] If the x-coordinates of the sample points are fixed and known, it is sufficient to feed only the y-coordinates of the lane markings and the odometry to the neural network. In this case, only the y-coordinates of the estimated odometry (FAP) are output.
[0105] In a further embodiment, however, the x-coordinates can additionally be supplied to the neural network and the x-coordinates of the estimated travel path can also be obtained.
[0106] Since the polylines of the lane markers can have different lengths at different times, but the number of inputs for the neural network is usually constant, in some embodiments of the present disclosure, a constant number of x and y coordinates for a lane marker can always be input to the neural network. If the detection time is shorter, unoccupied points are filled with 0. However, each point also receives an indication of whether it is filled or not (existence EX in Table 1). This information is fed to the neural network, so that the neural network receives a constant number of points at the input on the one hand, but is also informed of the length of the detected lane markers on the other.
[0107] Since the odometry can be selected any distance ahead of the vehicle during training, the neural network can learn to estimate a driving path that is longer than incoming lane detections. It can thus predict the further driving course, for example, in the case where vehicle markings are obscured by preceding vehicles. Fig. Figure 5 shows an example of a forecasted area PB.
[0108] However, in the predicted area PB, the output values are uncertain. Using suitable techniques, it is possible to also provide information about the uncertainty of the output values (polyline) (e.g., Bayesian neural network, ensemble, etc.). In one embodiment, the polyline, which is passed to the subsequent control components as a driving path, can also be shortened to the length of the incoming lane marking detections, i.e., no prediction is performed.
[0109] In some embodiments, in addition to the odometry data (x and y coordinates), curvature information is also available at these coordinates. This curvature information can also be used when training the neural network. The neural network can then not only estimate the x and y coordinates of the points of a polyline for a driving path, but also output the curvature for the points of this polyline. This can be utilized by some downstream control components.
[0110] The neural network trained using the method described above learns to output driving paths that run directly beneath the vehicle. When driving off-center, i.e., when the vehicle is not in the center of the lane, the trajectory controller requires a reference driving path located in the center of the lane to steer the vehicle back to the center of the lane. Fig. 6 shows, by way of example, a learned travel path GFAP which runs through the center of the vehicle 10 and a reference travel path RFAP for steering the vehicle 10 back to the center of the lane.
[0111] In one embodiment of the present disclosure, the neural network is therefore configured to estimate a path during off-center driving, which path can be used by the trajectory controller as a reference driving path RFAP to steer the vehicle 10 back to the center of the lane. For this purpose, the training data, in particular the sensor detections and odometry, can be modified such that the neural network outputs the desired reference driving path RFAP for the trajectory controller during off-center driving.
[0112] For example, when training the neural network, additional training data can be artificially generated in addition to the training data originally recorded during a vehicle journey, simulating off-center driving (so-called augmented data). The augmented data can be generated by augmenting the originally recorded training data ( Fig. 7) can be moved laterally by various offsets OFS ( Fig. 8), e.g. in the range from -3m to +3m and / or in steps of 20cm.
[0113] With such an expansion of the training data, not only lateral shifts can be used, but additionally or alternatively one or more further transformations, such as rotations (different angles, e.g. -90 degrees to +90 degrees in 10 degree steps), reflections, lane width changes (e.g. -2 m to +2 m in 20 cm steps) and / or shortening of the detection of the lane markings (e.g. 0 m to - 20 m in 2 m steps).
[0114] The input values for the neural network can be varied in constant steps within a certain range, as described above, e.g., shifting a lane detection point in the range from -3 m to +3 m in steps of 20 cm. However, a certain number of randomly selected values can also be used within such a range, such as 10 values evenly distributed from the range from -3 m to +3 m. This can increase efficiency with many different expansions.
[0115] A further expansion can add artificial noise to the sensor detection values, such as a normal distribution with a certain standard deviation. This can be used to simulate, for example, adverse weather conditions such as heavy precipitation affecting the camera(s).
[0116] In addition, artificial misdetections (so-called outliers) can be created by altering sensor detection values sufficiently. For example, it can be simulated that the camera incorrectly detects a tar seam located to the side of the correct lane marking as a lane marking for a certain period of time instead of the correct lane marking. This can be achieved by shifting the detection of a lane marking by a certain amount (e.g., 40 cm) over a certain time period (e.g., 1 second).
[0117] Since the sizes of the expansions are known, they can be fed to the neural network as additional labels during training and estimated as additional outputs at the time of use. This allows the neural network to estimate, for example, the lateral offset of the vehicle from the lane center during off-center driving and / or, for example, standard deviations for the y-values of the estimated driving path.
[0118] Using expansions or augmentations as additional labels, as well as outputs, makes it possible to verify the behavior of the neural network. For example, it can be verified whether the neural network correctly estimates a lateral offset of the driving path when the input data was shifted laterally during training. This increases insight into the behavior of the neural network (so-called Explainable AI).
[0119] In safety-critical applications, such as automated driving, it is necessary for a detection function to always provide, in addition to the output values, an indication of the uncertainty of these values. In the case of the present disclosure, this can be achieved by estimating the augmented sensor noise, but also, as mentioned above, by using certain network techniques (e.g., Bayesian networks, ensembles, etc.).
[0120] Sensor detections may be noisy and temporally unstable. Therefore, it is desirable to temporally smooth the sensor detections and to identify and filter out incorrect measurements. Referring to the Fig. 9 this can be achieved by supplying the neural network NN with not only the current sensor information and labels LAB for a time step to estimate the path, but also with sensor information from the previous time steps or a history HIST.
[0121] During training, the neural network can then learn to recognize and filter out certain temporal patterns in the sensor data (e.g., noise, sporadic incorrect measurements, etc.). This enables the provision of temporally stable driving paths (output OUT in Fig. 9) despite unstable sensor detections, e.g., in snow or heavy rain. Network architectures exist that are particularly well-suited for such a procedure, such as convolutional neural networks (CNNs) and attention networks.
[0122] A neural network usually consists of different parts that follow one another during processing. Fig. 10, a first part in this sequence extracts the relevant information from the set of all information (encoder ENC; input: features FEA and labels LAB), while the subsequent part serves to convert the relevant information RIN (hidden state) into the desired output representation OUT (decoder DEC), as in the present case into a line that describes a travel path.
[0123] In one embodiment of the present disclosure, instead of the sensor information of previous time steps, the history HIS is not based on the original incoming sensor information, but on the encoded information calculated therefrom in the first layers, as in the Fig. 11 is shown.
[0124] The decoder DEC can be designed to estimate not only future odometry (driving path) but also lane markings. Such a design is called an autoencoder. The estimated lane markings can be compared with the detected lane markings. By using the comparison during training, the network can learn to better understand the input data (encoding the relevant information). By comparing them during use, the network's functionality can be verified. If the estimated lane markings differ from the detections, this is an indication that the neural network is not interpreting the situation correctly. In this case, the automated vehicle's control system can be configured to disregard the estimated driving path and prompt the driver to take over steering. This serves to ensure vehicle safety.
[0125] The network's initial values can be influenced during training (so-called regularization). This regularization can be used to limit the initial values. For example, appropriate regularization can ensure that the estimated paths exhibit realistic shapes that occur in typical lane geometries. This makes it possible to avoid misestimations by the network as much as possible.
[0126] Since a neural network is a highly nonlinear function and misestimations can therefore never be ruled out, some embodiments of the present disclosure can monitor the estimated output values using thresholds. If the thresholds are exceeded, automated driving is aborted and the driver is prompted to take over control of the vehicle.
[0127] When training the neural network, not only camera detections of lane markings can be used, but also all other existing sensor detections from cameras or other sensors (e.g. lidar, radar and / or ultrasound).
[0128] The neural network can be trained not only with sensor detections but also with other information available in the vehicle. For example, detections of lane markings near intersections, lane widenings, or lane narrowings are often inaccurate. For example, the distance to the next intersection, lane split, or lane merge can be determined from a navigation map and fed into the neural network. From this information, the neural network can learn to exclude certain sensor detections near intersections, lane splits, or lane merges from path estimation, as they may be inaccurate.
[0129] If the map or odometry data contains elevation information, this can be additionally fed into the neural network. The neural network can then learn to better determine the driving paths, for example, if the camera's sensor detections are distorted due to perspective distortions when driving over hills and valleys and do not correspond to the actual course of the lane markings.
[0130] Further information to filter out false detections is, for example, strong windshield wiper activity, which may indicate heavy rain, and in this case the sensor information provided by the camera may be incorrect or unstable.
[0131] Other helpful information for estimating a correct route can be the navigation destination entered on the navigation device or the driver's activation of the turn signal.
[0132] Some downstream control components require more than just one path. Lane splits result in multiple possible paths (left, straight ahead, right) in the direction of travel. In this case, the neural network can be designed and trained to output multiple possible paths. These paths are either always output simultaneously, or the network receives an additional input (selector) that tells it which of several possible paths to output. By iterating over the possible values of the selector, all possible paths can again be obtained at the output, only this time one after the other.
[0133] When training the neural network, it is advantageous to avoid using training data in which the driver's trajectory (odometry) does not correspond to the desired target behavior, such as lane changes, weaving, or off-center driving. These intervals can be filtered out of the recorded data and not included in the training data.
[0134] When training a neural network, the selection of training data is a key factor. Ideally, training data should be provided from which the neural network learns particularly well. The training data can be selected manually by the developer or automatically based on a target-actual comparison and a similarity search (so-called active learning). For this purpose, after training the network, the network's output (estimated paths) can be determined based on a test dataset that was not used during training. The estimated paths are compared with the target trajectory (label). If the deviations are too large, the following cases can be distinguished: • The label (odometry) is unsuitable. In this case, this dataset is removed from the training data or the odometry is corrected. • The label (odometry) is correct. The neural network generated an incorrect output. In this case, data from similar driving situations are selected from the recorded data and added to the training dataset. This allows the network to learn to deliver a correct output in this situation.
[0135] When training or checking the output, not only the trajectory driven by the driver (odometry) can be used as a label, but also a trajectory created manually by a person using a suitable tool (labeling tool).
[0136] To learn driving paths that resemble human behavior, not only a neural network but also another algorithm from the field of machine learning, such as a Generalized Additive Model (GAM), can be used as the learning algorithm. A GAM is one of the interpretable algorithms. In these algorithms, there is a direct connection between input and output data. This has the advantage that, in the event of an incorrect output behavior, it is immediately apparent which value of an input variable led to the observed behavior at the output.
[0137] In summary, the embodiments of the present disclosure provide a reference path for trajectory control that, on the one hand, mimics human behavior to avoid unnatural driving paths, but on the other hand, can also be used to steer the vehicle back to the center of the road when driving off-center. No access to camera pixel information is required; preprocessed sensor detections and other information available in most vehicle types are used.
[0138] Fig. Figure 12 schematically shows a flowchart of a driver assistance method 1200 for automated driving of a vehicle according to embodiments of the present disclosure. The driver assistance method 1200 can be implemented by appropriate software executable by one or more processors (e.g., a CPU).
[0139] The driver assistance method 1200 comprises, in block 1210, inputting detections from an environmental sensor system into at least one neural network, wherein the detections indicate two or more lane boundaries in an area surrounding the vehicle, and wherein the at least one neural network is a neural network trained based on recorded training data from real manual journeys and / or simulated training data from driving simulations; in block 1220, determining, by the at least one neural network, a driving path based on the detections; and in block 1230, performing automated driving of the vehicle in an ego lane of the vehicle based on the driving path determined by the at least one neural network.
[0140] According to the invention, the driver assistance system uses at least one trained neural network to estimate the driving path, which mimics human behavior. The at least one neural network uses direct detections, i.e., preprocessed sensor information, and optionally further information, such as odometry information and information from a navigation map, to estimate the driving path. The detections can, for example, be lane polynomials. This means that no sensor data, such as pixel data, is input into the at least one neural network, so that fewer computing resources are required for estimating the driving path. This allows, for example, a CPU to be used instead of an expensive GPU to estimate the driving path.
[0141] Although the invention has been illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned by way of example are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.
Claims
[1] Driver assistance system (200) for automated driving of a vehicle (10), comprising: a driving path determination module (210) with at least one neural network (NN), wherein the driving path determination module (210) is configured to input detections (DET) of an environmental sensor system (12, 30) into the at least one neural network (NN), wherein the detections (DET) indicate two or more lane boundaries (SM) in a surrounding area of the vehicle (10), wherein the at least one neural network (NN) is configured to determine and output a driving path (FAP) based on the detections (DET), and wherein the at least one neural network (NN) is a neural network trained based on (i) recorded training data from real manual trips and / or (ii) simulated training data from driving simulations; and a control module (220) configured to perform automated driving of the vehicle (10) on an ego lane of the vehicle (10) based on the driving path (FAP) determined by the driving path determination module (210). [2] Driver assistance system (200) according to claim 1, wherein the at least one neural network (NN) is configured to further determine the travel path (FAP) based on odometry information (ODO) of the vehicle (10). [3] Driver assistance system (200) according to claim 1 or 2, wherein the two or more lane boundaries (SM) comprise: - at least one lane marking; and / or - at least a physical limitation. [4] Driver assistance system (200) according to one of claims 1 to 3, wherein the ego lane is delimited on a first side thereof by a first lane boundary of the two or more lane boundaries and is delimited on a second side opposite the first side by a second lane boundary of the two or more lane boundaries, in particular wherein the first lane boundary is a first lane marking and / or the second lane boundary is a second lane marking. [5] Driver assistance system (200) according to one of claims 1 to 4, wherein the detections (DET) comprise or are polynomials corresponding to the two or more lane boundaries (SM), in particular wherein: - the travel path determination module (210) is configured to input coefficients of the polynomials into the at least one neural network (NN); and or - the travel path determination module (210) is configured to input distance validity information of the polynomials into the at least one neural network (NN). [6] Driver assistance system (200) according to one of claims 1 to 4, wherein the detections (DET) comprise or are polylines. [7] Driver assistance system (200) according to one of claims 1 to 5, wherein the driving path determination module (210) is configured to input map information into the at least one neural network (NN) that is used in determining the driving path (FAP). [8] Driver assistance system (200) according to one of claims 1 to 6, wherein the at least one neural network (NN) is configured to: - output the driving path (FAP) as a polyline; and / or - further determine and output a curvature of the travel path (FAP); and / or - to determine the travel path (FAP) longer than the entered detections (DET); and / or - to output multiple travel paths (FAP); and / or - to output reference paths for adjacent lanes, in particular to enable a lane change; and / or - to determine and output an uncertainty related to the specific travel path (FAP). [9] Driver assistance system (200) according to one of claims 1 to 7, wherein the at least one neural network (NN) is trained to determine a reference travel path (RFAP) in a center of the ego lane during off-center driving, and wherein the control module (220) is configured to guide the vehicle (10) into the center of the ego lane using the reference travel path (RFAP). [10] Driver assistance system (200) according to one of claims 1 to 8, wherein the driving path determination module (210) is configured to input history information (HIST) into the at least one neural network (NN), wherein the history information (HIST) comprises at least sensor information of the environmental sensor system (12, 30) from previous time steps. [11] Driver assistance system (200) according to one of claims 1 to 9, wherein the at least one neural network (NN) comprises an encoder (ENC) and a decoder (DEC), wherein the decoder (DEC) uses a history (HIST) as input, and wherein the history (HIST) comprises the information encoded by the encoder (ENC) from sensor information of previous time steps. [12] Driver assistance system (200) according to claim 10, wherein the decoder (DEC) is configured to estimate lane markings, wherein the driver assistance system (200) is configured to compare the estimated lane markings with detections (DET) of lane markings. [13] Driver assistance system (200) according to one of claims 1 to 11, wherein the driver assistance system (200) is configured to monitor outputs of the at least one neural network (NN) comprising the driving path (FAP) using limit values. [14] Vehicle (10), in particular a motor vehicle, comprising the driver assistance system (200) according to one of claims 1 to 12. [15] Driver assistance method (1200) for automated driving of a vehicle, comprising: Inputting (1210) detections of an environmental sensor system into at least one neural network, wherein the detections indicate two or more lane boundaries in an area surrounding the vehicle, and wherein the at least one neural network is a neural network trained based on recorded training data from real manual drives and / or simulated training data from driving simulations; Determining (1220), by the at least one neural network, a travel path based on the detections; and Carrying out (1230) an automated driving of the vehicle on an ego lane of the vehicle based on the driving path determined by the at least one neural network. [16] A method for training a neural network to output driving paths for automated driving, comprising: Providing training data, wherein the training data comprises recorded training data from real manual drives and / or simulated training data from driving simulations, wherein the training data comprises sensor detections of lane boundaries and odometry information, and wherein the sensor detections indicate two or more lane boundaries; and Training the neural network using the training data. [17] The method of claim 15, wherein providing the training data comprises: - Providing the sensor detections as polynomials corresponding to the two or more lane boundaries; and - Sampling the polynomials at equidistant intervals to generate a sequence of sample points specifying a polyline. [18] The method of claim 15 or 16, wherein providing the training data comprises: - Providing the sensor detections as polylines; and - Sampling the polylines at equidistant intervals using interpolation. [19] A method according to any one of claims 15 to 17, wherein the odometry information comprises coordinate values and a curvature. [20] Method according to one of claims 15 to 18, wherein a section of odometry information is converted from a global coordinate system into a local ego coordinate system via a time stamp comparison and a coordinate transformation. [21] The method of any one of claims 15 to 19, wherein providing the training data comprises generating augmented training data from the recorded training data and / or simulated training data, the augmented training data indicating off-center driving in a lane. [22] The method of claim 20, wherein generating the augmented training data comprises: - generating lateral displacements and / or rotations and / or reflections and / or track width changes and / or shortenings of the sensor detections in the recorded training data and / or simulated training data; and / or - Adding artificial noise to the recorded training data and / or simulated training data; and / or - Creating artificial false detections in the recorded training data and / or simulated training data. [23] Method according to one of claims 15 to 21, wherein the training of the neural network is further carried out using digital map data. [24] Method according to one of claims 15 to 22, wherein the selection of the training data is carried out automatically by a target-actual comparison and a similarity search. [25] System for training neural networks to output driving paths for automated driving, comprising: one or more processors; and at least one memory connected to the one or more processors and containing instructions executable by the one or more processors to perform the method for training neural networks to output driving paths for automated driving according to any one of claims 15 to 23.
Citation Information
Patent Citations
BEHAVIOR-BASED PATH PLANNING IN AUTONOMOUS MACHINE APPLICATIONS
DE102019113114A1
Cited By
Method and device for detecting deviations in lane recognition of a vehicle's lane
DE102024135384A1
Method for generating a training data set, method for training a model to predict lane changes of vehicles, method and device for controlling a vehicle based on the trained model
DE102025103162A1
