Method and system for predicting the trajectory of a target vehicle in a vehicle's environment

Abstract

Method for predicting the trajectory of a target vehicle in a vehicle's environment, comprising the steps, a) Detection of the state of the target vehicle, detection of the state of other vehicle objects in the vicinity of the vehicle and detection of road markings using a camera-based detection device (2), b) Preprocessing the data obtained in step a), removing outliers and calculating missing states, c) Calculating an estimated trajectory using a physical model based on the data preprocessed in step b), d) Calculating a driver behavior-based trajectory based on the data preprocessed in step b), e) Combining the trajectories calculated in steps c) and d) to form a predicted trajectory of the target vehicle, wherein, in the preprocessing of the data obtained in step a), the road course is estimated based on measured states of the target vehicle as well as other vehicle objects in the environment and the vehicle itself, and based on the road markings captured by the camera-based detection device (2) at a defined sampling rate, wherein the road markings captured by the camera-based detection device (2) are provided to an algorithm for extracting the road course, which is configured to estimate the road course from the road markings captured by the camera-based detection device (2) by generating a movable window (101) with a defined width and, for each time sample and for a given window (101), converting the lines (102, 102a-102e) captured by the camera-based detection device (2), which represent the road markings, into an image (103),wherein a histogram is calculated from the image (103) so that the image position where most of the captured lines (102, 102a-102e) are located is determined, and a point is created there, and wherein, after determining all points in each of the windows (101), in a subsequent step the actual road course is estimated by fitting a spline to all points.

Description

[0001] The present invention relates to a method for predicting the trajectory of a target vehicle in a vehicle's environment. Furthermore, the invention relates to a system for carrying out such a method.

[0002] Modern motor vehicles often feature advanced driver assistance systems (ADAS), such as adaptive cruise control, collision warning systems, emergency braking systems, highway driving assistants, or traffic jam assist systems, to reduce the risk of vehicle collisions and increase driving comfort. These driver assistance systems, which operate semi-autonomously or autonomously, require the prediction of a target vehicle's trajectory within the vehicle's surroundings. This trajectory prediction serves, for example, as the basis for functional decisions, path planning, or braking interventions. A predicted trajectory typically consists of a number N of predicted states of the target vehicle in two dimensions, such as its position, velocity, yaw, and yaw rate.

[0003] The methods known from the state of the art for predicting the trajectory of a target vehicle in a vehicle's environment use very different physical models or driving maneuver-based models, which require relatively complex adjustments.

[0004] From EP 3 467 799 A1, a method for predicting the trajectory of a target vehicle in the vicinity of another vehicle is known, in which the vehicle type of a vehicle moving along a lane of a road is detected and motion prediction information is generated to predict the movement of the target vehicle based on the vehicle type of the target vehicle. The movement is then assigned to the lane.

[0005] A method for predicting the trajectory of a target vehicle in a vehicle's environment is known from DE 10 2012 009 555 A1. This method comprises the following steps: a) Detecting the state of the target vehicle, detecting the state of other vehicle objects in the vicinity of the vehicle, and detecting road markings using a camera-based detection device, b) Preprocessing the data obtained in step a), removing outliers and calculating missing states, c) Calculating an estimated trajectory using a physical model based on the data preprocessed in step b), d) Calculating a driver behavior-based trajectory based on the data preprocessed in step b), and e) Combining the trajectories calculated in steps c) and d) to form a predicted trajectory of the target vehicle.

[0006] German patent DE 10 2018 008 599 A1 discloses a control system for use in a motor vehicle, which is designed and intended to determine a trajectory for another motor vehicle based on environmental data obtained from at least one environmental sensor located on the vehicle. The at least one environmental sensor is designed to provide an electronic control unit of the control system with lane markings, lane boundaries, lane geometries, and / or driving-related information about other motor vehicles in an area in front of, to the side of, and / or behind the vehicle.The control system is at least designed and intended to determine a property of a future driving maneuver of the other motor vehicle based on the provided environmental data, to determine a temporal information component based on the provided environmental data, and to determine the trajectory for the other motor vehicle based on the property of the future driving maneuver and the temporal information component.

[0007] DE 10 2004 003 850 A1 discloses a method for detecting markings on a roadway based on distance images of a detection area intersected by a roadway surface, captured sequentially by at least one electromagnetic radiation sensor, in particular a laser scanner, mounted on a vehicle. The method identifies distance image points corresponding to at least one of the markings in at least one of the distance images. Furthermore, the position and / or shape of the marking is estimated based on these distance image points.

[0008] One challenge in predicting the trajectory of a target vehicle is to use sensor signals and measurement data from a camera-based detection system to predict the road's path and the target vehicle's states, particularly its position, speed, yaw, and yaw rate, for up to five seconds in advance. Furthermore, the trajectory prediction method should be as robust as possible in a vehicle's environment to minimize the influence of outliers caused, for example, by random or systematic errors in the measured states, especially the target vehicle's position and speed.Starting from this, the present invention aims to provide a further improved method and a system for predicting the trajectory of a target vehicle in a vehicle's environment.

[0009] The solution to this problem is provided by a method for predicting the trajectory of a target vehicle in the environment of a vehicle having the features of claim 1 and a system having the features of claim 7. The dependent claims relate to advantageous embodiments of the invention.

[0010] A generic method for predicting the trajectory of a target vehicle in a vehicle's environment comprises the following steps: a) Detecting the state of the target vehicle, detecting the state of other vehicle objects in the vicinity of the vehicle, and detecting road markings using a camera-based detection device, b) Preprocessing the data obtained in step a), removing outliers and calculating missing states, c) Calculating an estimated trajectory using a physical model based on the data preprocessed in step b), d) Calculating a driver behavior-based trajectory based on the data preprocessed in step b), e) Combining the trajectories calculated in steps c) and d) to form a predicted trajectory of the target vehicle.

[0011] Furthermore, it is provided that during the preprocessing of the data obtained in step a), the road course is estimated based on measured states of the target vehicle as well as other vehicle objects in the environment and the vehicle itself, and based on the road markings captured by the camera-based detection device at a defined sampling rate. The road markings captured by the camera-based detection device are provided to an algorithm for extracting the road course. This algorithm is designed to estimate the road course from the road markings captured by the camera-based detection device by creating a movable window with a defined width and converting the lines representing the road markings captured by the camera-based detection device into an image for each time sample and for a predefined window.where a histogram is calculated from the image to determine the image position where most of the captured lines are located, and a point is created there; and where, after determining all points in each of the windows, the actual road alignment is estimated in a subsequent step by fitting a spline to all points. A spline is a polygon of degree n. The output of this algorithm is an estimated road alignment.

[0012] The method according to the invention makes it possible to predict the states of the target vehicle, in particular its position, speed, yaw, and yaw rate, preferably for a period of up to five seconds in advance, based on the data provided by the camera-based acquisition device. Furthermore, a robust method for predicting the trajectory of a target vehicle in its environment is provided in order to minimize the influence of outliers that may be caused, for example, by random or systematic errors in the measured states, especially in the positions and speeds of the target vehicle. The prediction of the target vehicle's trajectory can advantageously take into account physical and environmental aspects of the respective driving situation and weight them accordingly.

[0013] The algorithm used in the present invention for extracting the road course provides a novel approach to extract an estimate of the actual road course, in particular the number of lanes and their shape, from camera recordings of the camera-based detection device using inaccurate measurements of the road markings, in particular the lane markings.

[0014] The camera-based detection system can capture images of the area in front of the vehicle, including the road markings, at a specific sampling rate. It detects the lines representing these road markings.

[0015] For each recorded time sample Ts, the algorithm for extracting the road course is provided with a set of parameters as input variables, which describe the road markings as they are captured by the camera-based detection system. These can be, for example, clothoid parameters or polynomial coefficients.

[0016] The algorithm generates a movable window with a defined width depending on the length of the time samples Ts, for example, a width that corresponds to 50 times a time sample Ts. The movable window is aligned with the vehicle's coordinate system.

[0017] For each time sample Ts and for a given window, the lines recorded by the camera-based detection device, which represent the road markings and are relatively short, are converted into an image.

[0018] A histogram is calculated from the image, and the image position where most of the captured lines are located is determined. A point is generated at this image position. In addition to histogram estimation, heuristics can preferably also be used to identify merging and branching points of roads.

[0019] In a preferred embodiment, it is proposed that the trajectories calculated in steps c) and d) be optimized by an optimization algorithm before being merged. Preferably, a simulated annealing algorithm can be used as the optimization algorithm.

[0020] In one embodiment, a RANSAC filter algorithm can be used to preprocess the data obtained in step a). This allows potential outliers in the measurement data to be reliably detected and eliminated.

[0021] A common practical problem is that the camera sensors of the camera-based detection system typically cannot provide a reliable estimate of the yaw and yaw rate of the target vehicle. This is because the target positions of the vehicle, as detected by the camera-based detection system, are typically overlaid with both white, Gaussian noise and non-white, non-Gaussian noise. Therefore, in an advantageous embodiment, it is proposed that an alpha-beta filter algorithm be used to remove noise components from the data obtained in step a).

[0022] In one embodiment, a modified CYRA model is used to calculate the estimated trajectory in step c) using the physical model. In this modified model, individual trajectories are calculated in parallel by several physical models, and a weighted trajectory is calculated by combining the individual trajectories. The modified CYRA model assumes a constant yaw rate and constant acceleration (CYRA = "Constant Yaw Rate and Acceleration").

[0023] A system according to the invention for carrying out a method according to one of claims 1 to 6 comprises a camera-based detection device designed to detect states of the target vehicle, to detect states of other vehicle objects in the vicinity of the vehicle and to detect road markings according to step a), and a computing device in which at least means for carrying out steps b) to e) of the method are implemented.

[0024] In one embodiment, the computing device may include a prediction module, in which, in particular, a driver behavior classification module is implemented. This module includes a Markov state machine in addition to a driving maneuver detection system. This enables improved behavior classification. Based on a normalized lane assignment of the target vehicle, the derivation of the normalized lane assignment of the target vehicle, and the velocity vector, the target vehicle's behavior is classified into a plurality of categories, such as "lane change," "lane keeping," "accelerating," "braking," and "maintaining speed." The input variables of the driver behavior classification module preferably consist of a number N of estimated historical states of the target vehicle, a number N of estimated historical states of the other vehicle objects, and the estimated road course.

[0025] Further features and advantages of the present invention will become clear from the following description of a preferred embodiment with reference to the accompanying figures. Fig. 1 a schematic representation of a system designed to carry out a method for predicting the trajectory of a target vehicle in a vehicle's environment according to a preferred embodiment of the present invention, Fig. 2 a schematic representation illustrating details of the preprocessing of the data captured by a camera-based acquisition device of the system, Fig. 3 a schematic representation showing details of an optimization algorithm, Fig. 4 a schematic representation illustrating details of an extraction of the road course, Fig. 5 a schematic representation illustrating further details of the extraction of the road route.

[0026] With reference to Fig. 1 comprises a system 1 for carrying out a method for predicting a trajectory of a target vehicle in a vehicle's environment, a camera-based detection device 2, which is particularly designed to detect and track the target vehicle and other vehicle objects in the vehicle's environment, as well as to detect road markings.

[0027] Furthermore, system 1 comprises a computing unit 3 in which a trajectory prediction module 4 is implemented. This module receives the measurement data from the camera-based detection device 2 as input and is configured to use this measurement data to predict the trajectory of the target vehicle in its vicinity using a computer-implemented method as described below. For this purpose, several software modules 40, 41, 42, 43, and 44 are implemented within the trajectory prediction module 4, the functions of which will be explained in more detail below.

[0028] The prediction module 4 includes a preprocessing module 40, which is configured to preprocess the data acquired by the camera-based acquisition device 2 and made available to the prediction module 4 for further processing. The preprocessing module 40 is configured to remove any outliers in the measured states of the target vehicle and other vehicle objects from the measurement data using a suitably designed data preprocessing algorithm, and to calculate missing states of the target vehicle and other vehicle objects.

[0029] In many situations, the camera-based detection device 2 cannot reliably detect road markings during the vehicle's journey for subsequent processing. Therefore, the preprocessing module 40 also implements an algorithm designed to estimate the road's course and lane alignment based on measured states of the target vehicle, other vehicle objects in the vicinity, and the vehicle itself, as well as on the road markings and static objects detected by the camera-based detection device 2, such as guardrails, gradients, etc.

[0030] Furthermore, a physical trajectory calculation module 41 is implemented in the prediction module 4. This module can calculate the trajectory of the target vehicle in its environment based on a physical trajectory calculation model and the data provided by the preprocessing module 40. Preferably, a modified CYRA model is used, in which individual trajectories of the target vehicle are calculated in parallel by several physical models, and a weighted combination of these individual trajectories is calculated. The modified CYRA model assumes a constant yaw rate and constant acceleration (CYRA) of the target vehicle.

[0031] Furthermore, a driver behavior classification module 42 is implemented in prediction module 4. In addition to a driving maneuver detection system, this module includes a Markov state machine to improve behavior classification. Based on a normalized lane assignment of the target vehicle, the derivation of this normalized lane assignment, and the velocity vector, the target vehicle's behavior is classified into multiple categories, such as "lane change," "lane keeping," "accelerating," "decelerating," and "maintaining speed." Inputs to driver behavior classification module 42 include a number N of estimated historical states of the target vehicle, a number N of estimated historical states of the other vehicle objects, and the estimated road path.

[0032] Prediction module 4 also includes a path planning module 43, which receives input data from the driver behavior classification module 42 and is configured to predict and output a behavior-based trajectory of the target vehicle. Path planning module 43 is configured to calculate this trajectory based on a behavior category representing driver behavior, as well as on the target vehicle's states and history.

[0033] Furthermore, a trajectory merging module 44 is implemented in the prediction module 4. This trajectory merging module 44 is configured to merge, based on an optimization algorithm, the parameters of the trajectory calculated by the physical trajectory calculation module 41 and the parameters of the trajectory calculated by the path planning module 43. For this purpose, an optimization module 5 is implemented in the computing unit 3. This module is configured to jointly adjust all parameters of the target vehicle's trajectory, obtained by the physical trajectory calculation module 41, the driver behavior classification module 42, and the path planning module 43, using appropriate optimization techniques.

[0034] With reference to Fig. 2. Further details of the preprocessing of the data acquired by the camera-based detection device 2 using the preprocessing module 40 will be explained in more detail below. A novel method for estimating the yaw and yaw rate of the target vehicle is used, which is based on a number N (buffer 400) of previous target positions (x, y positions) of the target vehicle.

[0035] A key challenge is that the camera sensors of the camera-based detection device 2 cannot normally provide a reliable estimate of the yaw and yaw rate of the target vehicle. This is because the target positions of the target vehicle, as detected by the camera-based detection device 2, are overlaid with both white, Gaussian noise and non-white, non-Gaussian noise.

[0036] In the method used here, the target yaw and the target yaw rate are estimated based on camera measurements by a yaw calculation module 402, whereby outliers, which are preferably detected by an adaptive RANSAC algorithm 401, are disregarded and noise influences are eliminated - preferably by an alpha-beta filter algorithm 403. In addition to the number N of previous target positions (x, y positions) of the target vehicle from the buffer 400, the speed of the target vehicle in the x-direction is also incorporated into the RANSAC algorithm 401.

[0037] The camera-based acquisition device 2 outputs the vehicle states of the target vehicle as positions in the vehicle's xy-coordinate system, where the x-axis represents the vehicle's forward direction. The novel algorithm for estimating yaw and yaw rate is based on a modified RANSAC filter algorithm 401 and an alpha-beta filter 403. A number N of measurements of the target vehicle's positions are recorded. The target vehicle's positions are fitted to a line using the RANSAC algorithm 401, with the line's parameters adjusted online based on the target position and velocity. The angle between the fitted line and the target vehicle's x-coordinate can be easily calculated and provides a measure of the target vehicle's yaw.The yaw calculated in this way is provided to an Alpha-Beta algorithm 403, which is trained to output a filtered yaw and an estimated yaw rate.

[0038] With further reference to Fig. 3. Further details of the optimization procedure, which is carried out using the optimization module 5, will be explained in more detail below.

[0039] Using the camera-based acquisition device 2, object data of the target vehicle, as well as other vehicle objects in the vehicle's vicinity and detected road markings, are recorded. Each target vehicle and each other vehicle object is provided via a camera interface of the camera-based acquisition device 2 as a two-dimensional box with its corresponding covariance matrix. Each detected road marking is provided via the camera interface as a clothoid parameter.

[0040] The optimization module 5 has a reference extraction module 50, which processes the data obtained from the camera-based acquisition device 2 in the manner described above as follows: - Recorded road markings (road lines) that were recorded as clothoids are fitted with a cubic polynomial. - The road markings (road lines) are fed into a novel algorithm for extracting the road course, using which an estimate of the road course is calculated. - The recorded positions of the target vehicle and other vehicle objects are processed using a smoothing function. The result of this smoothing is an estimate of the actual trajectories of the target vehicle and the other vehicle objects. - Based on the estimation of the road course and the trajectories of the target vehicle and other vehicle objects, the driver's basic truth behavior is characterized.

[0041] This data is provided to an evaluation module 51. The prediction module 4 reads the data recorded by the camera-based acquisition device 2 and, after preprocessing this data, provides a prediction for the trajectory of a specific target vehicle. Errors in the x- and y-coordinates between the prediction and the estimated baseline behavior of the driver are calculated using the evaluation module 51. The difference between this prediction and the driver behavior classified by the driver behavior classification module 42 is compared with the driver's labeled baseline behavior.

[0042] An optimization algorithm 52, preferably based on a simulated annealing method, calculates updated parameters for the prediction module 4 to minimize errors in the x and y directions as well as errors in the driver behavior classification module 42.

[0043] With reference to Fig. 4 and Fig. 5. Further details of the algorithm for extracting the road course will be explained in more detail below.

[0044] The road alignment extraction algorithm provides a novel way to extract an estimate of the actual road alignment, in particular the number of lanes and their shape, using imprecise measurements of road markings from camera images of the camera-based acquisition device 2. Fig. Figure 4 indicates the actual course of a road marking 100. The camera-based detection device 2 can capture images of the area in front of the vehicle, including the road markings present there, at a specific sampling rate. In doing so, it detects corresponding lines 102, 102a-102e, which represent the road markings.

[0045] For each recorded time sample (Ts), the algorithm for extracting the road course is provided with a set of parameters as input variables, which describe the road markings as they are captured by the camera-based detection device 2. These can be, for example, clothoid parameters or polynomial coefficients.

[0046] The algorithm generates a movable window 101 with a defined width depending on the length of the time samples Ts, for example, a width that corresponds to 50 times a time sample Ts. The movable window 101 is aligned with the vehicle's coordinate system.

[0047] In each time sample (Ts) and for a given window 101, the relatively short lines 102a-102e captured by the camera-based detection device 2 are converted into an image 103, which is displayed in Fig. 5 is shown in a schematically simplified form.

[0048] A histogram H_i is calculated from image 103, and the image position where most lines 102a-102e are located is determined. A point P_i is generated at each of these image positions. In addition to histogram estimation, heuristics can preferably also be used to identify merging and branching points of roads.

[0049] After determining all points P_i in windows 101, an estimate of the actual road marking is created by fitting a spline to all points P_i. The output of this algorithm is an estimated road alignment.

[0050] The road alignment extraction algorithm provides a novel method for extracting an estimate of the actual road alignment using imprecise measurements of road markings from camera images of the camera-based acquisition device 2. This algorithm utilizes a window 101 containing recorded, imprecise, and relatively short road markings (lines 102, 102a-102e) and determines the actual position of the road marking using histogram estimations. In addition to histogram estimation, the algorithm preferably also uses heuristics to identify merging and branching points of the road. The output of this algorithm is an estimated road alignment.

Claims

[1] Method for predicting the trajectory of a target vehicle in a vehicle's environment, comprising the steps, a) Detection of the state of the target vehicle, detection of the state of other vehicle objects in the vicinity of the vehicle and detection of road markings using a camera-based detection device (2), b) Preprocessing the data obtained in step a), removing outliers and calculating missing states, c) Calculating an estimated trajectory using a physical model based on the data preprocessed in step b), d) Calculating a driver behavior-based trajectory based on the data preprocessed in step b), e) Combining the trajectories calculated in steps c) and d) to form a predicted trajectory of the target vehicle, wherein, in the preprocessing of the data obtained in step a), the road course is estimated based on measured states of the target vehicle as well as other vehicle objects in the environment and the vehicle itself, and based on the road markings captured by the camera-based detection device (2) at a defined sampling rate, wherein the road markings captured by the camera-based detection device (2) are provided to an algorithm for extracting the road course, which is configured to estimate the road course from the road markings captured by the camera-based detection device (2) by generating a movable window (101) with a defined width and, for each time sample and for a given window (101), converting the lines (102, 102a-102e) captured by the camera-based detection device (2), which represent the road markings, into an image (103),wherein a histogram is calculated from the image (103) so that the image position where most of the captured lines (102, 102a-102e) are located is determined, and a point is created there, and wherein, after determining all points in each of the windows (101), in a subsequent step the actual road course is estimated by fitting a spline to all points. [2] Method according to claim 1, characterized by , that the trajectories calculated in steps c) and d) are optimized before being combined by an optimization algorithm (52). [3] Method according to claim 2, characterized by , that a simulated annealing algorithm is used as the optimization algorithm (52). [4] Method according to any one of claims 1 to 3, characterized by , that a RANSAC filter algorithm (401) is used for preprocessing the data obtained in step a). [5] Method according to any one of claims 1 to 4, characterized by , that an alpha-beta filter algorithm (403) is used to remove noise components from the data obtained in step a). [6] Method according to any one of claims 1 to 5, characterized by , that for calculating the estimated trajectory using the physical model in step c), a modified CYRA model is used in which individual trajectories are calculated in parallel by several physical models and a weighted combination of the individual trajectories is calculated. [7] System (1) for carrying out a method according to one of claims 1 to 6, comprising a camera-based detection device (2) designed to detect states of the target vehicle, to detect states of other vehicle objects in the vicinity of the vehicle and to detect road markings according to step a), and a computing device (3) in which at least means for carrying out steps b) to e) of the method and for carrying out the algorithm for extracting the road course are implemented. [8] System (1) according to claim 7, characterized by , that the computing device (3) includes a prediction module (4) in which, in particular, a driver behavior classification module (42) is implemented, which, in addition to a driving maneuver detection system, has a Markov state machine.

Images