Method and computer program for characterizing future trajectories of road users

By mapping trajectory histories and environmental features into a second feature space with an adaptive loss function, the method addresses ambiguity in mode definitions and prevents averaging, stabilizing the learning process and enhancing prediction accuracy for road users.

DE102022201127B4Active Publication Date: 2024-12-05ZF FRIEDRICHSHAFEN AG
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
DE102022201127
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-02-03
Publication Date
2024-12-05
Estimated Expiration
2042-02-03

AI Technical Summary

Technical Problem

Existing trajectory prediction methods for road users in dynamic environments suffer from ambiguity in mode definitions, leading to unstable learning processes and unclear interpretability, as well as issues with inter-mode averaging and variable mode numbers, affecting the reliability and stability of predictions.

Method used

The method involves mapping trajectory histories and environmental features into a second feature space using an adaptive loss function to identify distinct modes, ensuring clear definitions and preventing inter-mode averaging by minimizing distances between latent representations of ground truth and predicted trajectories, allowing for a variable number of modes based on learned characteristics.

Benefits of technology

This approach enhances the stability and interpretability of the learning process, providing clear mode definitions and preventing averaging, thereby improving the reliability and accuracy of trajectory predictions for road users in dynamic environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Methods for characterizing future trajectories (Y) of road users comprising the steps: • Received from ◯ Trajectory histories of road users, encompassing positions of road users measured by means of sensors of the road users and / or simulated by means of vehicle dynamics and / or motion models and / or extracted from map data, depending on time and ◯ Environmental characteristics of a current traffic scene in the form of an input (X) (V1); • for each of the road users, determine an embedding (E) of trajectory and / or environment features relating to that road user in a first feature space (V2); • Mapping the image area of ​​the embedding (E) onto a second feature space (C) comprising (V3) o a first number (Nc) of characteristics (C[i]) that each characterize the future trajectories (Y) for this road user; o a list (P) whose length is equal to the first number (Nc) and whose entries each represent probabilities (P[i]) for the occurrence of one of the future trajectories (Y) i ) specify the respective characteristics (C[i]); • Predicting trajectories (Y) for this road user (V4); • Determine whether the trajectory predictions (Y) are based on different characteristics (C[i]) or are realizations of the same characteristic (C[i]) (V5).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method and a computer program for characterizing future trajectories of road users.

[0002] In the context of AD / ADAS applications, but also in the context of Industry 4.0 and collaborative human-robot interaction, mere sensory detection of the environment is not sufficient. Rather, the temporal prediction of the further development of the dynamic scene with all its independent interactors, such as people, vehicles, and cyclists, is becoming increasingly important in order to be able to make intelligent decisions for automated vehicles, for example. Not only the interaction of all interactors, such as road users, with each other is important, but also their interaction with their immediate environment, such as the traffic space and / or the infrastructure.

[0003] To ensure reliable and powerful scene prediction, all these explicit, implicit, regional, and event-based rules / information must be considered and used for temporal prediction. DE 10 2020 210 379 A1 discloses a hybrid scene representation that models interactions between static and dynamic objects and / or information.

[0004] DE 10 2021 203 440 A1 discloses a modeling of road user interactions across automated driving systems across the entire traffic space and over a specified time period in the past, merging the road user history with all static and dynamic parts of the scene. This enables a prediction of the trajectories / behavior of all road users for a specific time period in the future.

[0005] From DE 10 2012 009 297 A1 a method for predicting a collision risk is known, whereby it is intended to predict trajectories of the ego vehicle and of other road users based on maneuvering intentions and, based on this, to establish several situation hypotheses and to evaluate them with regard to the collision risk.

[0006] From DE 10 2019 216 235 A1 a method for predicting the trajectory of an ego vehicle and other vehicles is known, wherein it is provided to predict the trajectory of the other vehicles based on their driving condition and on the possible traffic development.

[0007] From DE 10 2019 218 455 A1 a method for predicting the future locations of other road users is known, whereby the prediction is provided based on the current position, on environmental data and on predetermined characterization methods.

[0008] Further state of the art regarding interaction modeling is disclosed, for example, in Gao et al.: VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation, Computer Vision and Pattern Recognition 2020, arXiv:2005.04259 and in Zhao et al.: Multi-Agent Tensor Fusion for Contextual Trajectory Prediction, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12126-12134).

[0009] By decoding individual trajectory histories on multiple possible predicted trajectories, a multitude of possible future modes are calculated, resulting in one-to-many mapping, which involves learning multiple modes. One-to-many mapping enables the history of a road user or traffic agent in a specific environment to be mapped onto multiple future predictions or multiple modes. For example, the environment is a T-intersection. A car approaches this T-intersection. Possible modes include driving straight ahead, turning left, and turning right.

[0010] Without a mode definition, when predicting the future movement of a traffic agent, it is unclear a priori whether two obtained modes are realizations of two different modes or two different realizations of one mode. Thus, it is also unclear whether averaging is allowed or not during training.

[0011] The unclear dividing lines between modes not only impair the stability of the learning process but also the assessment of the quality of the results. Averaging the output for identical input within a correctly defined mode is even useful in this case.

[0012] Furthermore, a mode itself lacks a clear definition. This can impact the user experience, the training and debugging process, and the interpretation of results.

[0013] Another problem is that the number of modes is usually fixed, but is not known a priori.

[0014] Furthermore, the multi-mode learning process can become unstable. During training, for example, the output mode closest to a reference trajectory, i.e., the ground truth, is considered for backpropagation. This poses a problem if the closest mode is another, already well-learned mode and was simply chosen because it is closer to the ground truth than another, not yet learned, random output. In this way, a well-learned mode would be degraded again to approximate a second mode yet to be found.

[0015] The invention was based on the task of preventing inter-mode averaging in one-to-many mapping and improving the interpretability of learned modes and the stability of the learning process of this mapping.

[0016] The subject matter of claims 1 and 13 each solves this problem by identifying a mode in a second feature space, which corresponds to a latent representation. The second feature space represents a characteristic that is interpretable in an output space, i.e., the space of future trajectories or the trajectory predictions of the traffic agents. This makes the modes interpretable. Furthermore, users, developers, or expert knowledge can be involved in defining what distinguishes two modes. In another configuration, the characteristic is learned from the data using an adaptive loss function. This loss causes the learned model, for example, an artificial neural network, to map a ground-truth trajectory to a specific characteristic or to deviate from it.

[0017] During training, the distance between the ground truth and the output closest to the ground truth is not minimized as usual. Instead, the distance between the ground truth and the output whose latent representation is closest to the latent representation of the ground truth is minimized. This prevents inter-mode averaging of the outputs.

[0018] The same function can be used to generate all modes. This refers to a function whose parameters, but not its input, are independent of the mode, for example, an artificial neural network whose parameters are independent of the mode. This function accepts two inputs: In addition to the usual input, i.e., the history of all traffic agents and the representation of the scene, as disclosed, for example, in the prior art cited above, a latent representation, i.e., the characteristic. This makes it possible to generate a variable number of modes for each input. The network is trained to produce an output that satisfies a characteristic. The second feature space is heavily regularized.

[0019] One aspect of the invention relates to a method for characterizing future trajectories of road users. The method comprises the steps: • Receive ◯ Trajectory histories of road users comprising positions of road users measured by sensors of the road users and / or simulated by vehicle dynamics and / or movement models and / or extracted from map data as a function of time and ◯ Environmental features of a current traffic scene in the form of an input; • for each of the road users, determining an embedding of trajectory and / or environmental features related to this road user in a first feature space; • Mapping the image area of ​​the embedding to a second feature space comprising ◯ a first number of characteristics each characterising the future trajectories for this road user; ◯ a list whose length is equal to the first number and whose entries each specify probabilities for the occurrence of one of the future trajectories with the respective characteristics; • Predicting trajectories for this road user; • Determine whether the trajectory predictions are based on different characteristics or are realizations of the same characteristics.

[0020] Another aspect of the invention relates to a computer program for characterizing future trajectories of road users. The computer program comprises instructions that cause a hardware platform of a remote system, a driver assistance system, or an autonomous driving system to execute the steps of a method according to one of the preceding claims when the computer program is executed by the hardware platform.

[0021] The instructions of the computer program according to the invention comprise machine instructions, source code or object code written in assembly language, an object-oriented programming language, for example C++, or in a procedural programming language, for example C. According to one aspect of the invention, the computer program is a hardware-independent application program which is provided, for example, via a data carrier or a data carrier signal using software over the air technology.

[0022] Advantageous embodiments of the invention emerge from the definitions, the subclaims, the drawings and the description of preferred embodiments.

[0023] Road users include interactors in scenes of an environment, for example, scenes of a traffic space. Road users include, for example, people, such as pedestrians, vehicles, driving systems, and cyclists. Driving systems include automated driving systems, from automated to autonomous vehicles, road vehicles, people movers, shuttles, robots, and drones. Road users also include intelligent agents, such as self-driving vehicles, robots, or intelligent infrastructure elements, such as intelligent lighting systems such as traffic lights, which communicate with other road users via wireless technology, e.g., Car-to-X communication.

[0024] Road user sensors include cameras, radar sensors, lidar sensors, ultrasonic sensors, acoustic sensors, Car2X units.

[0025] Vehicle dynamics and / or motion models, for example, utilize the coordinates of road users, their positions, and / or orientations over a specific period of time to generate trajectory histories. The vehicle dynamics or motion model is based, for example, on a Kalman filter. Simulation is performed using software, hardware, model, and / or vehicle-in-the-loop methods.

[0026] Map data includes data from real-time and / or offline maps.

[0027] Environmental features include houses, streets, especially road geometry and / or condition, signs, lane markings, vegetation, moving road users, vehicles, pedestrians, and cyclists. Static environmental features are further divided into two categories. Elements that change virtually nothing or only change after extended periods of time do not change their state in the short term and are referred to as rigid. In contrast, elements that can change state frequently are considered state-changing. Traffic lights and variable message signs, for example, fall into the latter category. Dynamic environmental features relate to the moving road users in a traffic scene and include their positions and / or orientations.

[0028] In the first feature space, each road user is identifiable by coordinates. The inputs, i.e., the trajectory histories and the environmental features, are obtained, for example, in the form of a tensor.

[0029] The embedding represents information related to the road user, also called agent-centric, which is fused from the history of the road user in question, the traffic scene, and the history of other road users. A respective embedding can be obtained for each road user. According to one aspect, the embedding is obtained using a learnable function, for example, an artificial neural network. The embedding can be a tensorial quantity.

[0030] The second feature space is the characteristic feature space, whose elements identify a trajectory prediction and / or a reference trajectory through a characteristic. The characteristic characterizes the trajectory prediction and / or the reference trajectory. This allows the second feature space to be interpreted in the space of the road users' trajectory predictions. The features that each characterize the future trajectories for this road user are, according to one aspect, elements of a tensor and form the first number of characteristics. The list, whose length is equal to the first number and whose entries each specify probabilities for the occurrence of one of the future trajectories with the respective characteristics, i.e., the probability of the respective characteristic, is, according to one aspect, a probability vector whose length is equal to the first number.

[0031] According to a further aspect, the method comprises the following steps: • Determining a second number of residual characteristics from the first number of characteristics whose probability is greater than a first threshold; • Inputting the second number of data pairs, each comprising the embedding and one of the residual characteristics, into a first machine learning model that is or has been trained on training data comprising the trajectory predictions and reference trajectories to infer trajectory predictions with the given residual characteristics for this road user; • Output the trajectory predictions inferred with the first machine learning model.

[0032] The first number is greater than the maximum number of modes. The second number of residual characteristics, a function of the embedding, is the number of modes that exceed a minimum residual probability, which is the first threshold. For example, for a residual probability of 0.001, modes with probabilities below 0.1% are ignored. The residual probability is a configuration parameter and can be adjusted according to various criteria, including the ASIL level of the respective application. For each characteristic whose corresponding probability is greater than the residual probability, the embedding and this characteristic are mapped to a trajectory prediction using the first machine learning model, for example, an artificial neural network. According to one aspect, the first machine learning model is used with the same set of parameters for all modes.

[0033] A machine learning model corresponds to a learnable function.

[0034] According to a further aspect, the second feature space comprises • predefined characteristics with predefined characteristic ranges; • predefined features with learnable feature ranges; • learnable features or a combination of these features. The predefined features include, for example, • Length of a trajectory; • spatial distribution of waypoints on a trajectory • Orientation of a trajectory; • Fourier descriptor of a trajectory; • Environmental features of the respective traffic scene, or a combination of these. In the case of a combination, a characteristic is a feature list, for example, a vector, comprising the individual features.

[0035] The specified characteristics can be based on expert knowledge.

[0036] According to one aspect, the length of the trajectory is expressed by an average velocity. The average velocity corresponds to the mean of the first derivatives of the trajectory.

[0037] The spatial distribution of the points along the trajectory is expressed in one aspect by a mean acceleration. The mean acceleration corresponds to the second derivative of the trajectory.

[0038] The orientation of the trajectory is expressed, according to one aspect, by an orientation bin up to a specific point in the trajectory: this is the angle of the vector formed by the first point of the trajectory and the specific point of the same trajectory. The reference for measuring this angle can be either the orientation of the road user or the road user's current direction of movement. An example configuration is the angle of the vector defined by the first and last points of the trajectory.

[0039] The Fourier descriptor is a scale- and orientation-invariant representation of closed contours. However, there are variants that account for open contours. The low-frequency features provide a rough characterization of the trajectory shape.

[0040] In another configuration, features from the traffic scene, i.e. not only features of the trajectory, can be included, for example, whether the movement of the road user represented by the trajectory takes place on a road or a sidewalk. Here, for example, stereotypical behavior on typical traffic infrastructure can be modeled. One example is the approach behavior to a traffic light, also taking into account the currently active traffic light phase and even the local customs at the time of the traffic light. For example, in one part of a country, it may be common practice to react to the traffic light transition from green to red by briefly accelerating in front of the traffic light. In one configuration of the proposed solution, the characteristics of the modes can be learned for each region or country.

[0041] The second feature space can be formed from these predefined features or from a subset of these features. An example of a simple second feature space is the space spanned by the features orientation bin, average speed, and average acceleration. For an embedding, for example, a fused representation of the history of the road user in question, the history of other traffic agents, and a representation of the traffic scene and a latent vector (45, 2, 4) in this second feature space, the first machine learning model will generate a trajectory that has an angle of approximately 45 degrees with the orientation of the road user in question or their direction of movement. The average difference between two consecutive points of this trajectory is approximately 2 meters, assuming one frame per second for simplicity.The average difference of the differences is approximately 4 meters / s. In this example, the features orientation bin, average velocity, and average acceleration are specified. The feature ranges of these fixed features can either be specified or learned. Otherwise, the features are learned automatically from the data.

[0042] In another configuration, any combination of the three aforementioned cases could be considered. For example, some features could be learned, some could be fixed with learnable feature ranges, and some could be fixed with fixed feature ranges.

[0043] According to a further aspect, the image area of ​​the embedding is mapped to the second feature space by means of a second machine learning model that has been or is trained on training data comprising the trajectory predictions and reference trajectories to determine the second feature space, wherein the second machine learning model • in the case of given features with given feature ranges, has been or is being trained to determine the probabilities for the occurrence of one of the future trajectories with the respective features; • in the case of given features with learnable feature ranges, training is carried out on the training data to additionally determine cluster centers in the second feature space; • in the case of learnable features, the training data is trained using regularizations to additionally learn the features.

[0044] The second machine learning model, for example, is an artificial neural network with a bottleneck, such as a bottleneck structure or a pooling layer.

[0045] Here, “additionally” means in the case of given features with learnable feature ranges that probabilities are also learned and that in this case the case of given features with given feature ranges is included.

[0046] In the case of learnable features, “additionally” means that the case of given features with given feature ranges and the case of given features with learnable feature ranges are included.

[0047] In the case of predefined features with predefined feature ranges, the feature tensor is a fixed lookup table; it can also be defined implicitly using bins in each feature. The output of the second machine learning model is the probability vector. An example is the aforementioned feature space of orientation bin, average velocity, and mean acceleration, where fixed bins are predefined in each of the three features.

[0048] In the case of predefined features with learnable feature ranges, the characteristic and probability are tensor outputs of the second machine learning model. An example is the aforementioned feature space of orientation bin, average speed, and average acceleration. However, the features do not have predefined bins. Instead, the cluster centers of the characteristics in this feature space are learned from the training data.

[0049] In the case of learnable features, the characteristic and probability are tensor outputs of the second machine learning model. The features are not predefined; they are learned.

[0050] According to a further aspect, the trajectory predictions and the reference trajectories are mapped to the second feature space by means of a function, wherein the function • in the case of given features with given feature ranges, a given calculation graph that calculates the features for a trajectory; • in the case of given features with learnable feature ranges, a given computational graph and cluster centers of the characteristics in the second feature space are learned from training data; • in the case of learnable features, a third machine learning model that is or has been trained on the training data to learn the features, wherein a trajectory prediction and a reference trajectory are assigned to different modes if a distance of the trajectory prediction to the reference trajectory exceeds a second threshold.

[0051] The definitions of the characteristic features are encapsulated in this function.

[0052] For a given computational graph, the function calculates, for example, the orientation bin, average velocity, and mean acceleration for a trajectory. This corresponds to a classification problem with the first number of classes. In this approach, the order information in the bins is not used. In a further aspect, the order relationship in each feature is taken into account using ordinal regression. The problem is thus considered as NF Ordinal regression problem with N each B Ranks, with N F the number of features of the second feature space and N B is the number of bins per feature.

[0053] The third machine learning model learns the learnable feature regions and learnable features.

[0054] According to a further aspect, a reference trajectory is mapped to the second feature space using the function described above. In a training forward path, • from the first number of characteristics, the reference characteristic is determined whose distance to the image of the reference trajectory is minimal; • the trajectory prediction is output, which results from the input of the embedding and the pre-determined residual feature into the first machine learning model.

[0055] The reference trajectory is thus described using the characteristics of the second feature space. The second machine learning model maps the first feature space, comprising the history of all road users and the information of the traffic scene, to the second feature space. The function, for example, the third machine learning model, maps the trajectory predictions and the reference trajectories to the second feature space. According to the invention, during training, only the reference characteristic that is closest to the image of the reference trajectory is considered to generate a trajectory prediction. In other words, a single output is generated per iteration on a training sample.

[0056] Distance is a measure of distance, for example, L1-norm, L2-norm, p-norm. In this disclosure, the L1-norm is applied to unnormalized probabilities. This applies to all places where the term ||P||1 appears in the text and figures.

[0057] According to a further aspect, a trajectory prediction and a reference trajectory are each mapped to the second feature space using the function described above. In the case of predefined features with predefined feature ranges, in a training backward path • a first distance of the trajectory prediction to the reference trajectory; • a second distance of a given reference characteristic, whose distance to the image of the reference trajectory is minimal, to the image of the trajectory prediction is determined and a loss function comprising the first and the second distance is minimized.

[0058] This creates a trajectory prediction that is as close as possible to the reference trajectory and whose characteristics are as close as possible to the specified residual characteristics.

[0059] According to a further aspect, different terms of the loss function are weighted differently.

[0060] According to a further aspect, a trajectory prediction and a reference trajectory are each mapped to the second feature space using the function described above. In the case of predefined features with learnable feature ranges, in a training backward path • a first distance of the trajectory prediction to the reference trajectory; • a second distance of a reference characteristic, whose distance to the image of the reference trajectory is minimal, to the image of the reference trajectory • a third distance of the image of the trajectory prediction to the image of the reference trajectory; • a first regularization term that regularizes that the reference characteristic is learned from the image of the reference trajectory and respective distances of the remaining characteristics to the image of the reference trajectory are relatively large, and a loss function comprising the first, second and third distances and the first regularization term is minimized.

[0061] In this case, the entries of the characteristic tensor are also learned and thus the deviation from the reference trajectory characteristic is also backpropagated, but with derivatives with respect to the characteristic and the output trajectory.

[0062] In another configuration, expert knowledge about possible feature ranges can be incorporated into the first regularization term. For example, the constraint could be imposed that the pairwise distance in each dimension should exceed a given minimum threshold.

[0063] According to a further aspect, in the case of learnable features, the function that maps the trajectory predictions and the reference trajectories to the second feature space is learned in a training backward path of the third machine learning model by minimizing an adaptive loss function such that, in the event that the first distance of the trajectory prediction to the reference trajectory falls below the second threshold, the function maps the reference trajectory to the reference characteristic.

[0064] The adaptive loss function causes the function to map the reference trajectory image to the same reference characteristic if the reference characteristic performs well, or to a different characteristic if it does not. In the latter case, a different characteristic can be selected the next time the training procedure uses the same training sample. This prevents mode averaging.

[0065] The other characteristic is randomly selected according to one aspect. The distribution from which this other characteristic is taken can be the uniform distribution. However, this can also be the distribution of probabilities, especially if backpropagation through the function only begins after several epochs.

[0066] According to another aspect, the adaptive loss function includes a second regularization term that regularizes the function's parameters. At the beginning of training, the function's weights are random. To ensure that the function maps similar output trajectories to similar characteristics, the function's parameters should be regularized.

[0067] The backpropagation of the adaptive loss function by the third machine learning model can also begin after several training epochs to stabilize the training behavior. On the other hand, the backpropagation of the second regularization term should begin from the first training epoch, as the smoothness of the third machine learning model is a desired property that contributes to stabilizing the training from the beginning.

[0068] In this case, the first regularization term can be similar to the first regularization term for given features with learnable feature ranges. In another configuration, the rows of the characteristic tensor can be assumed to be singular vectors of a matrix resulting from the second machine learning model. This is a strong regularization term and has the advantage that any change in one characteristic automatically changes the others, so they are always orthogonal.

[0069] According to a further aspect, the respective loss function includes at least one parsimony regularization term, also called a sparsity regularizer. For example, the parsimony regularization term is the L1 norm of the probability vector. The loss function can additionally include a so-called negative log likelihood term. The parsimony regularization term enforces parsimony with respect to the number of modes.

[0070] According to another aspect, the embedding is defined as a multi-agent scene embedding as follows: • Coding of environmental features including trajectory histories; • Encoding traffic scene information comprising fixed static environment features and state-changing static environment features; • Combining the preceding encodings into a hybrid scene representation comprising at least a first layer comprising the rigid static environmental features, a second layer comprising the state-changing static environmental features and a third layer comprising dynamic environmental features comprising the trajectory histories; • Determining interactions between the static and dynamic environmental features based on the hybrid scene representation, generating a first tensor embedding of the rigid static environmental features, a second tensor embedding of the state-changing static environmental features, and a third tensor embedding of the dynamic environmental features, and merging the first, second, and third tensor embeddings into a multi-agent scene tensor; • Extract the features of the multi-agent scene tensor per road user at the location corresponding to the road user's coordinates, fuse these features with the third tensor embedding of the road user and generate the multi-agent scene embedding per road user and per traffic scene.

[0071] The environmental features are encoded according to one aspect using a recurrent network that detects temporally encoded data in the trajectory histories. The traffic scene information is encoded according to one aspect using a convolutional network.

[0072] Merging means spatial merging. For example, spatial coordinates of road users and / or environmental features are represented in pixels. Position data of the road user and / or environmental features are captured according to one aspect using map information. A map section is created by assigning a value to each pixel of the map information corresponding to the layer of the environmental scene representation. The values ​​are based on discrete labels of the map, for example, numerical codes for street, pedestrian path, broken line, double line, etc. In addition to the map, the right-of-way rules are represented using the traffic rule information. For this purpose, a line is drawn in the middle of each lane. At intersections, additional lines are drawn that represent all permissible maneuvers. Merging is performed according to one aspect using an interaction tensor pooling module, which can include software and / or hardware components.

[0073] The hybrid scene representation layers a scenario into multiple layers. A real-world scenario is represented as a hybrid of static and dynamic information. The scenario is, for example, an image with i pixels in the x-direction and j pixels in the y-direction, in which the spatial coordinates of the road users are represented in pixels. The individual layers can also be represented as images and are arranged congruently with one another; for example, the layers lie congruently on top of one another spatially. The hybrid scene representation according to the invention can be imagined as a stack of superimposed digital photos, for example, taken from a bird's eye view of an intersection. On the other hand, this stack of images is combined with further layers of partly purely semantic information, which are represented, for example, as pure feature vectors.

[0074] Determining interactions involves predicting possible future interactions based on the eight layers of the hybrid scene representation. Interactions refer to any interactions between static and static, static and dynamic, and dynamic and dynamic environmental features. In an example interaction scenario, a car is at an intersection. A pedestrian is at the intersection. Right of way is controlled by a traffic light. One interaction is the traffic light switching. For example, if the traffic light switching shows the green light phase to the car and the red light phase to the pedestrian, then the further interactions, learned or present in the trajectory histories, are that the pedestrian stops and the car enters the intersection. The determination of interactions is carried out according to one aspect using a convolutional interaction network.

[0075] According to one aspect, the features of the multi-agent scene tensor are extracted by means of a per-road user interaction vector extraction module, which may comprise software and / or hardware components.

[0076] According to another aspect, the multi-agent scene embedding is decoded, and a trajectory prediction is generated for each road user. According to one aspect, the decoding is performed using a recurrent network.

[0077] The invention is illustrated in the following exemplary embodiments. They show: Fig. 1 an example of modes, Fig. 2 an example of an unstable learning process, Fig. 3 an embodiment of a second feature space according to the invention, Fig. 4 an embodiment of an interference according to the invention, Fig. 5 an embodiment of a training forward path according to the invention, Fig. 6 an embodiment of a training backward path according to the invention in the case of predetermined features with predetermined feature ranges, Fig. 7 an embodiment of a training backward path according to the invention in the case of predetermined features with learnable feature ranges and Fig. 8 an embodiment of a training backward path according to the invention in the case of learnable features.

[0078] In the figures, identical reference symbols designate identical or functionally similar reference parts. For clarity, only the relevant reference parts are highlighted in the individual figures.

[0079] Fig. Figure 1 shows the predictions of four future modes 1-4 for the movement of a road user. A mode itself initially has no definition. This can impair the user experience, the training and debugging process, and the interpretation of results. Without a definition of the mode, it is not clear whether mode 3 and mode 3 are two different modes or two realizations of one mode. Thus, it is also unclear whether averaging is permitted during training or not. The method according to the invention provides a solution to this problem by determining, through its execution, whether trajectory predictions Y are based on different characteristics or are realizations of the same characteristic. These characteristics characterize the modes. According to the invention, modes 1-4 are identified by characteristics C[i] of a second feature space C; see, for example, Fig. 3. This makes modes 1-4 interpretable according to the invention. Furthermore, users and / or developers and / or expert knowledge can be involved in defining what distinguishes the two modes.

[0080] Fig. Figure 2 shows how the multi-mode learning process becomes unstable in the known prior art. During training, a mode corresponding to a reference trajectory Y GT is taken into account for the backpropagation. The problem arises in the case where the nearest mode was already a well-learned mode 1 and was simply selected because it was closer to the reference trajectory Y GT than another, not yet learned mode 2. The method according to the invention provides a solution to this, namely in that by its execution a distance between the reference trajectory Y GTand that trajectory prediction Y is minimized whose latent representation, i.e. its characteristic, of the latent representation of the reference trajectory Y GT closest, see Fig. 5. This prevents averaging between the modes.

[0081] In Fig. 3, the second feature space C is defined by the three given features (θ ∞ ,s,α). θ ∞ characterizes the orientation of a trajectory prediction Y, namely, for example, the angle of the vector defined by the first and last point of the trajectory. s characterizes the length of the trajectory prediction Y or the average velocity. α characterizes the spatial distribution of the points on the trajectory or the average acceleration. The characteristic C[i 1]= (45, 2, 4) corresponds to a latent vector in the second feature space C. For an embedding E and the latent vector (45, 2, 4), the first machine learning model h will then, according to the invention, generate a trajectory prediction Y1 that has an angle of approximately 45 degrees with the orientation or direction of movement of the road user under consideration, whereby the average difference between two consecutive points of this trajectory is approximately 2 meters and the average difference of the differences is approximately 4 meters per second. For the sake of simplicity, a frame rate of 1.0 frame per second is assumed. For the embedding E and the latent vector C[i2]=(-30, 1, 0), the first machine learning model h will then, according to the invention, generate a trajectory prediction Y2 accordingly.

[0082] In Fig. 3, as in the other figures, the circle with a solid line indicates that the circled function is a learnable function. The first machine learning model h is trained and is therefore a learnable function.

[0083] Fig. 4 shows the inference setup.

[0084] In a process step V1, trajectory histories of all road users and environmental features of a current traffic scene are received as inputs.

[0085] In a method step V2, for each of the road users, an embedding E of trajectory and / or environmental features related to this road user, comprising the trajectory histories of this road user, is determined in the first feature space by means of a learnable function f, for example a fourth artificial neural network.

[0086] For example, embedding E is a multi-agent scene embedding as disclosed in the German patent application with the file number DE 10 2021 203 440.3. The multi-agent scene embedding is obtained by the following steps: • Coding of environmental features including trajectory histories; • Encoding traffic scene information comprising fixed static environment features and state-changing static environment features; • Combining the preceding encodings into a hybrid scene representation comprising at least a first layer comprising the rigid static environmental features, a second layer comprising the state-changing static environmental features and a third layer comprising dynamic environmental features comprising the trajectory histories; • Determining interactions between the static, dynamic and between the static and dynamic environment features based on the hybrid scene representation, wherein a first tensor embedding of the rigid static environment features, a second tensor embedding of the state-changing static environment features and a third tensor embedding of the dynamic environment features are generated and the first, second and third tensor embeddings are merged into a multi-agent scene tensor; • Extract the features of the multi-agent scene tensor per road user at the location corresponding to the road user's coordinates, fuse these features with the third tensor embedding of the road user and generate the multi-agent scene embedding per road user and per traffic scene.

[0087] Alternatively, the embedding E can be generated using the VectorNet disclosed in Gao et al.: VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation, Computer Vision and Pattern Recognition 2020, arXiv:2005.04259, or similar approaches or a pure convolutional network.

[0088] The method according to the invention solves the stability of the learning process, including the mode averaging problem and the problem of interpretability of learned modes.

[0089] In a method step V3, the image area of ​​the embedding E is mapped to the second feature space C using a second machine learning model g. The second feature space C comprises a first number Nc of characteristics C[i], each of which characterizes the future trajectories Y for this road user, and a list, for example a vector, P, with a length equal to the first number Nc and entries each equal to the probabilities P[i] for the occurrence of one of the future trajectories Y i with the respective characteristics C[i].

[0090] Thus, the method according to the invention makes it possible to generate a variable number of modes for each embedding E.

[0091] In a method step V4, the trajectories Y are predicted for this road user and in a method step V5, a training process determines whether the trajectory predictions Y are based on different characteristics or are realizations of the same characteristics.

[0092] For this purpose, a second number N M of residual characteristics C[i m ] from the first number Nc of characteristics C[i] whose probability P[i] is greater than a first threshold p ε In a method step V7, the second number N M of data pairs comprising the embedding E and one of the residual characteristics C[i m ] are input into the first machine learning model h. In a process step V8, the trajectory predictions Y inferred with the first machine learning model h are calculated for this embedding E 1,...,mFor each embedding E, a variable number 1,..., N M of modes. The first machine learning model h is trained to produce trajectory predictions Y 1,..,m which each have a residual characteristic C[i 1,...,m] fulfilled.

[0093] In Fig. 4, as in the other figures, the circle with a dashed line indicates that the circled function is a function with limited learning capability. The second machine learning model g, for example, is an artificial neural network with a bottleneck structure.

[0094] Fig. 5 shows a training forward path. A reference trajectory Y GT is mapped to the second feature space C using a function q. From the first number Nc of characteristics C[i], the reference characteristic C[k̂] is determined, whose distance ||q(Y GT ) - C[k]|| to the image q(Y GT) of the reference trajectory Y GT is minimal: k^=argmink‖q(YGT)−C[k]‖.

[0095] The trajectory prediction Y is output, which results from the input of the embedding E and the previously determined reference characteristic C[k̂] into the first machine learning model h. This generates a single output per iteration on a training sample. The latent representation of the thus determined trajectory prediction Y then has a minimal distance from the latent representation of the reference trajectory Y GT .

[0096] In Fig. 5, as in the other figures, the circle with a dotted line indicates that the circled function is a restricted predefined function. In the case of predefined features with predefined or learnable feature ranges, the function q is a fixed computational graph; in the case of learnable features, the function q is a restricted machine learning model, for example, an artificial neural network with a bottleneck.

[0097] Fig. Figure 6 shows a training backward path in the case of given features with given feature ranges. A first distance ||Y - Y GT || the trajectory prediction Y to the reference trajectory Y GT certainly.

[0098] Furthermore, a second distance ||C[k̂] - q(Y)|| of a given reference characteristic C[k̂], whose distance k̂ = argmin k ||q(Y GT ) - C[k]|| to the image q(Y GT ) of the reference trajectory Y GTis minimal, to the image q(Y) of the trajectory prediction Y. The characteristic q(Y) of the desired trajectory Y will thus be as close as possible to the given reference characteristic C[k̂].

[0099] Furthermore, the following term is minimized: ||P||1 - logP k̂ .

[0100] P is the probability vector and can be obtained as the output of the second machine learning model g. The term ||P||1 minimizes the L1 norm of the probability vector. This means that the second machine learning model g is learned such that the majority of the components of P are smaller than the first threshold p. ε or zero. This creates few modes -logP k̂ is the negative log-likelihood loss function that maximizes the probability P[k̂].

[0101] In this training backward path, the loss function L=‖Y−YGT‖+‖C[k^]−q(Y)‖+‖P‖1−logPk^ minimized.

[0102] Fig. Figure 7 shows a training backward path in the case of given features with learnable feature ranges. The value of the reference characteristic C[k̂] is here, in contrast to the Fig. 6, but must be learned. Therefore, a second distance ||C[k̂] - q(Y GT )|| and a third distance ||q(Y) - q(Y GT )|| determined.

[0103] The second distance ||C[k̂] - q(Y GT )|| is a distance of a reference characteristic C[k̂] to the image q(Y GT ) of the reference trajectory Y GT The second distance describes a deviation from the ground truth characteristic with respect to the reference characteristic C[k̂].

[0104] The third distance ||q(Y) - q(Y GT )|| is the distance of the image q(Y) of the trajectory prediction Y to the image q(Y GT) of the reference trajectory Y GT The third distance describes a deviation from the ground truth characteristic with respect to the trajectory prediction Y.

[0105] Furthermore, a first regularization term RC=−∑j−1, j≠k^NC‖C[j]−q(YGT)‖ which regularizes that the reference characteristic C[k̂] of the image q(Y GT ) of the reference trajectory Y GT is learned and respective distances (||C[j] - q(Y GT )||) of the remaining characteristics C[j] to the image q(Y GT ) the R e reference trajectory Y GT are relatively large. In another configuration, the expert knowledge about possible ranges of characteristics in R c For example, the operator can enforce that the pairwise distance in each dimension should exceed a specified minimum threshold.

[0106] In this training backward path, the loss function L=‖Y−YGT‖+‖C[k^]−q(YGT)‖+‖q(Y)−q(YGT)‖+‖P‖1+RC−logPk^ minimized.

[0107] Fig. Figure 8 shows a training backward path in the case of learnable features. In this case, the function q is obtained in a training backward path of a third machine learning model by minimizing an adaptive loss function La=α‖q(YGT)−C[l]‖+β‖q(YGT)−C[k^]‖+Rq is learned in such a way that in case the first distance (||Y - Y GT ||) of the trajectory prediction Y to the reference trajectory Y GT falls below a second threshold t, the function q the reference trajectory Y GT maps to the reference characteristic C[k̂].

[0108] Where R q a second regularization term R q = ||W q ||2, the parameter W = (w1, ..., w q) of the function q. l ≠ k̂ is a number from the first set 0, 1, ..., N c - 1.

[0109] The second threshold can correspond to a certain deviation of a trajectory prediction Y from the reference trajectory Y GT where Y and Y GT be assigned to two different modes.

[0110] The factor α can be defined, for example, as follows: α ≡ relu(||Y - Y GT || - t), where relu means rectifier linear unit. That is: α=0⇔‖Y−YGT‖ <t.

[0111] Accordingly, the factor β can be written as β ≡ relu(t - ||Y - Y GT ||), that is: β=0⇔‖Y−YGT‖>t.

[0112] So, if α = 0, then β is relatively large; if β = 0, then α is relatively large. Thus, for a certain trajectory prediction Y and its reference trajectory Y GTone of the two factors α or β is zero, while the other takes a positive value.

[0113] The adaptive loss function L a causes q to q(Y GT ) to the same latent characteristic C[k̂] if C[k̂] does a good job (α is large and β = 0, that is ||Y - Y GT|| is small), or to another C[l] if this is not the case (β is large and α = 0). In the latter case, a different characteristic may be selected the next time the training procedure visits the same training sample. In other words, a different k̂ might be selected in the next forward path, which could prevent mode averaging. C[l] may be chosen randomly. The distribution from which l is taken can be the uniform distribution. However, this can also be the distribution P (output of g), especially if backpropagation by q only begins after some epochs (and thus P-values ​​are partially meaningful). Reference symbol 1-4 modes V1-V8 process steps Y trajectory prediction Y GT Reference trajectory X Input E Embedding C second feature space s length of a trajectory a spatial distribution of waypoints θ orientation C[i] Characteristic C[i m ] Residual characteristic (inference) C[k̂] Reference characteristic (training) Nc first number of characteristics N M second number of residual characteristics P list, vector of probabilities P[i] Probabilities for the occurrence of one of the future trajectories Y i with the respective characteristics C[i] p ε first threshold t second threshold h first machine learning model g second machine learning model q Function implemented either as a given computational graph or a third machine learning model f fourth machine learning model W vector of parameters w1,...,w q from q L loss function L a adaptive loss function α, β factor Rc first regularization term R q second regularization term || || Distance, distance measure || ||1 L1 norm || ||2 L1 norm relu rectifier linear unit

Claims

[1] Method for characterizing future trajectories (Y) of road users comprising the steps: • Receive ◯ Trajectory histories of road users comprising positions of road users measured by sensors of the road users and / or simulated by vehicle dynamics and / or movement models and / or extracted from map data as a function of time and ◯ environmental features of a current traffic scene in the form of an input (X) (V1); • for each of the road users, determining an embedding (E) of trajectory and / or environmental features related to this road user in a first feature space (V2); • Mapping the image area of the embedding (E) to a second feature space (C) (V3) comprising o a first number (Nc) of characteristics (C[i]) each characterising the future trajectories (Y) for that road user; o a list (P) whose length is equal to the first number (Nc) and whose entries each have probabilities (P[i]) for the occurrence of one of the future trajectories (Y i ) with the respective characteristics (C[i]); • Predicting trajectories (Y) for this road user (V4); • Determine whether the trajectory predictions (Y) are based on different characteristics (C[i]) or are realizations of the same characteristic (C[i]) (V5). [2] A method according to claim 1, comprising the steps of: • Determine a second number (N M ) of residual characteristics (C[i m ]) from the first number (Nc) of characteristics (C[i]) whose probability (P[i]) is greater than a first threshold (p ε ) is (V6); • Enter the second number (N M ) of data pairs comprising the embedding (E) and one of the residual characteristics (C[im ]) into a first machine learning model (h) which is based on training data comprising the trajectory predictions (Y) and reference trajectories (Y GT ) is or has been trained, trajectory predictions (Y 1,...,m ) with the given residual characteristics (C[i m ]) for this road user (V7); • Outputting the trajectory predictions (Y 1,...,m ) (V8). [3] Method according to one of the preceding claims, wherein the second feature space (C) • predefined characteristics with predefined characteristic ranges; • predefined features with learnable feature ranges; • includes learnable features or a combination of these features, and the given features • Length (s) of a trajectory; • spatial distribution (a) of waypoints on a trajectory • Orientation (θ) of a trajectory; • Fourier descriptor of a trajectory; • include environmental features of the respective traffic scene or a combination of these, whereby in the case of the combination a characteristic (C[i]) is a feature list comprising the respective individual features. [4] Method according to one of the preceding claims, wherein the image area of the embedding (E) is determined by means of a second machine learning model (g) based on training data comprising the trajectory predictions (Y) and reference trajectories (Y GT ) has been or is trained to determine the second feature space (C), is mapped to the second feature space (C), wherein the second machine learning model (g) • in the case of given features with given feature ranges, the probabilities (P[i]) for the occurrence of one of the future trajectories (Y i ) with the respective characteristics; • in the case of given features with learnable feature ranges, training is carried out on the training data to additionally determine cluster centers in the second feature space (C); • in the case of learnable features on the training data using regularizations (R c , R q ) is trained to additionally learn the features. [5] Method according to one of the preceding claims, wherein the trajectory predictions (Y) and the reference trajectories (Y GT ) are mapped to the second feature space (C) by means of a function (q), where the function (q) • in the case of given features with given feature ranges, a given calculation graph that calculates the features for a trajectory; • in the case of given features with learnable feature ranges, a given computation graph is used and cluster centers of the characteristics (C[i]) are learned in the second feature space from training data; • in the case of learnable features, a third machine learning model is trained or has been trained on the training data to learn the features, with a trajectory prediction (Y) and a reference trajectory (Y GT ) can be assigned to different characterizing features if a distance (||Y - Y GT ||) of the trajectory prediction (Y) to the reference trajectory (Y GT ) exceeds a second threshold (t). [6] Method according to one of claims 2 to 5, wherein a reference trajectory (Y GT ) is mapped onto the second feature space (C) by means of a function (q) according to claim 5 and in a training forward path • from the first number (Nc) of characteristics (C[i]) the reference characteristic (C[k̂]) is determined, whose distance (||q(Y GT ) - C[k]||) to the image (q(Y GT )) of the reference trajectory (Y GT ) minimal (k̂ = argmin k ||q(Y GT ) - C[k]||); • the trajectory prediction (Y) is output, which results from the input of the embedding and the previously determined reference characteristic (C[k]) into the first machine learning model (h). [7] Method according to one of claims 2 to 6, wherein a trajectory prediction (Y) and a reference trajectory (Y GT ) are mapped onto the second feature space (C) by means of a function (q) according to claim 5 and, in the case of predetermined features with predetermined feature ranges, in a training backward path • a first distance (||Y - Y GT ||) of the trajectory prediction (Y) to the reference trajectory (Y GT ); • a second distance (||C[k̂] - q(Y)||) of a given reference characteristic (C[k̂]), whose distance (k̂ = argmin k ||q(Y GT ) - C[k]||) to the image (q(Y GT )) of the reference trajectory (Y GT ) is minimal, to the image (q(Y)) of the trajectory prediction (Y) and a loss function (L) comprising the first (||Y - Y GT ||) and the second distance (||C[k̂] - q(Y)||) is minimized. [8] Method according to one of claims 2 to 6, wherein a trajectory prediction (Y) and a reference trajectory (Y GT ) are mapped onto the second feature space (C) by means of a function (q) according to claim 5 and, in the case of predetermined features with learnable feature ranges, in a training backward path • a first distance (||Y - Y GT ||) of the trajectory prediction (Y) to the reference trajectory (Y GT ); • a second distance (||C[k̂] - q(Y GT)||) of a reference characteristic (C[k]) to the image (q(Y GT )) of the reference trajectory (Y GT ); • a third distance (||q(Y) - q(Y GT )||) of the image (q(Y)) of the trajectory prediction (Y) to the image (q(Y GT )) of the reference trajectory (Y GT ); • a first regularization term (RC=−∑j=1, j≠k^NC‖C[j]−q(YGT)‖), which regularizes that the reference characteristic (C[k̂]) of the image (q(Y GT )) of the reference trajectory (Y GT ) is learned and respective distances (||C[j] - q(Y GT )||) of the remaining characteristics (C[j]) to the image (q(Y GT )) of the reference trajectory (Y GT ) are relatively large, and a loss function (L) comprising the first (||Y - Y GT ||), second (||C[k̂] - q(Y GT )||) and third distance (||q(Y) - q(Y GT )||) and the first regularization term (RC=−∑j=1, j≠k^NC‖C[j]−q(YGT)‖) is minimized. [9] Method according to claims 2, 5 and 8, wherein in the case of learnable features, the function (q) in a training backward path of the third machine learning model is determined by minimizing an adaptive loss function (L a = a||q(Y GT ) - C[l]|| + β||q(Y GT ) - C[k̂]||) is learned in such a way that in case the first distance (||Y - Y GT ||) of the trajectory prediction (Y) to the reference trajectory (Y GT ) falls below the second threshold (t) (α = relu(||Y - Y GT || - t), β = relu(t - ||Y - Y GT ||)), the function (q) the reference trajectory (Y GT ) to the reference characteristic (C[k̂]). [10] Method according to claim 9, wherein the adaptive loss function (L a ) a second regularization term (R q = ||W q ||2), the parameter (W=(w1, ..., w q)) of the function (q) is regularized. [11] Method according to one of claims 7-10, wherein the respective loss function has at least one term (||P||1 - logP k̂ ) includes. [12] Method according to one of the preceding claims, wherein the embedding (E) is determined as a multi-agent scene embedding as follows: • Coding of environmental features including trajectory histories; • Encoding traffic scene information comprising fixed static environment features and state-changing static environment features; • Combining the preceding encodings into a hybrid scene representation comprising at least a first layer comprising the rigid static environmental features, a second layer comprising the state-changing static environmental features and a third layer comprising dynamic environmental features comprising the trajectory histories; • Determining interactions between the static, dynamic and between the static and dynamic environment features based on the hybrid scene representation, generating a first tensor embedding of the rigid static environment features, a second tensor embedding of the state-changing static environment features and a third tensor embedding of the dynamic environment features, and merging the first, second and third tensor embeddings into a multi-agent scene tensor; • Extract the features of the multi-agent scene tensor per road user at the location corresponding to the road user's coordinates, fuse these features with the third tensor embedding of the road user and generate the multi-agent scene embedding per road user and per traffic scene. [13] Computer program for characterizing future trajectories (Y) of road users, comprising commands that cause a hardware platform of a remote system, a driver assistance system or an autonomous driving system to carry out the steps of a method according to one of the preceding claims when the computer program is executed by the hardware platform.

Citation Information

Patent Citations

  • Method for assisting rider when feeding e.g. vehicle, involves proving information, warning and automatic engagement, which results during risk of collision and / or secondary collision with highest priority in priority list

    DE102012009297A1

  • Method for predicting the movement of a vehicle and at least one other vehicle

    DE102019216235A1

  • Method for operating a driver assistance device of a vehicle, driver assistance device and vehicle, comprising at least one driver assistance device

    DE102019218455A1

  • Computer-implemented method and computer program product for obtaining an environment scene representation for an automated driving system, computer-implemented method for learning a prediction of environment scenes for an automated driving system, and control unit for an automated driving system.

    DE102020210379A1

  • Computer-implemented method, computer program and arrangement for predicting and planning trajectories

    DE102021203440A1