Method for predicting trajectory of traffic participant

Abstract

According to various embodiments, a method for predicting a trajectory of a traffic participant is provided, the method comprising: representing a traffic scene as a proxy interaction graph having nodes and a plurality of edges for traffic participants corresponding to a target vehicle and for one or more other traffic participants, assigning a respective edge type to each edge between two of the nodes, the edge type indicating the types of movement of the traffic participants represented by the nodes relative to each other on the respective lanes; processing the proxy interaction graph by a graph converter to determine embedding of the target vehicle and one or more other traffic participants, where the graph converter has an attention mechanism that takes into account a side type of a side of the proxy interaction graph; and predicting at least one trajectory of the target vehicle based on the embedding.

Description

Technical Field

[0001] The present disclosure relates to a method for predicting trajectories of traffic participants. Background Art

[0002] In the field of autonomous systems, the prediction of the behavior of mobile objects in the environment of a controlled agent (such as a vehicle) is an important task in order to reliably control the agent and, for example, avoid collisions.

[0003] For example, to achieve high-performance and safe autonomous driving, an autonomous vehicle must be able to predict the future development of the driving situation, particularly including the behavior of other vehicles in its environment. Therefore, determining the control of the autonomous vehicle, for example, by the future trajectory that the autonomous vehicle should follow, must also include the behavior of other vehicles. Vehicles considered by the autonomous vehicle (host vehicle) are also called target vehicles.

[0004] Correspondingly, reliable methods are needed to predict the behavior of agents, i.e., to determine (expected) trajectories in multi-agent scenarios.

[0005] The publication “nuScenes: A multimodal dataset for autonomous driving” by H. Caesar et al., 2020, https: / / arxiv.org / abs / 1903.11027, hereinafter referred to as Reference 1, describes the nuScenes dataset. Summary of the Invention

[0006] According to various embodiments, a method for predicting a trajectory of a traffic participant is provided, the method comprising:

[0007] a representation of the traffic scenario as an agent interaction graph, the agent interaction graph comprising a node for each traffic participant corresponding to the target vehicle and for one or more other traffic participants and a plurality of edges, wherein each edge between two of the nodes is assigned a corresponding edge type, which indicates a type of movement of the traffic participants represented by the nodes relative to one another on the corresponding lane;

[0008] processing the agent interaction graph through a graph transformer to determine embeddings for a target vehicle and one or more other traffic participants, wherein the graph transformer has an attention mechanism that considers edge types of edges of the agent interaction graph; and

[0009] Based on these embeddings, predict at least one trajectory of the target vehicle.

[0010] The attention algorithm typically has a computation of an attention ("Attention") between two nodes, with which a message ("Message") between these two nodes is weighted, with which a node attribute of one of these two nodes is weighted.

[0011] The method enables a prediction of trajectories of surrounding vehicles for an autonomous driving system of a controlled vehicle (also referred to as "ego vehicle"). Here, for example in the agent interaction graph, the influence of surrounding (or "nearby") traffic participants (especially other vehicles) is explicitly modeled by defining various relationship types between the target vehicle and the surrounding traffic participants, such as "driving in front of", "on a neighboring lane and driving in the same direction", "on a neighboring lane and driving in the opposite direction", "crossing (possibly colliding)", "pedestrian crossing (possibly colliding)". This enables a differentiated and accurate modeling of the interaction and influence of other (i.e. surrounding) traffic participants, which enables a high-precision trajectory prediction.

[0012] In other words, for the prediction of vehicle trajectories, various relationship types between a target vehicle, i.e. the vehicle whose trajectory is to be predicted, and surrounding traffic participants that can influence the trajectory of the target vehicle are used. The surrounding traffic participants (especially other vehicles) typically have a strong influence on the behavior of the target vehicle. According to various embodiments, this influence is explicitly modeled by considering the position of the surrounding vehicles on the lane and the map topology with respect to the target vehicle when predicting the trajectory. For example, the speed, the driving direction and the distance between the vehicles can also be considered.

[0013] Various embodiments are explained in the following.

[0014] Embodiment 1 is a method for predicting a trajectory of a traffic participant as described above.

[0015] Embodiment 2 is the method according to embodiment 1, wherein the attention mechanism considers the edge type of the edges of the agent interaction graph in that the attention mechanism has for each edge type a respective set of attention mechanism parameters (weights, e.g. a respective weight matrix, e.g. for the combination, e.g. a weighted multiplication, of key and value, or also for the multiplication, e.g. a weighted multiplication, of the node attribute of the node in order to compute the message of the node), wherein the set of attention mechanism parameters is individually trainable (i.e. thus also possibly different, i.e. if this comes out during training, then different attention weights (e.g. weight matrices of the attention algorithm) can be used for different edge types).

[0016] Heterogeneous graphs (proxy interaction graphs) with different edge types (and different node types such as vehicles and pedestrians) can be converted or processed in this way, wherein a model for predicting trajectories is first trained and then used for predicting trajectories, which model takes into account different edge types. Since the edge types can reflect, for example, whether further traffic participants are important (i.e. have an influence) for the target vehicle (e.g. in terms of the lane on which both move), it is thereby possible to effectively take this into account when predicting trajectories.

[0017] Embodiment 3 is the method of embodiment 1 or 2, wherein each of the edges has one or more edge attribute values, the edge attribute values being indicative of quantitative characteristics of the movement of the traffic participants represented by the nodes relative to each other, and the attention mechanism takes into account the edge attribute values.

[0018] Thus, in addition to the edge types, also quantitative parameters of the (relative) movement can be taken into account, e.g. the Euclidean distance between the traffic participants, the distance along the lane between the traffic participants, the speed difference between the traffic participants, the direction difference between the traffic participants (e.g. the angular difference relative to a reference direction), and the time to collision. The edge attributes are taken into account, for example, by adding them to the previous node embedding (from the previous graph transformer layer) when updating the embedding of a node (i.e. the node attributes) in the graph transformer layer.

[0019] Embodiment 4 is the method of any one of embodiments 1 to 3, wherein the movement type is one of side-by-side, front-back, and crossing.

[0020] Which of these movement types of vehicles relative to each other is present typically has a significant influence on the way the vehicles continue to move (if they are close enough to each other). The edge type “crossing” can exist, for example, in two forms: “crossing - for another vehicle” and “pedestrian crossing”.

[0021] Embodiment 5 is the method of any one of embodiments 1 to 4, wherein the trajectory is determined further in dependence on at least one of: an encoding of the movement of the target vehicle; for each of the other traffic participants, an encoding of the movement of the other traffic participant; and an encoding of the lane nodes of one or more graphs representing one or more lanes of the traffic scene.

[0022] Thus, the embedding provided by the graph transformer is combined (or fused) with further encodings about the movement of the traffic participants or the one or more lanes for the trajectory prediction. This improves the quality of the trajectory prediction.

[0023] Embodiment 6 is a method according to any one of embodiments 1 to 5,

[0024] The method further has: controlling the vehicle (ego) in view of the at least one predicted trajectory.

[0025] Embodiment 7 is a vehicle control device set up for performing the method according to any one of embodiments 1 to 6.

[0026] Embodiment 8 is a computer program having instructions which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 6.

[0027] Embodiment 9 is a computer readable medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of embodiments 1 to 6. BRIEF DESCRIPTION OF DRAWINGS

[0028] In the drawings, like reference numerals refer to like parts throughout the various views. These drawings are not necessarily to scale, with emphasis instead generally being placed upon presenting the principles of the application. In the following description, various aspects of the application are described with reference to the following drawings.

[0029] Figure 1 A vehicle is shown.

[0030] Figure 2 A trajectory prediction process according to an embodiment is illustrated.

[0031] Figure 3 Two edge types of a lane graph are illustrated.

[0032] Figure 4 A GRU (Gated Recurrent Unit) according to an embodiment is shown.

[0033] Figure 5 Encoding of an agent trajectory by a neural network formed of GRUs is illustrated.

[0034] Figure 6 Encoding of a lane graph by a neural network formed of GRUs is illustrated.

[0035] Figure 7 A real traffic scenario and a corresponding agent interaction graph are shown.

[0036] Figure 8 A dynamic heterogeneous graph encoder (DHGE) according to an embodiment is illustrated.

[0037] Figure 9 A graph transformer is illustrated.

[0038] Figure 10 Clarified the encoding of agent interaction graphs.

[0039] Figure 11 A (machine) information fusion model according to an embodiment is shown.

[0040] Figure 12 Explained through Figure 11 The CA (cross attention) processing is performed by the sub-model of the information fusion model.

[0041] Figure 13 A flow chart is shown which illustrates a method for predicting the trajectory of a road user. DETAILED DESCRIPTION

[0042] The detailed description below refers to the accompanying drawings, which, for purposes of illustration, illustrate specific details and aspects of the present disclosure in which the present invention may be implemented. Other aspects may be used and structural, logical, and electrical modifications may be performed without departing from the scope of the present invention. The different aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0043] Various examples are described in more detail below.

[0044] Figure 1 A vehicle 101 is shown.

[0045] exist Figure 1 In the example of FIG. 1 , a vehicle 101 , for example a passenger vehicle (PKW) or a truck (LKW), is equipped with a vehicle control device 102 .

[0046] The vehicle control device 102 has data processing components, such as a processor (eg, a CPU (Central Processing Unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates and data processed by the processor 103 .

[0047] For example, the stored control software (computer program) has commands that, when executed by the processor, cause the processor 103 to implement a machine learning (ML) model 107 .

[0048] The data stored in the memory 104 can for example comprise image data detected by one or more cameras 105. The one or more cameras 105 can for example take one or more grayscale or color pictures of the environment of the vehicle 101. The vehicle control device 102 can detect objects, in particular other vehicles 108, in the environment of the vehicle 101 using the image data (or also other sources of information, like data of other types of sensors or vehicle-to-vehicle communication), and determine previous trajectories of these objects, and thus detect a traffic scenario.

[0049] The vehicle control device 102 can examine the sensor data and control the vehicle 101 as a result, i.e. determine control actions for the vehicle and signal these control actions to the respective actuators of the vehicle. In this way, the vehicle control device 102 can for example control the actuators 106, e.g. brakes, in order to control the speed of the vehicle, e.g. in order to brake the vehicle.

[0050] Here, when the control device 102 determines the future trajectory 101 of the vehicle 101, it has to include the behavior of other vehicles 108, i.e. the future trajectories of these other vehicles, into account. Therefore, the control device 106 has to predict the (future) trajectories, i.e. in other words, the traffic movements, of the other vehicles 108, which are usually "agents". In the following, the vehicle 101 for which the prediction is made, i.e. which controls or is controlled based on the prediction, is also referred to as ego vehicle. In the following, the vehicles 108 whose trajectories are predicted are also referred to as target agents or target vehicles.

[0051] While there are many models for predicting vehicle trajectories, most of them fail to adequately model the interactions between traffic participants. Some models use agent graphs, but these models often lack semantic meaning and relevant features in their relationships. However, understanding the semantic relationships between agents and the features associated with them can be crucial for accurate trajectory prediction. To fill this gap, according to various embodiments, a trajectory prediction method is used that integrates a dynamic heterogeneous agent interaction graph and a fusion module to combine different information and ensure a comprehensive understanding of the scenario graph.

[0052] Figure 2 The trajectory prediction process according to embodiments is illustrated.

[0053] As Figure 2As shown, according to one embodiment, the trajectory prediction is divided into three different stages. First, the agent information 201 is encoded into a target agent (motion) encoding 207 and all surrounding vehicles (motion) encodings 208 by means of the agent encoder 204. Furthermore, the lane graph 202 is encoded into lane encodings 209 by means of the lane encoder 205. Moreover, the agent interaction graph 203 is encoded into surrounding agent node embeddings 210 (such surrounding agent node embeddings with edges to the target agent node) and a dynamic target agent node embedding 211 by means of the dynamic heterogeneous graph encoder (DHGE) 206 (agent interaction information encoder).

[0054] Then, the information fusion model (or information fusion module) 212 integrates the different encodings (embeddings) 207-211 by means of four sub-models 213, 214, 215, 216 and forms an overall representation, i.e. a fusion encoding, in this way. Finally, the decoder 217 uses the fusion encoding to predict the multi-modal trajectory. Moreover, the dynamic target agent node embedding 211 from the agent interaction graph 203 is directly used by the interaction-based predictor 218 to predict the trajectory (as a side task). From the output of the decoder 217 and the output of the interaction-based predictor 218, the trajectories 219 of the agents (all surrounding vehicles and the target agent) are predicted.

[0055] According to one embodiment, a graph representation of the environment map in the form of a lane graph 202 is used. The motivation to use a lane graph representation is complex. In a complex urban traffic environment, an autonomous vehicle needs not only to understand the physical structure of the road, but also to understand the allowed paths available to the autonomous vehicle. This mainly includes allowed maneuver actions like turning at an intersection, changing lanes on a highway, and driving into a ramp. By representing these paths in the form of a graph, the autonomous vehicle can efficiently plan its route and ensure that it drives safely and in compliance with traffic rules.

[0056] The centerline of the lanes is located in the center of the figure representation. By focusing on the centerline that marks the middle path of each lane on the road, the overall geometry and structure of the road can be fully detected. To determine the lane graph 202, after vectorizing the map information (from, for example, an environment map available to the vehicle 101), the centerlines of the lanes are extracted. These centerlines are then represented as a directed graph, denoted by G = {V, E}. Instead of considering all lanes of the map, only the lanes within an 80-meter radius of the target agent and their connected lanes are limited. These lanes are then divided into segments, each spanning 20 meters, which are then discretized into a sequence of poses with a 1-meter pitch. Thus, each node corresponds to a 20-meter long lane segment. For other road elements, the stop lines and crosswalks within the same 80-meter radius are considered. When a polygon and a lane line intersect, the corresponding signs (which represent a stop line or a crosswalk) are combined by One-Hot encoding (i.e., an encoding that has a digit position for each such type, and contains a one in the digit position if the type applies, and zero otherwise). With this method, it can be found whether the lane matches a stop line or a crosswalk. The features of the lane nodes contain data about traffic controls. Each node v in this representation represents a sequence of pose vectors

[0057]

[0058] where each pose is characterized by the following equation:

[0059]

[0060] where the two signs represent whether the pose is on a stop line or on a crosswalk.

[0061] Here, is the local coordinate of the nth pose, and is the yaw angle of the corresponding pose.

[0062] As for the relationship between the nodes, there are two edge types in the lane graph 202.

[0063] Figure 3 The two edge types of the lane graph 202 are illustrated.

[0064] These two types are successor edges 301 and near edges 302. The successor edges 301 ensure continuity along a lane to the next node, such that a legal trajectory is maintained. If a lane node (e.g. node A) is at an intersection, then this lane node can have multiple successor connections. In contrast, these near edges 302 are intended to represent legal lane changes between adjacent lanes with the same driving direction. To this end, they connect adjacent lane nodes, e.g. within a distance of 4 meters. This threshold is chosen based on typical lane widths and a safety buffer required for a lane change. It is kept below The yaw angle difference below ensures that only lanes that run approximately parallel to each other and are not at an acute angle to each other are connected.

[0065] This is crucial to prevent illegal overtaking processes or lane changes on opposite lanes.

[0066] The target agent and its surrounding agents (in the agent information 201) are represented as a sequence of time vectors:

[0067]

[0068] where n denotes the individual agents. The surrounding agents include all agents in the considered traffic scene except the target agent, regardless of their distance or relevance to the target agent. These agents can be categorized as humans or vehicles.

[0069] Each is represented as

[0070]

[0071] The index t extends from the earliest observable frame to the current frame of the considered scene, where For example, several seconds are covered. The value yr represents the yaw rate.

[0072] As mentioned above, the agent encoder 204 encodes the trajectories of the agents (from the agent information 201), including those of the target agent and the surrounding agents, and the lane encoder 205 encodes the lane graph 202 (and thereby the features of these lane nodes).

[0073] The main goal of these encodings is to transform the raw lane and agent data into a format that can be easily processed by subsequent encoders. In one embodiment, this encoding is performed using three MLPS (Multi Layer Perceptrons).

[0074] According to one embodiment, the encoding is performed using GRUs (Gated Recurrent Units) taking into account the sequential nature of the data, such as lane node features and trajectories of agents. GRU networks, i.e. neural networks from GRUs, are a variant of recurrent neural networks. The advantage of this network is that it efficiently detects long-term dependencies contained in sequential data. By introducing two gates, an update gate and a reset gate, the GRU network is able to retain relevant data over time.

[0075] Figure 4 A GRU 400 according to an embodiment is shown.

[0076] The GRU contains an update gate 401 and a reset gate 402. These gates are crucial for determining the flow of information through the neural network (formed by multiple such GRUs). The update gate assesses how many previous states (h t-1 ) are to be retained, while the reset gate decides how to integrate the current input with the past information. Both gates act through vectors with values between 0 and 1, which model the interaction between the input data and the previous hidden state. This operational paradigm can be mathematically expressed as follows:

[0077] r t = σ(W r x t + U r h t-1 + b r ) (5)

[0078] z t = σ(W z x t + U z h t-1 + b z ) (6)

[0079]

[0080] where x t is the input at time point t, h t-1 is the previous (hidden) state, and r t and z characterize the reset gate 402 and the update gate 401. The symbol denotes element-wise multiplication. The weight matrices W, U and the bias vector b are parameters learned in training.

[0081] GRUs are particularly suitable for this task, as they can handle sequential data by detecting temporal dependencies, making them a suitable choice for the encoding of time series such as trajectories.

[0082] In this way, lane node features and agent trajectories are encoded separately from each other. In the following, for the target agent, the index ta is used, and for the surrounding agents, the index sa is used. Lane node features follow the lane direction order, while agent trajectories have a temporal order.

[0083] Figure 5 The encoding of agent trajectories by a neural network formed by GRUs (GRU encoder) 501, 502 is illustrated.

[0084] Figure 6 The encoding of lane graphs by a neural network formed by GRUs (GRU encoder) 600 is illustrated.

[0085] As shown in Figure 5 and Figure 6 , the states are updated by GRU cells and transferred to subsequent GRU cells during sequence processing. When these states pass through the GRU encoders, the final hidden states contain the spatial information contained in the sequences. This output from each GRU encoder 501, 502, 600 is used as an embedding of the movement of the target agent (h ta ), the surrounding agents (h sa ) or the lane node features (h l ). All these embeddings are represented in a 32-dimensional space.

[0086] By the above encoding of the agent information 201, i.e. the target agent encoding 207 and all surrounding vehicle encodings 208, especially all agents surrounding the (target agent, i.e. the target vehicle) within the considered scenario are represented. However, not all of these agents will influence the target vehicle. When driving, humans rely on a complex interplay of factors to decide which vehicles (or generally agents) in the environment are most important. Factors like the relative speed of surrounding agents, the distance of these surrounding agents to the target vehicle, the trajectories of these surrounding agents and traffic rules determine these decisions. Thus, for example, a vehicle located directly in front of the target vehicle and driving significantly slower can have a greater influence than a fast vehicle on a non-crossing parallel lane. Likewise, a vehicle waiting to turn from a crossing lane can have priority in the target vehicle’s decision due to traffic rules and turn-related risks.

[0087] According to one embodiment, an agent interaction graph 203 is used as input, which is derived, for example, from the nuScenes trajectory prediction graph dataset (see reference 1) and identifies the most important agents in the environment associated with the movement and decisions of the target vehicle. This agent interaction graph can also be derived from any other driving scenario, provided that map information and agent information are present. The agent interaction graph also indicates semantic relationships and properties between them.

[0088] Figure 7 A real traffic scene 701 and a corresponding agent interaction graph 702 are shown. This agent interaction graph shows the semantic relationships between agents, which are characterized by edge types: lateral (side-by-side), longitudinal (front-back), crossing, and pedestrian (or "pedestrian crossing"). The relationships between agents are characterized by these four different edge types. The first three types, i.e. longitudinal 703, lateral 704, and crossing 705, are equipped with three edge attributes, which are denoted by The attribute "distance" quantifies the Euclidean distance between two agents. The path distance measures the distance traveled along a specific path, and the edge probability indicates the probability of a specific relationship. The edge type "pedestrian 706" does not have a "path distance" attribute.

[0089] In the following it is described how the dynamic heterogeneous graph encoder (DHGE) 206 processes the agent interaction graph 702.

[0090] Figure 8 A dynamic heterogeneous graph encoder (DHGE) 800 according to an embodiment is illustrated. The dynamic heterogeneous graph encoder consists of three parts: a specific type encoder 801, an EHGT (Edge-enhanced Heterogeneous Graph Transformer) 802, and a temporal encoder (based on GRU) 803.

[0091] A different MLP is used as specific type encoder 801 in order to encode the node features of different agent types, since vehicles and pedestrians can have different behavior patterns. Then, an EHGT 802 is used in order to encode the agent interaction graph 702. Finally, a temporal encoder 803 is used in order to detect temporal information from the graph encoding across the individual timestamps.

[0092] Heterogeneous graph transformers are capable of detecting and representing different types of nodes and relationships within a single graph. The design of the heterogeneous graph transformer enables a flexible integration of various entities and their relationships, which makes the heterogeneous graph transformer particularly suitable for the current application.

[0093] In essence, HGTs (Heterogenous Graph Transformers) work with interaction matrices, which model specific elements in the relationships and jointly use parameters for better generalization. Additionally, HGTs use meta-relations in order to parameterize the weight matrices.

[0094] Figure 9 A graph transformer 900 is illustrated.

[0095] The graph transformer 900 takes as input a sub-heterogeneous graph 904 and has three components: heterogeneous mutual attention 901, heterogeneous message passing 902 and target-specific aggregation 903.

[0096] In HGT, the attention mechanism is different from the standard GAT (Graph Attention Network). Instead of relying on a single weight matrix assuming a uniform feature distribution between source and target nodes, HGT introduces mutual attention based on relation triples structured as <source node, edge, target node>. The target node t is projected onto the query vector while the source node s is mapped onto the key vector:

[0097]

[0098] where H (l-1) is the input from the previous layer and i refers to a specific attention head. Each node type has a unique linear projection.

[0099] This attention head is computed as described by the following equation:

[0100]

[0101] where is a matrix specific to the edge type 905, which aims to detect semantic relationships between nodes Φ(e). This matrix enables the model to identify and detect different semantic relationships even between nodes of the same type. The attention mechanism is determined by the dot product operation. Furthermore, d is an adaptive parameter indicating the overall meaning of each relation triple, while d denotes the dimension of the vector.

[0102] Next, the h attention heads are concatenated and a Softmax function is applied in order to determine the final attention weight for each relation triple, as defined by the following equation:

[0103]

[0104] (Att stands for “attention”).

[0105] The transmission of information from source nodes to target nodes is facilitated by the heterogeneous message passing 902. This process is done in parallel with the computation of the mutual attention. In order to obtain the i-th message head MSG_head i (s, e, t), the representation H l-1 of the source node from the previous layer is passed through a special type encoder 905.[s] M-linear projection is performed. Then, this representation is multiplied by the edge type matrix in order to integrate edge dependencies:

[0106]

[0107] Next, for each pair of nodes, h messages nodes are concatenated (tied) together in order to obtain the message

[0108] Msg HGT (s, e, t) = || MSG_head i∈[1,h] MSG_head i (s, e, t) (14).

[0109] In the subsequent aggregation phase, all messages from source nodes s to target nodes t are aggregated. The attention vector defined in equation (12) is used as weights for averaging the corresponding messages from the source nodes, as shown in equation (14). The result is the updated

[0110] Then, contains comprehensive information about the neighbors of the target node t and the relations associated with them. Next, the target node t is reclassified into its type-specific distribution and complemented with the remaining connections from the previous layer (provided through the residual connection 906):

[0111]

[0112] The Heterogeneous Graph Transformer (HGT), while similar to many other graph operators of its kind, is able to encode heterogeneous graphs, but has an important limitation: it cannot incorporate edge attributes. This means that while the HGT can detect and represent different types of nodes and relations, it does not use additional information that can be provided through attributes related to the edges. In view of this limitation, according to various embodiments, for the dynamic heterogeneous graph encoder 206, an edge-enhanced Heterogeneous Graph Transformer (EHGT for “edge-enhanced” HGT) 802 is used that can incorporate edge attributes into the mutual attention 901 and the message passing 902 and thereby provide a comprehensive graph representation.

[0113] In this case, the strategy to incorporate edge features is the Edge Concatenate HGT (ECHGT), where the edge attributes of an edge are directly appended to the source node of the edge. Essentially, in this approach, the edge attributes are treated as an extension of the source node features within the triplets. This incorporation of edge attributes is first manifested in the computation of the mutual attention, as shown in the following equation.

[0114]

[0115] In this case, the edge attribute is denoted with e_a or in the following also with E_A (standing for edge attribute).

[0116] Furthermore, the edge attribute is added to the message passing stage, as shown in the following equation.

[0117]

[0118] However, this approach encounters difficulties in the scenario of triples with multiple relations.

[0119] Accordingly, according to one embodiment, an alternative EHGT strategy is used, also referred to as EAHGT (edge attribute heterogeneous graph transformer), in which the edge attribute matrix is not only used for the computation of the attention (as shown in equation (20) below) but also for the message passing (as shown in equation (21) below). This matrix serves as a representation of the edge attribute information.

[0120]

[0121] With the help of the EHGT, heterogeneous graphs with edge attributes can be processed.

[0122] According to various embodiments, the EHGT is used as a GNN (Graph Neural Network) operator in order to process the interactions of agents in the heterogeneous graph 203. In real-world scenarios, this is often just the direct neighbor of the target vehicle, which significantly influences the behavior of the target vehicle. Accordingly, according to one embodiment, the aggregation is limited to one hop, as Figure 10 shown.

[0123] Figure 10 The encoding of the agent interaction graph 1001 by two specific types of encoders is illustrated, a first encoder 1002 for pedestrians and cyclists and a second encoder 1003 for vehicles, followed by an EHGT 1004. The result is a graph embedding with a target node embedding 1005, as well as embeddings of surrounding nodes with directed edges (arrows) to the target node 1006 and surrounding nodes with directed edges (arrows) from the target node 1007.

[0124] It is to be noted that the neighbors reached by the arrows coming from the target (the surrounding nodes having a directed edge from the target node 1006) also have an influence on the target. Therefore, the embedding of these nodes is added to the surrounding agent node embedding 210.

[0125] It is also to be noted that the agent embeddings, as mentioned with reference to Figure 10 , are different from the agent embeddings described with reference to Figure 4 , Figure 5 and Figure 6 : the latter represent all the agents present in a particular scenario, whereas the agent embeddings in the agent interaction information encoder 206 are selected so that they represent the agents that have a high probability of influencing the behavior of the target agent.

[0126] After the encoding of the agent information, i.e. the encoding of the heterogeneous graph, the node embeddings are obtained, which contain both spatial and semantic data for each particular timestamp. However, the temporal link or continuity between these timestamp-specific embeddings is still missing. In order to integrate this temporal dimension, which records the development over time of the traffic scenario, a GRU network, i.e. a GRU-based temporal encoder 803, is used. The task of this GRU network 803 is to encode the target node embeddings from all the observed traffic scenarios in chronological order, which results in a dynamic target node graph embedding g t (corresponding to the dynamic target agent node embedding 211), as shown in Figure 8 .

[0127] At the same time, the embedding g s of the surrounding nodes from the last observed traffic scenario at t = 0 is kept, in particular the nodes from which the arrows come from the target agent. This step ensures that potential agents influencing the prediction are not ignored.

[0128] Therefore, the output of the heterogeneous graph encoder (DHGE) 206, 800 is the embeddings 804, which contain the surrounding agent node embedding 210 (or the surrounding agent node embeddings when considered separately) and the dynamic target agent node embedding 211.

[0129] As already mentioned above in connection with Figure 2 , now the information fusion model 212 fuses the various encodings (or "embeddings") 207-211 by means of four submodels 213, 214, 215, 216 in order to represent the dynamics and interactions in the traffic scenario comprehensively:

[0130] • the movement encoding (h t ) of the target agent: this encoding 207 detects the main movements and patterns related to the target agent and enables an understanding of the inherent movement dynamics of the target agent.

[0131] • Encoding of surrounding agents (h s ): This encoding 208 provides information about all non-target agents within the traffic scene.

[0132] • Encoding of lane nodes (h l ): This encoding 209 gives information about the structured lanes in the scene.

[0133] • Dynamic target node graph embedding (g t ): This encoding 210 does not rely on the movement pattern only, but represents the interactions and relationships of the target agent in the form of a dynamic graph. This encoding provides a deeper relationship understanding of the target position within the wider scene.

[0134] • Interaction embedding of surrounding agents (g s ): This embedding 211 is similar to the embedding of the target node in the graph, but for the surrounding agents at the current timestamp. It detects how these agents interact with each other and with the target, thereby forming a complete picture of the dynamics between the agents.

[0135] These encodings 207-211 form a multi-faceted representation of the scene and enable a more comprehensive analysis and more accurate predictions.

[0136] Figure 11 A (machine) information fusion model 1100 according to an embodiment is shown. This information fusion model is an example of the information fusion model 212 and serves to merge the encodings 207-211.

[0137] The output of the information fusion model 212 is denoted by f f (standing for “fusion”).

[0138] As already described in connection with Figure 2 The information fusion model 1100 has four sub-models (or sub-modules) 1101-1104, each of which is specifically set up for processing a different aspect of the information.

[0139] These sub-models 1101-1104 are described in the following.

[0140] First sub-model 1101 (surrounding agents and their interactions): The encoding h s contains information about all agents in the traffic scene except the target agent. To extract a meaningful representation highlighting the most influential surrounding agents and to understand the interaction implications of these surrounding agents, the first sub-model 1001 uses a Cross-Attention (CA) mechanism 1105 as follows:

[0141] sai = CA(h s , gs )+h s (22)

[0142] CA mechanism in h s (ambient agents' GRU embedding representation) and g s (embedding that detects the interaction between these agents) plays a role. The goal here is to weigh and understand which relationships of the surrounding agents are most important in a given context.

[0143] An important supplement to the CA mechanism is the integration of skip connections 1111 (implemented by adding h s ). This supplement facilitates the transmission of raw information across CA layers and ensures that essential information is not lost during the attention process. The use of skip connections within the model provides two main advantages:

[0144] - Handling graphs with few edges: In certain traffic scenarios, the lack of relationships between agents can pose difficulties. In this case, the direct path in the form of a skip connection helps to ensure that the model can still make reasonable predictions at all times without over-relying on non-existent predictions.

[0145] - Model stability: The skip connections for this stabilize deep models (i.e. models with multiple layers). These skip connections enable gradients to flow directly backwards through the respective neural network, which can prevent the vanishing gradient problem and support the convergence of the model.

[0146] Second sub-model 1102 (ambient agents with interaction and lane information): This second sub-model implements a (second) cross-attention mechanism 1106 between ambient agents with interaction encoding sa i and lane node representation h l and ensures that the embedding of a lane node is refined based on the presence and behavior of agents in its vicinity. The approach of only considering agents within a certain radius around each lane node ensures that the model focuses on the most relevant interactions and the influence of agents from far away, which can be irrelevant, is reduced.

[0147] Figure 12 The CA process by the second sub-model 1102 is illustrated.

[0148] The cross-attention mechanism 1106 takes keys K 1202 and values V 1203 from the ambient agents with interaction encoding sa i . Queries Q 1201 are derived from the lane encoding h l . The output 1204 of this attention process is a representation that fuses information about the environment of a lane node with the lane node's own features. This output is h lThe link provides a more comprehensive representation of each lane node, which now also contains the context of nearby agents.

[0149] The second sub-model 1102 also contains a GAT (Graph Attention Network) encoder for the lane nodes, i.e. a GNN (Graph Neural Network) 1107: According to the cross-attention mechanism 1106, it is ensured that the lane nodes also detect context information from their neighboring lane nodes. The use of the GAT encoder 1107 ensures that the embedding of a lane node is updated based on its interaction with other connected lane nodes. This step enables the model to detect more details about how lane configurations and neighbor relationships can influence agent behavior.

[0150] The combination of the cross-attention mechanism 1106 and the GAT encoder 1107 results in a final encoding h lf of the lane information, which is both context-related information (about the interactive behavior of the agents) and structured information (about the lane configuration). This dual context-related approach, as it is through the equation

[0151] h lf = GNN(CA(h l ,sa i )) (23)

[0152] As described, it can provide a more accurate recognition about the way an agent can behave in a particular traffic scenario.

[0153] The third sub-model 1103 (target agent and its dynamic interaction):

[0154] Similar to the first sub-model 1101, the third sub-model 1102 refines the encoding h t of the target agent by fusing it with the dynamic target node graph embedding g t according to

[0155] ta i = CA(h t ,g t )+h t (24)

[0156] By this, it is ensured that the target agent is aware of a broader context of its operation, especially when considering its interaction with nearby agents.

[0157] Thus, the third sub-model 1103 contains:

[0158] - (third) cross-attention mechanism 1108 for target encoding: this is an attention mechanism in which, from g t key and value are derived, which contain the dynamic interaction of the target agent with the surrounding agents. Queries come from h t which represents the embedding of the target agent. This attention medium selectively equips the representation of the target agent with relevant interaction context and thereby creates a hybrid embedding that balances individual behavior with interaction patterns.

[0159] - (second) skip connection 1109: this skip connection ensures that the original encoding h t is not completely overshadowed by the interaction information embedding.

[0160] The output of the third submodel 1103 is an encoding t ai (target agent with interaction) that contains not only information about its intrinsic properties but also information about its interaction with other agents.

[0161] Fourth submodel 1104 (interaction encoding with lane and final encoding of target agent): the fourth submodel 1104 contains a (fourth) CA mechanism 1110 that helps to measure the relevance of various features of the lane final encoding h lf for the target agent with interaction encoding t ai :

[0162] f f = Conc(ta i , CA(ta i , h lf )) (25)

[0163] In this case, "Conc" stands for concatenation. Intuitively, the fourth CA mechanism 1110 enables the model to focus on the most relevant lane-related information while taking into account the interaction of the target agent. To this end, the updated and aggregated lane final encoding h lf is linearly projected as key and value, and the target agent with interaction encoding t ai is linearly projected as attention query. According to the fourth CA mechanism 1110, t ai is linked with the CA output. This link ensures that not only the weighted information from the lane is detected, but also the original interaction details of the target agent are preserved. Finally, a fused encoding f f is obtained, which serves as the final encoding provided by the information fusion module.

[0164] Thus, the fourth submodel 1104 contains:

[0165] - linear projection: the encoding h lfFirst, a linear transformation is applied in order to derive keys and values, and to have an interaction coding t ai The target agent is projected as a query.

[0166] - CA mechanism for fusion 1110: Then, an attention mechanism 1110 is applied. This attention mechanism ensures that the fused encoding detects the most important details from the lane perspective as well as from the target agent perspective.

[0167] - Link for final fusion: The output of the attention mechanism 1110 is associated with t ai , which is a weighted value of the lane node information about the target agent. This provides refined target agent information with the most relevant lane context. The link represents the final encoding f f generated by the information fusion module 212. This representation contains a holistic and comprehensive understanding of the situation of the target agent in the traffic scene and records the behavior of the target agent, the interactions of the target agent, and the relationship of the target agent with the lane.

[0168] The fourth sub-model 1104 represents the final process of information fusion. This process provides a complete encoding that includes not only specific details from the target agent’s perspective but also the interactions of the target agent with other people and paths in its surroundings.

[0169] The decoder 217 uses the latent variable z in order to realize different movement profiles. With the latent variable z of a Gaussian distribution and the link of the fused encoding f f , the decoder 217 is able to generate different movement profiles in order to take into account the inherent uncertainty. Then, an MLP is used in order to output k modes of future trajectories :

[0170]

[0171] Then, K-Means-Clustering is used and the cluster centers are outputted in the form of K predictions as the final output.

[0172] Furthermore, the embedding g t of the dynamic target node graph directly becomes the trajectory and in this way again (in the same way as described above) the final output This ensures that the model can use rich relational information from the heterogeneous graph and does not neglect key social interactions when creating predictions.

[0173] The decoder 217 is trained with an average Winner-Takes-All average offset error. In this way two losses are determined: a fusion regression loss (where only the best mode (Winner-Takes-All) from the output of the information fusion model is considered) and a graph regression loss (from the embedding g t of the best mode) of the dynamic target node graph:

[0174]

[0175]

[0176] Then, the model is trained with a total loss

[0177]

[0178] combining the two losses.

[0179] The combined loss uses two scalar weights λ1and λ2to ensure that the model does not over-rely on a single information source. The combined loss integrates the advantages of the two individual losses in order to make the model learn from the rich spatial, semantic and temporal features embedded in the individual losses and to ensure a comprehensive learning process.

[0180] In summary, according to various embodiments, there is provided a method as shown in Figure 13

[0181] Figure 13 A flowchart 1300 is shown, illustrating a method for predicting a trajectory of a traffic participant.

[0182] At 1301, a traffic scene is represented as an agent interaction graph having nodes for a traffic participant corresponding to a target vehicle and for one or more other traffic participants and having edges, wherein each edge between two of the nodes (i.e. each edge comprised between two nodes of the agent interaction graph representing traffic participants) is assigned a respective edge type indicating a type of movement of the traffic participants represented by the nodes relative to each other on a respective lane.

[0183] At 1302, the agent interaction graph is processed by a graph transformer to determine an embedding of the target vehicle and the one or more other traffic participants, wherein the graph transformer has an attention mechanism (also commonly denoted in English as “Attention”) that takes into account the edge types of the edges of the agent interaction graph.

[0184] ​At 1303, at least one trajectory of the target vehicle (and, if necessary, also one or more trajectories of other traffic participants) is predicted in accordance with these embeddings.

[0185] Figure 13 The method of the present application can be executed by one or more computers having one or more data processing units. The term “data processing unit” can be understood as any kind of entity capable of processing data or signals. For example, these data or signals can be processed in accordance with at least one, that is, one or more than one, specific function performed by the data processing unit. The data processing unit can comprise an integrated circuit of an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or any combination or configuration thereof. Any other means for implementing the respective functions described in more detail herein can also be understood as a data processing unit or logic circuitry. One or more of the method steps described in detail herein can be implemented (e.g., realized) by a data processing unit by one or more specific functions performed by the data processing unit.

[0186] Accordingly, the method is, in accordance with various embodiments, computer-implemented, in particular.

[0187] Traffic situations can be detected in dependence on sensor data. For this purpose, various embodiments can receive and use sensor data from various sensors such as video, radar, lidar (LiDAR), ultrasound, motion, thermal imaging, etc.

[0188] These predicted trajectories can be used for controlling the ego vehicle (i.e., considered here, e.g., the trajectory of the ego vehicle is planned such that no collision occurs when the predicted trajectories are assumed to be correct). As mentioned above, the graph transformer can be part of a larger machine learning model, which is, for example, trained end-to-end, i.e., for example, in dependence on example scenarios having future trajectories as ground-truth for supervised learning.

Claims

1. A method for predicting a trajectory of a traffic participant (108), comprising: A traffic scenario is represented as an agent interaction graph (203, 1001), which has a node for a traffic participant corresponding to a target vehicle (108) and for one or more other traffic participants (108) and has a plurality of edges, wherein: Assigning a corresponding edge type to each edge between two of the nodes, the edge type indicating a type of movement of traffic participants (108) represented by the nodes relative to each other on the corresponding lane; processing the agent interaction graph (203, 1001) by a graph transformer (802, 900, 1004) to determine embeddings of the target vehicle (108) and one or more other traffic participants (108), wherein the graph transformer has an attention mechanism that considers edge types of edges of the agent interaction graph (203, 1001); and At least one trajectory of the target vehicle (108) is predicted based on the embedding.

2. The method according to claim 1, wherein The attention mechanism takes into account edge types of edges of the agent interaction graph (203, 1001) in such a way that the attention mechanism has a corresponding set of attention mechanism parameters for each edge type, wherein the sets of attention mechanism parameters are individually trainable.

3. The method according to claim 1 or 2, wherein: Each of the edges has one or more edge attribute values ​​that indicate quantitative characteristics of the movement of traffic participants (108) represented by the nodes relative to each other, and the attention mechanism takes the edge attribute values ​​into account.

4. The method according to any one of claims 1 to 3, wherein The movement type is one of side by side, front to back, and cross.

5. The method according to any one of claims 1 to 4, wherein The trajectory is also determined based on at least one of: an encoding of the movement of the target vehicle (108); for each of the other traffic participants, an encoding of the movement of the other traffic participant (108); and an encoding of lane nodes of one or more graphs (202) representing one or more lanes of the traffic scene.

6. The method according to claim 1, further comprising: controlling the vehicle (101) taking into account at least one predicted trajectory. 7 . A vehicle control device configured to carry out the method according to claim 1 . 8 . A computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to claim 1 .

9. A computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images