Method for predicting trajectories of road users

Abstract

According to various embodiments, a method for predicting trajectories of road users is provided, comprising representing a traffic scene as an agent interaction graph, which has a node for a road user corresponding to a target vehicle and for one or more other road users, and has a plurality of edges, wherein each edge between two of the nodes is assigned a respective edge type that indicates a type of movement of the road users represented by the nodes relative to each other on a respective roadway, processing the agent interaction graph by a graph transformer to determine embeddings of the target vehicle and the one or more other road users, wherein the graph transformer has an attention mechanism that takes into account the edge types of the edges of the agent interaction graph,and predicting at least one trajectory of the target vehicle from the embeddings.,

Description

[0001] The present disclosure relates to methods for predicting trajectories of road users.

[0002] In the field of autonomous systems, predicting the behavior of moving objects in the environment of a controlled agent (such as a vehicle) is an important task in order to reliably control the agent and, for example, to avoid collisions.

[0003] For example, to enable high-performance and safe automated driving, an autonomous vehicle must be able to anticipate the future development of a driving situation, which particularly includes the behavior of other vehicles in the autonomous vehicle's vicinity. The determination of the autonomous vehicle's control, e.g., represented by a future trajectory that the autonomous vehicle should follow, must therefore include the behavior of other vehicles. The vehicles to be considered by the autonomous vehicle (ego vehicle) are also called target vehicles.

[0004] Accordingly, reliable approaches to predicting agent behavior, i.e., to determine (expected) trajectories in a multi-agent scenario, are desirable.

[0005] The publication H. Caesar et al., “nuScenes: A multimodal dataset for autonomous driving,” 2020, https: / / arxiv.org / abs / 1903.11027, hereinafter referred to as Reference 1, describes the nuScenes dataset.

[0006] According to various embodiments, a method for predicting trajectories of road users is provided, comprising: • Representing a traffic scene as an agent interaction graph, which has one node for each road user corresponding to a target vehicle and one or more other road users, and has several edges, wherein each edge between two of the nodes is assigned a respective edge type that specifies a type of movement of the road users represented by the nodes relative to each other on a respective roadway • Processing the agent interaction graph by a graph transformer to determine embeddings of the target vehicle and the one or more other road users, wherein the graph transformer has an attention mechanism that takes into account the edge types of the edges of the agent interaction graph and • Predict at least one trajectory of the target vehicle from the embeddings.

[0007] The attention algorithm typically involves calculating an attention between two nodes, weighting a message between the two nodes, and weighting node attributes of one of the two nodes.

[0008] The method enables the prediction of the trajectories of surrounding vehicles for an automated driving system of a vehicle to be controlled (also referred to as an "ego vehicle"). For example, the influence of surrounding (or "surrounding") road users (in particular other vehicles) is explicitly modeled in the agent interaction graph by defining various relationship types between the target vehicle and the surrounding road users (such as "driving one behind the other", "in adjacent lanes and traveling in the same direction", "in adjacent lanes and traveling in the opposite direction", "crossing (potential collision)", "crossing pedestrians (potential collision)"). This enables the interaction and influence of the other (i.e., surrounding) road users to be modeled in a differentiated and precise manner, leading to a trajectory prediction with high accuracy.

[0009] In other words, for the prediction of vehicle trajectories, various types of relationships are used between the target vehicle, i.e., the vehicle whose trajectory is to be predicted, and the surrounding road users that could influence the trajectory of the target vehicle. Surrounding road users (especially other vehicles) typically have a strong influence on the behavior of a target vehicle. According to various embodiments, this influence is explicitly modeled by taking into account the position of the surrounding vehicles in the lane and the map topology relative to the target vehicle during trajectory prediction. For example, the speed, directions of travel, and distances between the vehicles can also be taken into account.

[0010] Various examples of implementation are given below.

[0011] Embodiment 1 is a method for predicting trajectories of road users as described above.

[0012] Embodiment 2 is a method according to embodiment 1, wherein the attention mechanism takes into account the edge types of the edges of the agent interaction graph by having a respective set of attention mechanism parameters (weights, e.g. a respective weight matrix, e.g. for combining, e.g. weighted multiplication, of keys and values ​​or also for, e.g. weighted multiplication of the node attributes of a node in order to calculate a message of the node) for each edge type, wherein the sets of attention mechanism parameters are individually trainable (ie can thus also be different, ie different attention weights (e.g. weight matrices for the attention algorithm) can be used for different edge types if this results in training).

[0013] A heterogeneous graph (agent interaction graph) with different edge types (and also node types such as vehicles and pedestrians) can be transformed or processed in this way. A model for predicting trajectories is first trained and then used to predict trajectories that take different edge types into account. Since an edge type can, for example, reflect whether another road user is important for a target vehicle (e.g., with regard to the lanes in which both are moving) (i.e., has an influence on the target vehicle), this can be effectively taken into account in trajectory prediction.

[0014] Embodiment 3 is a method according to embodiment 1 or 2, wherein each of the edges has one or more edge attribute values ​​that indicate quantitative properties of the movement of the road users represented by the nodes relative to each other and that the attention mechanism takes into account.

[0015] In addition to edge types, quantitative variables of (relative) movement can also be considered, e.g., the Euclidean distance between road users, the distance between road users along the lane, the speed difference between road users, the direction difference between road users (e.g., angle difference with respect to a reference direction), and the time to collision. Edge attributes are considered, for example, by adding them to a previous node embedding (from the previous graph transformer layer) in a layer of the graph transformer when updating the embedding of a node (i.e., the node attributes).

[0016] Embodiment 4 is a method according to any one of Embodiments 1 to 3, wherein the type of movement is one of side by side, side by side, and crossing.

[0017] Which of these movement types the vehicles exhibit relative to each other typically has a major influence on how the vehicles continue to move (provided they are sufficiently close to each other). For example, the "crossing" edge type can exist in two forms: "crossing - for another vehicle" and "crossing pedestrians."

[0018] Embodiment 5 is a method according to any one of embodiments 1 to 4, wherein the trajectories are further determined from at least one of a coding of the movement of the target vehicle, a coding, for each of the other road users, the movement of the other road user, and codings of lane nodes of one or more graphs representing one or more lanes of the traffic scene.

[0019] The embeddings provided by the graph transformer are combined (or fused) with additional encodings regarding the movements of road users or the lane(s) for trajectory prediction. This increases the quality of the trajectory prediction.

[0020] Embodiment 6 is a method according to one of the embodiments 1 to 5, further comprising controlling an (ego) vehicle taking into account the at least one predicted trajectory.

[0021] Embodiment 7 is a vehicle control device configured to carry out a method according to one of embodiments 1 to 6.

[0022] Embodiment 8 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 6.

[0023] Embodiment 9 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 6.

[0024] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a vehicle. Fig. 2 illustrates a trajectory prediction flow according to one embodiment. Fig. Figure 3 illustrates two types of edges of the lane graph. Fig. 4 shows a GRU (Gated Recurrent Unit) according to one embodiment. Fig. Figure 5 illustrates the encoding of agent trajectories by neural networks formed from GRUs. Fig. Figure 6 illustrates the encoding of the lane graph by a neural network formed by GRUs. Fig. Figure 7 shows a real traffic scene and a corresponding agent interaction graph. Fig. 8 illustrates a dynamic heterogeneous graph encoder (DHGE) according to one embodiment. Fig. Figure 9 illustrates a graph transformer. Fig. Figure 10 illustrates the coding of an agent interaction graph. Fig. 11 shows a (machine) information fusion model according to one embodiment. Fig. Figure 12 illustrates CA (cross attention) processing using a submodel of the information fusion model of Fig. 11. Fig. Figure 13 shows a flowchart illustrating a method for predicting trajectories of road users.

[0025] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0026] Various examples are described in more detail below.

[0027] Fig. 1 shows a vehicle 101.

[0028] In the example of Fig. 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device 102.

[0029] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates and data processed by the processor 103.

[0030] For example, the stored control software (computer program) includes instructions that, when executed by the processor, cause the processor 103 to implement a machine learning (ML) model 107.

[0031] The data stored in memory 104 can, for example, include image data captured by one or more cameras 105. The one or more cameras 105 can, for example, capture one or more grayscale or color photos of the surroundings of the vehicle 101. Using the image data (or data from other information sources, such as other types of sensors or vehicle-to-vehicle communication), the vehicle control device 102 can detect objects in the surroundings of the vehicle 101, in particular other vehicles 108, and determine their previous trajectories, thus capturing a traffic scene.

[0032] The vehicle control device 102 can examine the sensor data and control the vehicle 101 according to the results, ie, determine control actions for the vehicle and signal them to the respective actuators of the vehicle. For example, the vehicle control device 102 can control an actuator 106 (e.g., a brake) to control the speed of the vehicle, e.g., to decelerate the vehicle.

[0033] The control device 102 must take into account the behavior of the other vehicles 108, i.e., their future trajectories, when determining a future trajectory 101 for the vehicle 101. The control device 106 must therefore predict the (future) trajectories of the other vehicles 108 (generally "agents"), i.e., in other words, traffic movements. The vehicle 101 for which the prediction is made (i.e., which, for example, steers or is controlled based on the prediction) is also referred to below as the ego vehicle. A vehicle 108 whose trajectory is predicted is also referred to below as the target agent or target vehicle.

[0034] While many models exist for predicting vehicle trajectories, most models fail to adequately model the interactions between road users. While some models utilize agent graphs, they often lack semantic meaning and related features in their relationships. However, understanding the semantic relationships and related features between agents can be crucial for accurate trajectory prediction. To address this gap, various embodiments utilize a trajectory prediction approach that integrates a dynamic heterogeneous agent interaction graph and a fusion module to combine diverse information and ensure a comprehensive understanding of a scene graph.

[0035] Fig. 2 illustrates a trajectory prediction flow according to one embodiment.

[0036] As in Fig. As shown in Figure 2, according to one embodiment, the trajectory prediction is divided into three different phases. First, the agent information 201 is encoded by an agent encoder 204 into a target agent (motion) encoding 207 and an all-surrounding-vehicles (motion) encoding 208. Furthermore, a lane graph 202 is encoded by a lane encoder 205 into a lane encoding 209. Furthermore, an agent interaction graph 203 is encoded by a dynamic heterogeneous graph encoder (DHGE) 206 (agent interaction information encoder) into an environment agent node embedding 210 (one with an edge to the target agent node) and a dynamic target agent node embedding 211.

[0037] Subsequently, an information fusion model (or information fusion module) 212 integrates the various encodings (embeddings) 207-211 using four submodels 213, 214, 215, 216, thus forming a holistic representation, i.e., a fused encoding. Finally, a decoder 217 uses the fused encoding to predict a multimodal trajectory. Furthermore, the dynamic target agent node embedding 211 from the agent interaction graph 203 is directly used by an interaction-based predictor 218 to predict the trajectory (as an auxiliary task). The trajectories 219 of the agents (all surrounding vehicles and the target agent) are predicted from the output of the decoder 217 and the output of the interaction-based predictor 218.

[0038] According to one embodiment, a graphical representation for an environment map in the form of lane graph 202 is used. The motivation for using a lane graph representation is complex. In the complex environment of urban traffic, it is essential for an autonomous vehicle to know not only the physical layout of the road, but also the permissible paths available to it. These include, among other things, permissible maneuvers such as turning at intersections, changing lanes on highways, and entering ramps. By representing these paths in the form of a graph, the autonomous vehicle can plan its route efficiently and ensure that it adheres to traffic laws and drives safely.

[0039] The centerlines of the lanes are at the center of this graphical representation. By focusing on the centerlines, which mark the middle path of each lane on the road, the overall geometry and structure of the road can be adequately captured. To determine the lane graph 202, the centerlines of the lanes are extracted after vectorizing map information (from an environment map available, for example, to vehicle 101). They are then represented as a directed graph denoted by G = {V, E}. Instead of considering all lanes in the map, the focus is on those lanes and their connecting segments that are located within an 80-meter radius of the target agent. These lanes are then divided into segments, each extending over 20 meters, before being discretized into a sequence of poses spaced 1 meter apart.Thus, each node corresponds to a 20-meter-long roadway segment. For other road elements, stop lines and pedestrian crossings within the same 80-meter radius are considered. When polygons and lane lines intersect, the corresponding flags (indicating a stop line or pedestrian crossing) are combined using one-hot encoding (i.e., an encoding that has one bit position for each such type, containing a one in one bit position if the type applies and zero otherwise). This method can be used to determine whether lanes coincide with stop lines or pedestrian crossings. The features of the lane nodes contain the traffic control data. Each node v in this representation denotes a sequence of pose vectors. f1:Nv=[f1v,f2v,...,fNv] where each pose is characterized by: fnv=[xnv,ynv,θnv,flagnv,flagnv] where the two flags indicate whether the pose is on a stop line or on a pedestrian crossing.

[0040] These include xnv,ynv the local coordinates for the n-th pose and θnv the yaw angle of the respective pose.

[0041] Regarding the relationships between nodes, there are two types of edges in the lane graph 202.

[0042] Fig. 3 illustrates two types of edges of the lane graph 202.

[0043] The two types are successor edges 301 and proximal edges 302. A successor edge 301 ensures continuity to the next node along a lane, thus maintaining a legitimate trajectory. If a lane node (e.g., node A) is located at an intersection, it can be connected to multiple successors. In contrast, the proximal edges 302 are intended to represent legal lane changes between adjacent lanes that have the same direction of travel. For this purpose, they connect adjacent lane nodes that are within a distance of, for example, 4 meters. This threshold was chosen based on typical lane widths and the safety buffer required for a lane change. The yaw angle difference, which π4 ensures that only lanes that are approximately parallel to each other and not at a sharp angle to each other are connected.

[0044] This is crucial to prevent illegal overtaking or lane changes into lanes in the opposite direction.

[0045] Both the target agent and the agents surrounding it are represented (in the agent information 201) as sequences of temporal vectors: fnT=[fnt=−4,fnt=−3,...,fnt=0] where n indicates the individual agent. The surrounding agents include all agents other than the target agent in the considered traffic scene, regardless of their distance or relevance to the target agent. They can be categorized as either human or vehicle.

[0046] Each fnt is represented as fnt=[xt,yt,velt,acct,yrt]

[0047] The index t extends from the earliest observable frame to the current frame of the scene under consideration, where fnt for example, covering a few seconds. The value yr indicates the yaw rate.

[0048] As described above, the agent encoder 204 encodes the trajectories of the agents (from the agent information 201) including those of the target agent and the surrounding agents, and the lane encoder 205 encodes the lane graph 202 (and thus the features of the lane nodes).

[0049] The main goal of these encodings is to convert the raw lane and agent data into a format that can be easily processed by a subsequent encoder. In one embodiment, three MLPS (Multilayer Perceptrons) are used for encoding.

[0050] Given the sequential nature of the data, such as the characteristics of the lane nodes and the trajectories of the agents, one embodiment uses Gated Recurrent Units (GRUs) for encoding. A GRU network (i.e., a neural network composed of GRUs) is a variant of a recurrent neural network. Its strength lies in its ability to effectively capture long-term dependencies contained in sequential data. By introducing two gates, namely the update and reset gates, a GRU network is able to retain relevant data over time.

[0051] Fig. 4 shows a GRU 400 according to one embodiment.

[0052] The GRU contains an update gate 401 and a reset gate 402. These gates are crucial for determining the flow of information through the neural network (formed from several such GRUs). The update gate evaluates how much of the previous state (h t-1 ) should be maintained, while the reset gate decides how to integrate the current input with past information. Both gates operate on vectors with values ​​between 0 and 1 that modulate the interaction between the input data and the previous hidden state. This operational paradigm can be represented mathematically as follows: rt=σ(Wrxt+Urht−1+br) zt=σ(Wzxt+Uzht−1+bz) h˜t=tanh(Wxt+U(rt⊙ht−1)+b) ht=(1−zt)⊙ht−1+zt⊙h˜t, where x t the input at time t is h t-1is the previous (hidden) state, and rt and z characterize the reset gate 402 and the update gate 401, respectively. The symbol ⊙ symbolizes element-wise multiplication. The weight matrices W, U, and the bias vectors b are the parameters learned during training.

[0053] GRUs are particularly suited for this task because they can process sequential data by capturing temporal dependencies, making them a suitable choice for encoding time series such as trajectories.

[0054] In this way, the lane node features f1:N / A and the trajectories of the agents fnT coded separately. In the following, the index ta is used for the target agent, and the index sa for the surrounding agents. The lane node features follow the lane direction order, while the agents' trajectories follow a temporal order.

[0055] Fig. 5 illustrates the encoding of agent trajectories by neural networks formed from GRUs (GRU encoders) 501, 502.

[0056] Fig. 6 illustrates the encoding of the lane graph by a neural network formed from GRUs (GRU encoder) 600.

[0057] As in the Fig. 5 and Fig. As shown in Figure 6, the states are updated by the GRU cells and transferred to the subsequent GRU cell during sequence processing. When these states pass through the GRU encoder, the final hidden state contains the spatial information contained in the sequence. This output from each GRU encoder 501, 502, 600 serves as an embedding for the movement of the target agent (h ta ), of the surrounding agent (h sa ) or the roadway junction characteristics (h l ). All these embeddings are represented in a 32-dimensional space.

[0058] Through the encoding of the agent information 201 described above, i.e., the target agent encoding 207 and an all-surrounding-vehicles encoding 208, in particular, all agents surrounding the target agent (i.e., the target vehicle) within the viewed scene are represented. However, not all of these agents influence the target vehicle. When driving, humans rely on a complex interplay of factors to decide which vehicles (or agents in general) in the environment are most important. Factors such as the relative speed of the surrounding agents, their distance from the target vehicle, their trajectories, and traffic regulations determine these decisions. For example, a vehicle directly in front of the target vehicle and traveling at a significantly lower speed may have more influence than a fast vehicle in a non-intersecting parallel lane.Similarly, a vehicle waiting to turn from an intersecting lane may have priority in the target vehicle's decision-making due to traffic laws and the risk associated with the turn.

[0059] According to one embodiment, an agent interaction graph 203 is used as input, derived, for example, from the nuScenes Trajectory Prediction Graph dataset (see Reference 1), which identifies the most important agents in the environment related to the movement of the target vehicle and decision making. The agent interaction graph can also be derived from any other driving scene, provided the map information and agent information are available. It also reveals the semantic relationships and attributes between them.

[0060] Fig. Figure 7 shows a real traffic scene 701 and a corresponding agent interaction graph 702. The agent interaction graph shows the semantic relationships between the agents, which are characterized by edge types: lateral (side by side), longitudinal (one behind the other), crossing, and pedestrian (or "pedestrian crossing"). The relationships between the agents are characterized by these four different edge types. The first three types, longitudinal 703, lateral 704, and crossing 705, are equipped with three edge attributes, which are fedge(s,t)=[distance,path distance,edge probability] The Distance attribute quantifies the Euclidean distance between two agents. Path Distance measures the distance traveled along a specific path, and Edge Probability specifies the probability of a specific relationship. The Pedestrian 706 edge type does not have the Path Distance attribute.

[0061] The following describes how the dynamic heterogeneous graph encoder (DHGE) 206 processes the agent interaction graph 702.

[0062] Fig. Figure 8 illustrates a dynamic heterogeneous graph encoder (DHGE) 800 according to one embodiment. It consists of three parts: a specific-type encoder 801, an EHGT (edge-enhanced heterogeneous graph transformer) 802, and a temporal encoder (GRU-based) 803.

[0063] Different MLPs are used as the specific-type encoder 801 to encode node features of different agent types, since vehicles and pedestrians may exhibit different behavior patterns. Next, the EHGT 802 is used to encode the agent interaction graph 702. Finally, the temporal encoder 803 is used to capture temporal information from the graph encoding across individual timestamps.

[0064] A heterogeneous graph transformer is capable of capturing and representing different types of nodes and relationships within a single graph. Its design allows for the flexible integration of different entities and their relationships, making it particularly suitable for the present application.

[0065] At its core, an HGT (Heterogenous Graph Transformer) works with interaction matrices that model specific elements within a relationship and share parameters for better generalization. Additionally, HGT uses meta-relationships to parameterize weight matrices.

[0066] Fig. 9 illustrates a graph transformer 900.

[0067] The graph transformer 900 receives a heterogeneous subgraph 904 as input and has three components: heterogeneous mutual attention 901, heterogeneous message passing 902, and target-specific aggregation 903.

[0068] In HGT, the attention mechanism differs from the standard GAT (Graph Attention Network). Instead of relying on a single weight matrix, which assumes a homogeneous feature distribution between source and target nodes, HGT introduces mutual attention based on relation triples, which are called<Quellknoten, Kante, Zielknoten> are structured. The target node t is projected onto a query vector, while the source node s is mapped onto a key vector: K(s)i=K_Linearτ(s)i(Hl−1[s]) Q(s)i=Q_Linearτ(t)i(Hl−1[t]), where H (l-1) is the input from the previous layer, and i refers to the specific attention head. Each node type has a unique linear projection.

[0069] The attention head is calculated as described by the following equation: ATT_head(s,e,t)=(Ki(s)Wϕ(e)ATTQi(t)T)⋅μ<τ(s),ϕ(e),τ(t)>d where Wϕ(e)ATT is a matrix specific to edge type 905, intended to capture the semantic relationships Φ(e) between nodes. This matrix allows the model to detect and capture different semantic relationships even between nodes of the same type. The attention mechanism is determined by a dot product operation ⊙. Furthermore, d is an adaptive parameter that specifies the overall importance of each relationship triple, while d represents the dimensionality of the vector.

[0070] Then, h attention heads are concatenated, and the softmax function is applied to determine the final attention weights for each relation triplet, as defined by the following equation: ATTGHT(s,e,t)=Softmaxs∈N(t)(‖i∈[1,h]ATT_headi(s,e,t)) (Att for “attention”)

[0071] The transfer of information from the source node to the destination node is facilitated by heterogeneous message passing 902. This process runs in parallel with the computation of mutual attention. To obtain the i-th message head MSG_head i (s,e,t), the representation of the source node from the previous layer H l-1 [s] is projected through a special type encoder 905 M-Linear. It is then combined with an edge type matrix Wϕ(e)MSG multiplied to integrate the edge dependence: MSG_headi(s,e,t)=M_Linearτ(s)i(Hl−1[s]Wϕ(e)MSG)

[0072] Then, for each pair of nodes, h message nodes are concatenated to form the message MsgHGT(s,e,t)=‖i∈[1,h]MSG_headi(s,e,t) to obtain.

[0073] In the subsequent aggregation phase, the messages from all source nodes s are aggregated to the target node t. The attention vector defined in equation (12) serves as a weight to average the corresponding messages from the source nodes, as shown in equation (14). The result is the updated H̃ (l) [t]: H˜(l)[t]=⊕∀s∈N(t)(AttHGT(s,e,t)⋅MsgHGT(s,e,t))

[0074] After that, H contains (l) [t]comprehensive information about the neighbors of the target node t and the associated relationships. The target node t is then reclassified into its type-specific distribution and supplemented with a residual connection from the previous layer (supplied via a residual connection 906): H(l)[t]=A_Linearτ(t)(σ(H˜(l)))+H(l−1)[t]

[0075] While the Heterogeneous Graph Transformer (HGT), like many other graph operators in its category, is capable of encoding a heterogeneous graph, it has one critical limitation: it is unable to incorporate edge attributes. This means that while it can capture and represent various types of nodes and relationships, it does not utilize the additional information that can be provided by the attributes associated with the edges. Given this limitation, according to various embodiments, the dynamic heterogeneous graph encoder 206 utilizes the edge-enhanced heterogeneous graph transformer (EHGT) 802, which can incorporate edge attributes into the mutual attention 901 and message passing 902, thus providing a more comprehensive graph representation.

[0076] One strategy for incorporating edge features is Edge Concatenate HGT (ECHGT), in which the edge attribute of an edge is directly attached to the edge's source node. Essentially, this method treats the edge attribute as an extension of the source node features within a triple. This inclusion of the edge attribute is initially reflected in the computation of mutual attention, as shown in the following equations. K(s)'i=K_Linearτ(s)i(Hl−1[s]+e_a<s,e,t> i) ATT_headi(s,e,t)=(Ke'i(s)Wϕ(e)ATTQi(t)T).μ<τ(s),ϕ(e),τ(t)>d

[0077] The edge attribute is referred to as e_a or below as E_A (for edge attribute).

[0078] It is also included in the message transmission phase as shown in the following equation. MSG_headi(s,e,t)=M_Linearτ(s)i(Hl−1[s]+e_a<s,e,t> i)Wϕ(e)MGS

[0079] However, this approach has difficulties in scenarios with triples that have multiple relationships.

[0080] According to one embodiment, an alternative EHGT strategy is therefore used (also referred to as EAHGT (edge ​​attribute heterogeneous graph transformer)), in which an edge attribute matrix Wϕ(e)E_A used for both attention calculation (as shown in equation (20) below) and message passing (as shown in equation (21) below). This matrix serves as a representation of the edge attribute information. ATT_headi(s,e,t)=(Ki(s)Wϕ(e)ATTWϕ(e)E_AQi(t)T)⋅μ<τ(s),ϕ(e),τ(t)>d MSG_headi(s,e,t)=M_Linearτ(s)i(Hl−1[s])Wϕ(e)E_AWϕ(e)MSG

[0081] Using the EHGT it is possible to process heterogeneous graphs with edge attributes.

[0082] According to various embodiments, the EHGT is used as a GNN (Graph Neural Network) operator to process the interaction of agents in a heterogeneous graph 203. In practical scenarios, it is typically only the direct neighbor of a target vehicle that significantly influences its behavior. Therefore, according to one embodiment, the aggregation is limited to one hop, as in Fig. 10 shown.

[0083] Fig. Figure 10 illustrates the encoding of an agent interaction graph 1001 by two specific-type encoders, a first encoder 1002 for pedestrians and cyclists and a second encoder 1003 for vehicles, followed by an EHGT 1004. The result is a graph embedding with a target node embedding 1005, and embeddings of surrounding nodes with a directed edge (arrow) to the target node 1006 and of surrounding nodes with a directed edge (arrow) from the target node 1007.

[0084] It should be noted that neighbors to which arrows emanate from the target (surrounding nodes with a directed edge from the target node 1006) also influence the target. Therefore, embeddings of these nodes are included in the surrounding agent node embedding 210.

[0085] It should also be noted that the agent embeddings, as discussed with reference to Fig. 10, of which with reference to the Fig. 4, Fig. 5 and Fig. 6: While the latter represent all agents present in a particular scene, the agent embeddings in the agent interaction information encoder 206 are selected to identify agents that are highly likely to influence the behavior of the target agent.

[0086] After agent information encoding, i.e., heterogeneous graph encoding, we will obtain node embeddings containing both spatial and semantic data for each specific timestamp. However, the temporal linkage or continuity between these timestamp-specific embeddings is still missing. To integrate this temporal dimension, which captures the evolution of traffic scenes over time, a GRU network (i.e., a GRU-based temporal encoder) 803 is employed. This GRU network 803 has the task of encoding the target node embeddings from all observed traffic scenes in a temporal order, resulting in a dynamic target node graph embedding gt (corresponding to the dynamic target agent node embedding 211), as shown in Fig. 8 shown.

[0087] At the same time, the embedding of the surrounding nodes g sfrom the last observed traffic scene at t = 0, especially the nodes from which arrows originate from the target agent. This step ensures that no potential agents influencing the prediction are lost.

[0088] The output of the heterogeneous graph encoder (DHGE) 206, 800 are thus embeddings 804 that contain the environment agent node embedding 210 (or environment agent node embeddings when considered individually) and the dynamic target agent node embedding 211.

[0089] As mentioned above in connection with Fig. As already mentioned in section 2, the information fusion model 212 now fuses the various codings (or “embeddings”) 207-211 with the help of four submodels 213, 214, 215, 216 in order to comprehensively represent the dynamics and interactions in traffic scenes: • Movement coding of the target agent (h t): This coding 207 captures the primary movements and patterns associated with the target agent and allows us to understand its inherent movement dynamics. • Coding of the surrounding agents (h s ): This coding 208 provides information about all non-target agents within a traffic scene. • Coding of lane nodes (h l ): This coding 209 provides information about the structured lanes in the scene. • Dynamic target node graph embedding (g t ): Instead of relying solely on movement patterns, this encoding 210 represents the interactions and relationships of the target agent in a dynamic graph. It provides a deeper, relational understanding of the target's position within the broader scene. • Interactive embedding of surrounding agents (g s): This embedding 211 is similar to embedding the target node in the graph, but for the agents in the environment at the current timestamp. It captures how these agents interact with each other and with the target, creating a complete picture of the dynamics between the agents.

[0090] These codes 207-211 form a multifaceted representation of the scene and enable more comprehensive analysis and more precise predictions.

[0091] Fig. Figure 11 shows a (machine) information fusion model 1100 according to one embodiment. It is an example of the information fusion model 212 and serves to merge the encodings 207-211.

[0092] The output of the information fusion model 212 is f f (for “merged”).

[0093] As in connection with Fig. 2, the information fusion model 1100 comprises four submodels (or sub-modules) 1101-1104, each of which is specifically configured to process different facets of the information.

[0094] These submodels 1101-1104 are described below.

[0095] First submodel 1101 (surrounding agents and their interactions): The coding h s Contains information about all agents in a traffic scene, except for the target agent. To extract meaningful representations that highlight the most influential surrounding agents and understand the significance of their interactions, the first submodel 1001 employs a Cross-Attention (CA) mechanism 1105 as follows: sai=CA(hs,gs)+hs

[0096] The CA mechanism acts between h s (the GRU-embedding representations of the surrounding agents) and g s(the embeddings that capture the interactive relationships between these agents). The goal is to weigh and understand which relationships among the surrounding agents are most important in the given context.

[0097] An important addition to the CA mechanism is the integration of a skip connection 1111 (realized by the addition of h s This addition facilitates the transfer of the original information across the CA layers and ensures that the basic information is not lost during the attention process. Using the skip connection within the model offers two main advantages: - Handling graphs with few edges: In some traffic scenarios, the lack of relationships between agents can be a challenge. In such cases, a direct path in the form of a skip connection helps ensure that the model can still make meaningful predictions without overrelying on nonexistent ones. - Model stability: Skip connections stabilize deep models (i.e., models with many layers). They allow gradients to flow directly backward through the respective neural network, which can prevent the vanishing gradient problem and support model convergence.

[0098] Second submodel 1102 (surrounding agents with interaction and lane information): It implements a (second) cross-attention mechanism 1106 between the surrounding agents with the interaction coding sa i and the lane node representations h land ensures that the embedding of lane nodes is refined based on the presence and behavior of agents in their vicinity. The approach of only considering agents within a certain radius of each lane node ensures that the model remains focused on the most relevant interactions and reduces the influence of distant, likely irrelevant agents.

[0099] Fig. 12 illustrates the CA processing by the second submodel 1102.

[0100] The cross-attention mechanism 1106 obtains keys (keys K) 1202 and values ​​(values ​​V) 1203 from the surrounding agents with the interaction coding sa i . Queries (Q) 1201 are from the lane coding h lThe output 1204 of this attention process is a representation that fuses information about the environment of the lane node with the lane node's own features. Concatenating this output with h l provides a more comprehensive representation of each lane node, now including the context of nearby agents.

[0101] The second submodel 1102 further includes a GAT (Graph Attention Network) encoder (i.e., a GNN (graph neural network) 1107 for lane nodes: Following the cross-attention mechanism 1106, it is ensured that the lane nodes also capture contextual information from their neighboring lane nodes. The use of the GAT encoder 1107 ensures that the embedding for a lane node is updated based on its interactions with other connected lane nodes. This step allows the model to capture more details about how lane configurations and neighborhoods can influence agent behavior.

[0102] The combination of the cross-attention mechanism 1106 and the GAT encoder 1107 results in a final encoding h lfof lane information that is both contextual (in terms of the agents' interactive behavior) and structurally informed (in terms of the lane configurations). Such a dual contextual approach, as defined by the equation hlf=GNN(CA(hl,sai)) can provide more precise insights into how agents might behave in a particular traffic scene.

[0103] Third submodel 1103 (target agent and its dynamic interaction):

[0104] Similar to the first submodel 1101, the third submodel 1102 refines the coding of the target agent ht by combining it with the dynamic target node graph embedding gt according to tai=CA(ht,gt)+ht This ensures that the target agent is aware of the broader context in which it operates, particularly considering its interactions with nearby agents.

[0105] The third submodel 1103 thus contains - a (third) cross-attention mechanism 1108 for target encoding: This is an attention mechanism in which keys and values ​​are derived from gt, which contains the dynamic interaction of the target agent with the surrounding agents. The queries come from h t , which represents the target agent's embedding. This attention mechanism selectively provides the target agent's representation with relevant interactive context, thus creating a hybrid embedding that balances individual behavior with interaction patterns. - a (second) skip connection 1109: It ensures that the original encoding ht is not completely overshadowed by the interaction-informed embedding.

[0106] The output of the third submodel 1103 is a coding t ai(Target agent with interaction), which contains both information about its inherent properties and information about its interactions with other agents.

[0107] Fourth submodel 1104 (target agent with interaction coding and lane end coding): The fourth submodel 1104 contains a (fourth) CA mechanism 1110, which helps to determine the relevance of various features of the lane end coding h lf for the target agent with interaction coding t ai to consider: ff=Conc(tai,CA(tai,hlf))

[0108] Here, "Conc" stands for concatenation. Intuitively, the fourth CA mechanism 1110 allows the model to focus on the most relevant lane-related information while taking into account the interaction of a target agent. To do this, the updated and aggregated lane end coding h lf as keys and values ​​and the target agent-with-interaction coding tai as a query of attention is projected linearly. Following the fourth CA mechanism 1110, t ai concatenated with the CA output. This concatenation ensures that not only the weighted information from the lanes is captured, but also the raw interaction details of the target agents are preserved. Finally, the fused encoding f f which serves as the final encoding provided by the information fusion module.

[0109] The fourth submodel 1104 therefore contains: - a linear projection: The coding of the lane nodes h lf is first subjected to a linear transformation to derive keys and values, and the target agents are provided with the interaction coding t ai are projected as a query. - A CA mechanism 1110 for fusion: The attention mechanism 1110 is then applied. This ensures that the fused coding captures the most important details from both the lane perspective and the target agent perspective. - Concatenation for the final fusion: The output of the attention mechanism 1110, which is a weighted value of the lane node information with respect to the target agent, is multiplied by t ai This provides the refined target agent information with the most relevant lane context. The concatenated representation serves as the final encoding f f generated by the information fusion module 212. This representation contains a holistic and comprehensive understanding of the target agent's situation in the traffic scene and captures its behavior, interactions, and relationship to the lanes.

[0110] The fourth submodel 1104 represents the final process of information fusion. It provides a complete coding that includes both the specific details from the perspective of the target agent as well as its interactions with other people and the paths around it.

[0111] The decoder 217 uses a latent variable z to enable the output of different motion profiles. By concatenating a Gaussian distributed latent variable z with the fused coding f f , the decoder 217 is able to generate different motion profiles to account for inherent uncertainty. Then, an MLP is used to calculate k modes of the future trajectory Y^1:tfκ to spend: Y^1:tfκ=MLP(conc(ff,zκ))

[0112] A K-means clustering is then used and the cluster centers are presented as a final output in the form of K predictions [Y^1:tf1,Y^1:tf2,…,Y^1:tfκ] issued.

[0113] In addition, the embedding gt of the dynamic target node graph is directly converted to a trajectory Y˜1:tfκ and in this way again (in the same way as above) a final output [Y˜1:tf1,Y˜1:tf2,…,Y˜1:tfκ] This ensures that the model can leverage the rich relational information from the heterogeneous graph and does not overlook crucial social interactions when making predictions.

[0114] The decoder 217 is trained with an average winner-takes-all average offset error. In this way, two losses are determined: a fused regression loss Lfr (where only the best mode from the output of the information fusion model is considered (Winner-Take-All)) and a graph regression loss Lgr (the best mode from the embedding gt of the dynamic target node graph): Lfr=mink1tf∑t=1tf‖Y^tκ−Ytgt‖2 Lgr=mink1tf∑t=1tf‖Y^tκ−Ytgt‖2

[0115] The model is then evaluated with a total loss that combines these two losses. L=λ1Lfr+λ2Lgr trained.

[0116] This combined loss, with two scalar weights, λ1 and λ2, ensures that the model does not overrely on any one source of information. The combined loss combines the strengths of both individual losses to force the model to learn from the rich spatial, semantic, and temporal features embedded in the individual losses, ensuring a comprehensive learning process.

[0117] In summary, according to various embodiments, a method is provided as described in Fig. 13 shown.

[0118] Fig. 13 shows a flowchart 1300 illustrating a method for predicting trajectories of road users.

[0119] In 1301, a traffic scene is represented as an agent interaction graph having a node for each road user corresponding to a target vehicle and for one or more other road users, and having a plurality of edges, wherein each edge between two of the nodes (i.e., each edge contained in the agent interaction graph between two nodes representing road users) is assigned a respective edge type indicating a type of movement of the road users represented by the nodes relative to each other on a respective roadway.

[0120] In 1302, the agent interaction graph is processed by a graph transformer to determine embeddings of the target vehicle and one or more other road users, wherein the graph transformer has an attention mechanism (typically also referred to as “attention”) that takes into account the edge types of the edges of the agent interaction graph.

[0121] In 1303, at least one trajectory of the target vehicle (and possibly also trajectories of one or more other road users) is predicted from the embeddings.

[0122] The procedure of Fig.13 may be performed by one or more computers having one or more data processing units. The term "data processing unit" may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be handled according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0123] According to various embodiments, the method is therefore particularly computer-implemented.

[0124] The traffic situation can be detected using sensor data. Various embodiments can receive and use sensor data from various sensors, such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc.

[0125] The predicted trajectories can be used to control an ego vehicle (i.e., taking into account, e.g., planning a trajectory of the ego vehicle such that, if the predicted trajectories are assumed to be correct, no collision should occur). The graph transformer can be part of a larger machine learning model, as described above, which is trained end-to-end, e.g., using example scenarios with future trajectories as ground truth for supervised learning. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] H. Caesar et al., „nuScenes: A multimodal dataset for autonomous driving“, 2020, https: / / arxiv.org / abs / 1903.11027

[0005]

Claims

[1] Method for predicting the trajectories of road users (108), comprising: Representing a traffic scene as an agent interaction graph (203, 1001) which has one node and several edges for each road user corresponding to a target vehicle (108) and for one or more other road users (108), wherein each edge between two of the nodes is assigned a respective edge type which specifies a type of movement of the road users (108) represented by the nodes relative to each other on a respective roadway; Processing the agent interaction graph (203, 1001) by a graph transformer (802, 900, 1004) to determine embeddings of the target vehicle (108) and the one or more other road users (108), wherein the graph transformer has an attention mechanism that takes into account the edge types of the edges of the agent interaction graph (203, 1001); and Predictions of at least one trajectory of the target vehicle (108) from the embeddings. [2] Method according to claim 1, wherein the attention mechanism takes into account the edge types of the edges of the agent interaction graph (203, 1001) by having a set of attention mechanism parameters for each edge type, wherein the sets of attention mechanism parameters are individually trainable. [3] Method according to claim 1 or 2, wherein each of the edges has one or more edge attribute values ​​that specify quantitative properties of the movement of the road users (108) represented by the nodes relative to each other and which the attention mechanism takes into account. [4] Method according to any one of claims 1 to 3, wherein the type of movement is one of side by side, one behind the other and crossing. [5] Method according to one of claims 1 to 4, wherein the trajectories are further determined from at least one of a coding of the movement of the target vehicle (108), a coding for each of the other road users, the movement of the other road user (108) and codings of lane nodes of one or more graphs (202) representing one or more lanes of the traffic scene. [6] Method according to any one of claims 1 to 5, further comprising steering a vehicle (101) taking into account the at least one predicted trajectory. [7] Vehicle control unit, configured to carry out a method according to any one of claims 1 to 6. [8] Computer program with instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6. [9] Computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6.

Images