A method for trajectory prediction of heterogeneous traffic participants

Abstract

A method for predicting the trajectories of heterogeneous traffic participants has been developed. During the graph construction process, this method abandons the fixed-distance-based edge selection mechanism and instead considers more domain knowledge, including endpoint types, interactions, and the relative orientations of targets, enriching the information contained in the graph. Furthermore, factors influencing trajectory prediction, including motion and environmental characteristics, are converted into a common data format and feature type, thereby transforming heterogeneous points and edges into a unified form. Finally, the motion characteristics, interactions between various traffic participants, and their interactions with the road are integrated and fed into a multi-layer perceptron decoder to generate multimodal trajectory predictions. Compared to existing standard graph neural network models, this method significantly improves prediction accuracy, demonstrating accurate and stable prediction capabilities, particularly for complex, densely interactive, multi-target scenarios.

Description

Technical Field

[0001] The present invention relates to the field of automatic control and artificial intelligence technology, and is used for multimodal trajectory prediction of heterogeneous traffic participants (including motor vehicles, motorcycles, bicycles, pedestrians, etc.) in urban dynamic traffic environments, and in particular to a relative orientation relationship-assisted edge feature graph Transformer neural network model. Background Art

[0002] Autonomous driving is a cutting-edge modern technology. To navigate complex traffic environments safely and efficiently, self-driving cars must consider the possible behaviors of other traffic participants (including motor vehicles, bicycles, and pedestrians) based on observed information. Trajectory prediction is an effective tool for predicting the future. However, accurate prediction is challenging due to the complex and changing nature of urban traffic environments and the interactions between traffic participants.

[0003] Vehicle behavior is primarily influenced by three factors. The first is the vehicle's driving style, such as conservative or aggressive. This characteristic can be captured by observing and recording its historical trajectory, then encoding it using models such as recurrent neural networks (RNNs) or one-dimensional convolutional neural networks (CNNs). The second is the interaction between the vehicle and traffic infrastructure (including lane markings, crosswalks, and traffic signs, referred to as lanes below). This environment regulates and restricts vehicle behavior. The third is the interaction between vehicles, such as the need to determine the order in which vehicles passing through intersections and ramps converge.

[0004] While traditional physics-based models already account for the first factor, motion, the impact of the second two "interaction" factors still requires further exploration and utilization. For example, vehicles must always stay within road boundaries, brake and stop at red lights, and yield to pedestrians. These constraints and interactions between vehicles and traffic facilities, or between vehicles and other traffic participants, can be understood as domain knowledge related to "agent-to-road" and "agent-to-agent" interactions.

[0005] Considering the constraints of the above factors, most current methods involving trajectory prediction have the following limitations in handling interactions and different types of input features:

[0006] 1. Only using distance or coordinate difference to characterize the degree of attention fails to fully utilize the domain knowledge in the driving scene to explicitly consider the interaction between participants and between participants and the environment.

[0007] Generally speaking, using a "graph structure" to model traffic scenarios is more appropriate. In the graph, traffic participants are considered "nodes" and the connections between them are represented by "edges." That is, if the distance between two nodes is less than a fixed threshold, they are considered to be connected, and the corresponding element in the adjacency matrix is marked as 1. However, this graph construction method is not suitable for some complex scenarios:

[0008] First, the distance-based graph construction method assumes that the edge from node A to node B and the edge from node B to A are equal, that is, symmetrical, but Figure 1 In the scene of sub-figure a), the main vehicle marked with the letter E is about to merge into the main road. At this time, its attention to vehicle A is extremely high. However, since vehicle A has already passed the intersection, its attention to the main vehicle is relatively low ( Figure 1 In the left-hand scene, the host vehicle is merging onto the main road. Its primary interactions are with Car A in the target lane ahead and Car B approaching from behind. While the interaction between the host vehicle and Car B is mutual and nearly equivalent, the interaction between the host vehicle and Car A is more one-way: the host vehicle pays attention to Car A, but Car A rarely pays attention to the host vehicle. This demonstrates that the interactions between vehicles are not symmetrical.

[0009] Second, it is not enough to use distance alone to symbolize the degree of attention, e.g. Figure 1 In the scenario of sub-graph b), although the main vehicle E is closer to vehicles A and B, the main vehicle focuses more on vehicle C because of the risk of collision between them ( Figure 1 In the right-hand scene, the main car, which is about to complete a left turn, is competing for the right of way with car C coming from the right. Therefore, its attention is mainly distributed on car C, rather than on car A or car B, which are closer in space. This shows that the distribution of attention is not entirely determined by distance, that is, distance is not the only factor affecting the distribution of attention.

[0010] Considering the above two points, it is obviously more appropriate to use "directed graph" instead of "undirected graph" to describe traffic scenes. Although some methods have adopted directed graphs, they ignore the information of the edges in the graph, such as the different properties of the nodes connected by the edges, the relative orientation relationship represented by the edges, etc.

[0011] 2. Failure to uniformly manage the features of different data types and directly and violently fusing data will increase the difficulty of the model in extracting effective information.

[0012] There are three common methods for fusing different types of input data, mainly targeting static map features and dynamic trajectory features:

[0013] The most common and simple method is to directly arrange different features according to a certain dimension to form a new high-dimensional feature. However, this brute force operation method will bring difficulties to the downstream decoding model to learn features, because the meaning and measurement of the input data may not be consistent.

[0014] The second approach is to use the attention mechanism to fuse different data. Specifically, assuming there are two types of data, source features and target features, and the goal is to fuse the source features into the target features, the importance of each source feature to the target feature is calculated, and a weight is obtained. The weights are then combined to perform information aggregation operations (generally including summing, averaging, and maximizing). Although attention-based methods can ensure that the dimensionality of the fused features remains unchanged, they suffer from irrational arithmetic operations on features of different scales.

[0015] The third method is to use a neural network with a "gate" structure to select source features that are useful for target features, which is essentially a simple form of attention mechanism.

[0016] In addition, many models only consider the prediction of a single target participant, and the prediction process needs to be executed multiple times in a multi-target environment, which is inefficient; most models only consider homogeneous traffic participants (vehicles or pedestrians) in the scene, and cannot simultaneously predict the trajectories of motor vehicles, bicycles, pedestrians, etc. in the scene, making the model unsuitable for complex environments where heterogeneous targets coexist, such as intersections. Summary of the Invention

[0017] In view of this, this paper mainly focuses on and solves two important problems in trajectory prediction: one is how to make full use of prior domain knowledge to provide more effective and unique feature information for the model; the other is to find a universal feature representation form that can unify and fuse different types of unstructured features, thereby improving the network model's absorption and learning efficiency of input information.

[0018] Compared with other trajectory prediction networks, this paper innovates and improves the model in the following two aspects:

[0019] First, in order to model the "interaction behavior" in traffic scenarios, a graph structure considering relative orientation relationships is constructed, which contains rich edge feature information.

[0020] The nodes of a graph are traffic participants or road elements (defined as road segments of a certain length or traffic signs at a certain location). The edges of a graph represent the connections between nodes. This edge construction abandons the fixed-distance-based edge selection mechanism and considers more "domain knowledge" (endpoint types, interactions, and the relative positions of targets), enriching the content of the graph.

[0021] At the same time, the edge features of the graph can be further effectively utilized by inputting their features into the classification network that determines the closeness of the interaction, helping the model to focus on effective information more quickly.

[0022] Second, the factors that affect trajectory prediction results, motion characteristics and environmental characteristics, are converted into the same data format and feature type. Specifically:

[0023] For different point structures, whether they are road elements or moving objects, their types will be encoded to distinguish their identities;

[0024] At the same time, for the edge structure, in order to mark different types of edges and their unique meanings, traffic rules are referred to and they are divided into relative orientation relationships of "front, back, left, right, and conflict relationship" and encoded. Among them, front, back, left, and right correspond to their corresponding orientations, and the conflict relationship indicates that there may be overlapping positions in the future at a certain angle (for example, vehicles from different directions in the intersection, or vehicles converging on the ramp).

[0025] Through these encoding operations, heterogeneous points and edges are transformed into a unified form. This has the advantage of unifying the measurement and meaning of features, effectively performing feature fusion. Ultimately, the motion characteristics of various traffic participants, their interactions with each other, and their interactions with the road are integrated and transmitted to the multi-layer perceptron decoder to generate a multimodal trajectory prediction.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) A graph structure that considers relative orientation relationships is constructed. Compared with the edge selection mechanism based on fixed distance, the edge construction considers more "domain knowledge" (endpoint type, interaction relationship, and relative orientation relationship of the target), enriching the feature content contained in the graph;

[0028] (2) By converting the factors that affect trajectory prediction results: motion characteristics and environmental characteristics, into the same data form and feature type;

[0029] (3) Ability to predict the trajectories of various types of traffic participants;

[0030] (4) Compared with the current common graph neural network model, the proposed model has greatly improved the prediction accuracy;

[0031] (5) Especially for multi-target scenarios in complex and intensive interaction scenarios, the model shows accurate and stable prediction capabilities;

[0032] (6) The proposed method was tested and verified on two high-quality and high-difficulty public datasets in the field. The experimental results demonstrated the effectiveness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0034] Figure 1 Schematic diagram of vehicle attention in different traffic scenarios (shading represents attention), where the vehicle marked with the letter E represents the main vehicle, vehicles A, B, and C are other social vehicles distributed around the main vehicle, and the shading represents the attention distribution of the main vehicle.

[0035] Figure 2 A schematic diagram illustrating the idea of constructing the traffic map data structure disclosed in the present invention.

[0036] Figure 3 The overall model of multimodal trajectory prediction for heterogeneous targets disclosed in this paper is presented.

[0037] Figure 4 It is the basic cell unit of layer normalization GRU.

[0038] Figure 5 Shown is a normal graph Transformer network (left) and a graph Transformer network with edge features (left).

[0039] Figure 6 Prediction results of sparse traffic scenarios under different road structures.

[0040] Figure 7 The subfigures in the figure show the prediction results for different straight-line scenarios, where lane boundaries are drawn with solid gray lines, lane centerlines are light gray dashed lines, and vehicles are represented by rectangles. The vehicle's actual trajectory over the past 1 second and the next 3 seconds is marked with a dashed line, and the predicted multimodal trajectory (a total of 6 modes) is represented by a dashed line.

[0041] Figure 8 Figure 2 shows the trajectory prediction results for an intersection scenario. Vehicles are represented by rectangles, true trajectories are represented by solid lines, and predicted trajectories are represented by dashed lines.

[0042] Figure 9 The trajectory prediction results for the roundabout scenario are shown in Figure 2. Vehicles are represented by rectangles, true trajectories are represented by solid lines, and predicted trajectories are represented by dashed lines.

[0043] Figure 10 Prediction results for different scenarios under mixed road structures.

[0044] Figure 11 This is the prediction result for sparse scenarios with heterogeneous traffic participants. Motor vehicles are represented by rectangles, motorcycles and bicycles are represented by rectangles with a uniform length and width of 1 / 2 of the motor vehicle, and pedestrians are represented by square boxes.

[0045] Figure 12 Prediction results for dense interaction scenarios with heterogeneous traffic participants. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] The present disclosure provides a method for trajectory prediction based on heterogeneous traffic participants. This method is based on the further enrichment of edge features in the graph structure and can realize multimodal trajectory prediction of heterogeneous traffic participants (including motor vehicles, motorcycles, bicycles, pedestrians, etc.) in urban dynamic traffic environments.

[0048] An exemplary implementation includes the following steps:

[0049] Step 1: Variable Settings

[0050] The task of trajectory prediction of traffic participants can be transformed into such a problem: based on certain prior domain knowledge D, environmental information M, and historical trajectory information of vehicles Build a model f(θ) (θ is the model parameter) to map the input data to the future trajectory As shown in formula (1).

[0051]

[0052]

[0053] Among them, N0 is the number of target traffic participants in the scene at the last historical moment, T H is the duration of historical observation, T F is the future prediction duration. Assume that the set of all target persons is A0, for each ai ∈A0, i∈N0 (the serial number of the main vehicle is superscript O, and the serial numbers of other vehicles range from 1 to (N0-1).), we can get its historical trajectory sequence Where (x, y) are the x and y coordinates in the Cartesian coordinate system, v x , v y are the speeds in the x and y directions respectively, θ is the vehicle head angle, and other physical quantities such as object length, width, 3D bounding box, type, etc. can also be included. The output is the future trajectory of N0 targets in a total of K modes, each trajectory is of length T F A sequence of (x, y) coordinate points.

[0054] Static environmental features include road boundaries, lane centerlines, sidewalks, traffic signs, etc. Assume that the set of environmental elements M0 contains N m The basic unit of lane is m. j For example, a road segment is a lane divided into lanes of a certain length, each 10m long (it can also be based on the average speed of the target in the scene), which includes equally spaced waypoints (1m apart), such as Figure 2 As shown. Using vectorized expression, each road segment is represented as a starting point (first waypoint (x start ,y start )) points to the end point (the last waypoint (x end ,y end )) The center point is a middle waypoint (x c ,y c ) vector, that is, for m j ∈M0,j∈N m , its corresponding features Thus, the motion features and map features can be converted into a unified form. For other elements in the scene, missing features will be filled with 0.

[0055] Step 2: Construction of edge feature enhanced graph

[0056] To model the interactions between prediction targets and between prediction targets and environmental elements, this paper constructs a graph structure G = <V, E>, consisting of nodes V and directed edges E. Unlike ordinary graphs, the edges in graph G no longer simply indicate whether two points are associated (the Boolean value 0 represents no association and 1 represents association), but instead include characteristics describing the point type and the interaction between points.

[0057] Specifically, for any node μ i , μ∈{a, m}, whose features include motion state and node type, i.e. (x, y, v x , v y ,θ,bv，type ), where b v，type is the binary encoding of the node type, as shown in Table 1. Graph edges also carry important information, such as the interactions between nodes. Existing methods only consider the relative coordinates of nodes as edge features. In addition, this disclosure proposes the concept of "relative orientation relationships," as shown in Table 1, including relationships such as preceding, following, left, right, and conflicting edges.

[0058] The reason for adopting this design is: on the one hand, compared with simple distance, this traffic direction relationship can more accurately reflect the semantic relationship between two nodes, forming a directed graph; on the other hand, different relationships imply the traffic rules that need to be followed, such as the rear vehicle and the front vehicle keep a certain distance, the left / right vehicle may change lanes, and the conflicting vehicles need to negotiate the right of way, etc. The graph includes the features of points and edges, as shown in formula (2). μ Contains motion features and category coding features, edge E μ Including the relative coordinate difference of the two center points and the relative orientation relationship coding feature b rpr and edge type b e，type (different endpoint type combinations).

[0059]

[0060] Then, the input data is normalized. In this paper, the coordinate system transformation method called agent-centric is adopted, that is, the position of the last frame of historical data of the main vehicle is used. As the coordinate origin, the vehicle head is facing The x-axis is used as the positive direction of the coordinate system to construct the main vehicle body coordinate system. Assuming that the original global coordinates of a certain position are [x; y], the transformed coordinates are [x′; y′]. The transformation of other physical quantities such as velocity and azimuth is shown in equation (3).

[0061]

[0062] Table 1 Binary encoding of node and edge types

[0063]

[0064]

[0065] Figure 2 This is a schematic diagram of the traffic map data structure construction process disclosed in this disclosure, which unifies the characteristics of traffic participants (vehicles, motorcycles, bicycles, pedestrians, etc.) and map elements (lane segments, sidewalk segments, traffic signs, etc.). The traffic map structure contains two types of data features: points and edges, where:

[0066] Points can be traffic participants or road elements, which contain two types of features: motion state features and node type encoding features. Motion state features include position points: (x, y), speed in the x and y directions (v x , v y ) (the distance between the two endpoints of the displacement vector per unit time) and the vehicle head deviation angle θ. The motion features are converted into a coordinate system with the origin at the last historical moment of the main vehicle. To ensure the consistency of the data order of magnitude, the node type or attribute is encoded in binary format (0 or 1). Considering the possible node types in the scene, a three-digit code length (000-111) is used to encode traffic participants and map elements respectively (supporting 8 types of traffic participants and 8 types of map elements, and more categories can increase the code length).

[0067] The characteristics of an edge include the relative coordinates of the two endpoints (Δx ij , Δy ij ), the code of the relative position relationship, a total of three digits (000-111) and the code of the endpoint type combination (000-111).

[0068] The right image shows the relative positions of vehicles (traffic participants) in a traffic scene, and the left image shows the relative positions of lane segments. Note that traffic participants and maps share the same graph structure and data type, but each has its own independent graph.

[0069] Step 3: Feature Encoding

[0070] According to the overall trajectory prediction model disclosed in the present invention, Figure 3 As shown, it includes three parts: feature encoding, feature fusion and feature decoding.

[0071] First, the historical trajectory features are sent to the layer-normalized GRU (Gated Recurrent Unit) encoder to obtain the dynamic features O dyn , the graph features will be input into the edge-enhanced graph Transformer network (learned) to obtain the O-graph describing the interaction features between traffic participants. a，inter and O that describes the interactive features of environmental map elements m，inter ; (That is, provide these basic information, and learn their interactive relationships through the network, such as providing their relative positions and relative orientations, and the network learns who may occupy the dominant position on the road;) Then, the map feature O m，inter Meeting trajectory characteristics O dyn (or interactive feature O a，inter ) for fusion;

[0072] Finally, describe the characteristics of the movement O dyn , describing the characteristics of the interaction between traffic participants and characteristics describing the interaction between traffic participants and the map They will be connected in the same dimension and sent to the multi-layer perceptron decoder (Laplacian decoder) to obtain the multimodal future prediction trajectory of each individual.

[0073] 3.1 Layer-normalized GRU Encoder

[0074] Gated Recurrent Unit (GRU) is a variant of Recurrent Neural Network (RNN), such as Figure 4 It is good at filtering and memorizing useful features from serialized input data and has fewer parameters than the Long-Short Term Memory (LSTM).

[0075] The GRU consists of an "update gate" and a "reset gate" structure, and its forward information transmission process is shown in equation (4-1), where σ represents the sigmoid activation function and tanh is the tangent activation function. However, due to the uneven distribution of input data, the gradient of the GRU network is unstable and converges relatively slowly. In contrast, the layer-normalized (LN) GRU model normalizes the input data according to the dimension of the "layer", converting it into independent and identically distributed input data, making the model invariant to data rotation and scaling in different training scenarios, as shown in equation (4-2).

[0076] Figure 4 The ordinary GRU unit and LN-GRU unit are shown. The input is: the input feature x at time t t , the hidden state h at time t-1 t-1 , the output is the output feature y at time t t Compared with ordinary GRU neurons, the difference is that the input feature x t and hidden state h t-1 Layer normalization is performed to unify the data magnitude of the input data, eliminating the problems of slow network learning speed and unstable convergence caused by differences in the distribution of input data.

[0077]

[0078]

[0079] Assume that the historical trajectory input of multiple prediction targets is Where B is the batch size of each training, F is the dimension of the dynamic feature; the GRU encoder will output the encoding tensor of the motion feature and the hidden state feature tensor

[0080] 3.2 Edge Feature Enhanced Graph Transformer Encoding Network

[0081] The Transformer network was first proposed and applied to the field of natural language processing. It abandons the use of chained recurrent neural network structures to process sequential data, and instead adopts an attention mechanism to extract temporal or spatial features. The Transformer encoder contains three key information components: query, key, and value. The calculation process is as follows: first, the attention weight (the degree of correlation between different pieces of information) is calculated based on the degree of match between the query and key, and then the attention weight is fused with the value (generally by summing the feature data). There are generally two types of attention modes: self-attention and cross-attention. The difference is that the query and key of self-attention come from the same data, while the query and key of cross-attention come from different data. Based on this mechanism, the graph transformer can accept graph-structured input data and also use the attention mechanism to extract features.

[0082] However, the Graph Transformer assumes that the graph is homogeneous, meaning that all nodes have the same attributes. It also fails to consider the characteristics of edges. The edge-feature-enhanced Graph Transformer is an improvement on this. It was initially applied to the classification of the molecular structures of chemical substances, treating each atom as a node in the graph and the chemical bonds between atoms as edges. Because the type of chemical bond (the way atoms form bonds) has a significant impact on the chemical properties of a substance, the characteristics of edges in the graph have also attracted considerable attention. These edge characteristics can be considered domain knowledge. Similarly, in traffic scenarios, the type of interaction between participants will also have a significant impact on future developments.

[0083] In the edge-feature-enhanced graph network, the update method of nodes and edges is shown in Equation (5), where l is the lth layer in the stack of multiple graph Transformer layers, and i represents the i-th node from A or M.

[0084]

[0085] H is the number of "attention heads", and || represents the operation of merging according to a certain dimension. After the input graph G passes through the Transformer network, it will output a point encoding tensor representing the interaction between target individuals. and the edge encoding tensor and point-coded tensors representing interactions between map elements and the edge encoding tensor

[0086] Figure 5 The figure shows a normal graph Transformer network (left) and a graph Transformer network with edge features (left). Compared to normal graph networks, graph networks with edge features can better capture the characteristics of edges between nodes. Their inputs are point features and edge features (edges only exist between nodes with interactive relationships. Edges can carry interaction features, node combination information, etc., and they also participate in the fusion and integration of information. The red connecting lines in the right figure show the coupling and decoupling of edge information and point information.) The outputs are the encoded point features and edge features. In order to better understand the interaction pattern of vehicles, the edge interaction features are further utilized and input into the linear fully connected layer to obtain the interaction category prediction results (close interaction / non-close interaction).

[0087] 3.2 Converged Network

[0088] The dynamic feature coding of the vehicle is obtained Interactive features with maps Finally, the present invention uses the “attention mechanism” to fuse the two features to form the feature of “the degree of attention paid by the vehicle to each road section in the map”. Specifically, for each dynamic feature x of target i, i ∈O dyn , and the characteristics of each map j The fused feature tensor is shown in formula (6):

[0089] y i =x i W0+∑ j φ(concat(x i , Δ i，j , x j )W1)W2,y i ∈O fuse (6)

[0090] Where W is the weight matrix, φ is the combination of layer normalization and ReLU activation function, Δ i，j =MLP(c j -c i ), that is, the multi-layer perceptron mapping result of the coordinate difference between target j and i. The obtained fusion feature

[0091] At the same time, the interactive information of participants Combined with motion feature information Odyn It will be input into the classification network based on the multi-layer perceptron, and output two types of close interaction / non-close interaction, generating loss Specifically, for any edge (i, j), the corresponding motion feature x i ∈O dyn , x j ∈O dyn and interactive features Its connection features will be input into the classification network f close In the example, the softmax network outputs Y close There are two classification results: (i, j) = 0 and 1 (non-close interaction, close interaction), as shown in formula (7).

[0092]

[0093] 3.4 Decoding Network

[0094] After obtaining the motion feature O dyn , interaction characteristics between traffic participants and map fusion features O fuse Afterwards, the three types of features are connected in parallel in the same dimension and then passed through the decoder network f decoder The predicted result is shown in formula (8), where || represents parallel operation.

[0095]

[0096] Y output =f decoder (O input , H dyn ) (8)

[0097] The decoder is essentially a Laplacian Mixture Density Network, which takes the parallel features as input, the hidden state of the motion encoder as the initial hidden state value, and outputs a set of predicted trajectory distributions with different modes. π k represents the weight value corresponding to each mode, μ and b represent the mean (position coordinate) and scale parameter of the Laplace distribution corresponding to each mode, respectively. The specific operation is shown in formula (9).

[0098]

[0099] Where repeat(·, K) means copying the input variable K times to obtain the initial values of K modes. The output trajectory of each dynamic target is T in the future. Ftime, the number of modes is K, and it contains two-dimensional coordinate points (x, y).

[0100] 3.5 Loss Function and Training Process

[0101] The model is trained using end-to-end learning, and the loss function As shown in formula (10), it consists of three parts, namely, Represents the ranking loss of multimodal scoring and the loss representing the classification of edge interaction types Among them, α=1.0,β=0.1 are coefficients. The regression loss adopts smooth L1 loss; the scoring loss adopts “maximum margin loss”, k * is the modal number corresponding to the minimum trajectory endpoint error, where ∈ is a very small positive number to prevent division by zero; the classification loss uses CrossEntropyLoss, that is, cross entropy loss.

[0102]

[0103] The hyperparameter settings of the model are shown in Table 2. The important parameters are: the number of epochs (the process of completing one training on all data) is 50, the training batch size B is set to 32, and the initial learning rate is 0.001, which is reduced to 50% of the current value after every 15 epochs.

[0104] Table 2 Hyperparameter settings

[0105]

[0106] Application examples:

[0107] To validate the proposed method, experiments were conducted using the INTERACTION dataset, a challenging dataset that collects driving data from various traffic interaction scenarios across different countries. To verify the method's trajectory prediction capabilities for heterogeneous traffic participants, experiments were conducted using the Argoverse2 dataset, a large, publicly available dataset of real-world scenarios. This dataset includes 250,000 non-overlapping scenes mined from six unique urban driving environments in the United States, encompassing a variety of traffic participant types, including motor vehicles, motorcycles, bicycles, and pedestrians.

[0108] The INTERACTION dataset consists of 10 frames of historical observation trajectories over a period of 1 second at a frequency of 10 Hz. The predicted trajectories are 30 frames over a period of 3 seconds at a frequency of 10 Hz. A vehicle in the scene is randomly selected as the primary vehicle, and all other vehicles within a radius of 100 meters from the primary vehicle are defined as target vehicles. The primary vehicle and target vehicles together constitute the prediction object.

[0109] Argoverse2: Historical observations are 5 seconds long at a 10Hz frequency, for a total of 50 frames; future predictions are 6 seconds long at a 10Hz frequency, for a total of 60 frames. The data is divided into the main "focal agent" object (which can be any type of traffic agent), the "scored agent" traffic agent with good observation quality, and the "unscored agent" traffic agent with poor observation quality (containing a large number of missing keyframes). The "focal agent" and "unscored agent" traffic agents are used as prediction objects.

[0110] Figure 6-10 The test results of multi-modal trajectory prediction for multiple targets using the method of the present invention on the INTERACTION dataset are shown. The present invention mainly focuses on complex scenarios with high density, high dynamics and strong interactions in urban traffic environments, solves interactive relationships such as cooperation, competition and negotiation between vehicles, and generates future trajectory predictions that conform to the vehicle kinematic model and multimodal characteristics, are constrained by the lane environment, and meet the interaction constraints. The specific test environment includes various types of sparse traffic density scenes, straight road scenes, intersection scenes, roundabout scenes, and scenes where the above road structures coexist, with motor vehicles (homogeneous traffic participants) and a mixture of multiple types (heterogeneous) traffic participants, as shown in Table 3.

[0111] Table 3 Test scenario classification

[0112]

[0113] Figure 6 This demonstrates that the model of the present invention is applicable to sparse traffic scenarios with different road structures. Besides being able to handle complex scenarios, it can also provide accurate and reasonable trajectory predictions for simple and sparse scenarios.

[0114] Figure 7 The sub-figures in the figure show the prediction results of different straight road scenarios, where lane boundaries are drawn with gray solid lines, lane centerlines are light gray dashed lines, and vehicles are represented by rectangles. The actual trajectory of the vehicle in the past 1s and the next 3s is marked with the real trajectory, and the predicted multi-modal trajectory (a total of 6 modes) is represented by the dashed line. By observing Figure 7 , we can draw the following conclusions:

[0115] 1) The proposed model is applicable to scenarios of varying density on a straight road, including those at normal speeds and extremely congested (parked or braking) scenarios, and can predict future trajectories at different speed ranges, including low speeds (sub-figures c and d) and high speeds (sub-figure a).

[0116] 2) The proposed model can predict different vehicle behaviors on straight roads, such as the following behavior in sub-figures a)-d) and the lane-changing behavior of the vehicle indicated by the circle in sub-figure a);

[0117] 3) The model predicts all target vehicles at once and generates compatible future trajectories (without collisions or other trajectory conflicts).

[0118] Figure 8 The figure shows the trajectory prediction results in the intersection scene. Vehicles are represented by rectangles, the actual trajectory is represented by solid lines, and the predicted trajectory is represented by dotted lines. Figure 8 , we can draw the following conclusions:

[0119] 1) The proposed model is applicable to intersections of various geometric shapes and can learn environmental features such as lane shapes and predict trajectories that conform to road boundary constraints;

[0120] 2) The model can learn different intentions and behaviors of vehicles at intersections and predict future trajectories that match the driving intention, such as slowing down and stopping (vehicle in the circle (d), right turning (vehicle in the circle (a), left turning (vehicle in the circle (c), vehicle going straight (a)-d), etc.);

[0121] 3) The model can learn the interactive behaviors between vehicles, generate “give way / take over” game outcomes, and predict future trajectories without conflicts, such as the vehicle in the box (d).

[0122] Figure 9 The trajectory prediction results of the roundabout scene are shown in Figure 2. Vehicles are represented by rectangles, the actual trajectory is represented by solid lines, and the predicted trajectory is represented by dotted lines. Figure 9 , we can draw the following conclusions:

[0123] 1) The proposed model is applicable to roundabouts of various geometric shapes and can learn environmental features such as lane shapes and predict trajectories that conform to road boundary constraints;

[0124] 2) The model can predict multimodal trajectories based on the characteristics of the roundabout environment. For example, the "umbrella-shaped" trajectory prediction results in each subgraph represent different possible behaviors of the vehicle, such as continuing to maintain its lane within the roundabout, changing lanes, or exiting the roundabout.

[0125] 3) The model can learn different intentions and behaviors of vehicles on roundabouts, thereby predicting future trajectories that match the driving intention, such as slowing down and stopping (a) vehicles in the circle, turning right (a) vehicles in the square, turning left (b) vehicles in the circle, and going straight.

[0126] 4) The model can learn the interactive behaviors between vehicles, generate “give way / take over” game outcomes, and predict future trajectories without conflicts, such as the vehicles in the box in Figure (b).

[0127] Figure 10 The figure shows a complex scene composed of different road structures. In the scene with mixed road structures, the model still maintains the advantages of the single road structure scene.

[0128] Figure 11-12 The results of multimodal trajectory prediction for multiple targets using the proposed method are presented on the Argoverse2 dataset. The main feature of this method is that the scene contains a variety of heterogeneous traffic participants, such as motor vehicles (cars, buses), motorcycles, and pedestrians, which increases the difficulty of prediction in terms of their motion characteristics.

[0129] Figure 11 、 12 This shows a scene with multiple types of heterogeneous traffic participants. Motor vehicles are represented by rectangles, motorcycles and bicycles are represented by rectangles with a uniform length and width of 1 / 2 of the length of the motor vehicle, and pedestrians are represented by squares.

[0130] Figure 11 The interaction between the various traffic participants in the image is relatively simple. The following conclusions can be drawn from the two subgraphs:

[0131] 1) The model can capture and distinguish the motion characteristics of different types of traffic users and predict trajectories that conform to their motion characteristics;

[0132] 2) The model can better identify environmental features and predict trajectories that meet road geometry constraints;

[0133] 3) Ability to output future trajectories of different modes, such as going straight, turning left, turning right, etc.

[0134] Figure 12 The interactions between the agents are complex. Figure a) shows a vehicle in the rightmost lane (circled) changing lanes; Figure b) shows a pedestrian (arrowed) crossing the road at an intersection. The following conclusions can be drawn:

[0135] 1) The model can predict lane-changing behavior and accurately output the corresponding trajectory based on historical information and road characteristics;

[0136] 2) The model can capture and distinguish the motion characteristics of different types of traffic users, and also has certain recognition capabilities for special scenes (crosswalks).

[0137] The above technical solutions are only exemplary embodiments of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and are not limited to the methods described in the above specific embodiments of the present invention. Therefore, the methods described above are only preferred and do not have a restrictive meaning.

Claims

1. A method for predicting trajectories of heterogeneous traffic participants, comprising the following steps: S1, using the same graph structure, constructs graphs with nodes as traffic participants and nodes as map road elements. The edge features in the graphs include: relative coordinates of endpoints, relative orientation relationship, and endpoint type; S2, transforms the features of the traffic participant graph and the map road element graph into a unified form; S3, input the historical trajectory features of each traffic participant into the recurrent neural network encoder to obtain the dynamic feature code O dyn ; S4, the graph basic features of traffic participants and map road elements are respectively input into the edge feature enhanced graph Transformer network to obtain the point feature encoding describing the interaction between traffic participants Point feature encoding that interacts with map elements S5, feature fusion: map features and trajectory characteristics O dyn Fusion, forming features that represent the impact of roads on driving behavior O fuse ; or, map features Characteristics of traffic participants Fusion, forming features that represent the impact of roads on traffic participants O fuse ; S6, the motion feature O dyn , interaction characteristics between traffic participants and map fusion features O fuse Connect them in parallel on the same dimension and send them to the multi-layer perceptron decoder to obtain the multimodal future prediction trajectory of each individual; The step S2 specifically includes: The node features in the traffic participant graph include: motion state features and node type features. The node features in the map road element graph include: static features and node type features. The node type features of both are encoded using binary numbers with the same number of bits; Set the motion state characteristics of the traffic map node, including: location point (x, y); speed in the x and y directions (v x ,v y ), i.e., the distance between the two endpoints of the displacement vector per unit time; and the vehicle head deviation angle θ; all motion features are converted into a coordinate system with the last historical moment of the main vehicle as the origin; Set the static characteristics of the map road element graph node: Use vectorized expression to represent each road segment as a starting point, that is, the first waypoint (x start ,y start ) points to the end point, which is the last waypoint (x end ,y end ), the center point is a middle waypoint (x c ,y c ) vector, that is, for m j ∈M0,j∈N m , M0 is the map road element collection, N m is the number of elements: its corresponding characteristics In both graphs, the relative orientation relationship and endpoint type in the edge features are also encoded using binary numbers with the same number of bits.

2. The method according to claim 1, characterized in that The relative orientation relationship in step S1 includes: front, back, left, right, and conflict relationship; wherein, the front, back, left, and right correspond to their respective orientations, and the conflict relationship indicates that there may be a situation in which positions overlap in the future at a certain angle.

3. The method according to claim 1 or 2, characterized in that In the step S1: Traffic participants include one or more of motor vehicles, motorcycles, bicycles, and pedestrians; Map road elements include one or more of road boundary lines, lane center lines, sidewalks, and traffic signs.

4. The method according to claim 1, wherein The recurrent neural network encoder in step S3 adopts a layer normalized GRU encoder.

5. The method according to claim 1, wherein The method of step S4 is: inputting the graph containing the node V and the directed edge E into the edge feature enhanced graph Transformer network for feature extraction.

6. The method according to claim 1, characterized in that In step S5, the attention mechanism is used to fuse the two features; Among them, map features and trajectory characteristics O dyn Fusion can be expressed as: For each dynamic feature x of target i i ∈O dyn , and the characteristics of each map element j The fused feature tensor is shown in formula (6): and i =x i W0+∑ j φ(concat(x i ,Δ i,j ,x j )w1)w2,y i ∈O fuse (6) Where W is the weight matrix, φ is the combination of layer normalization and ReLU activation function, Δ i,j =MLP(c j -c i ), that is, the multi-layer perceptron mapping result of the coordinate point difference between target j and i; the obtained fusion feature 7. The method according to claim 1, characterized in that The step S5 further includes: The interactive information of traffic participants Combined with motion feature information O dyn Input into the multi-layer perceptron-based classification network for close interaction / non-close interaction classification, including: For any edge (i, j), the corresponding motion feature x i ∈O dyn ,x j ∈O dyn and interactive features These connection features are input into the classification network f close In the example, the softmax network outputs Y close (i, j) = 0, 1, respectively representing the classification results of non-close interaction and close interaction.

8. The method according to claim 1, characterized in that The multi-layer perceptron decoder in step S6 adopts a Laplace mixture density neural network, which takes the parallel features as input, uses the hidden state of the motion encoder as the initial hidden state value, and outputs a set of predicted trajectory distributions with different modes: π k represents the weight value corresponding to each mode, μ and b represent the mean value of the Laplace distribution corresponding to each mode, i.e., the position coordinate and scale parameter, respectively.

9. The method according to claim 7, characterized in that The method also includes the step of training the network model in steps S3-S6 using an end-to-end learning method, wherein the loss function As shown in formula (10): in, represents trajectory regression loss; represents the ranking loss for multimodal scoring; Represents the loss of edge interaction type classification, that is, the loss of the interaction information of participants Combined with motion feature information O dyn Input the classification network based on the multi-layer perceptron and output the classification loss of close interaction / non-close interaction; α and β are weight coefficients; Regression loss Use smooth L1 loss, score loss "Maximize Margin Loss" is used, and the classification loss uses CrossEntropyLoss, that is, cross entropy loss.

Images