A vehicle trajectory prediction method, device, system and storage medium

Abstract

The application discloses a vehicle trajectory prediction method, device and system and a storage medium, and belongs to the technical field of automatic driving. The method comprises the following steps: constructing a vectorized map, and constructing a city road network structure by using the vectorized map; capturing complex road information and continuous vehicle trajectory information according to the city road network structure; calculating and acquiring multi-line level features according to the complex road information and the continuous vehicle trajectory information; constructing a global graph model according to the multi-line level features; and decoding a target node by using a recurrent neural network decoder according to the global graph model to obtain a vehicle trajectory prediction result. The method can high-quality extract complex road information and continuous vehicle trajectory information, so that the vehicle trajectory prediction precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a vehicle trajectory prediction method, device, system and storage medium, belonging to the technical field of automatic driving. BACKGROUND

[0002] Autonomous driving is attracting more and more attention from academia and industry, because it has great potential in solving many long-term traffic challenges such as safety, congestion, energy saving, etc. In recent years, we have witnessed the rapid development of perception, planning and control systems for autonomous vehicles. However, only when the safety of autonomous driving is verified can mass production be realized. In order to further improve safety, one of the most critical technologies is that autonomous vehicles should be able to predict the trajectories of surrounding vehicles in real time like human drivers in order to make judgments.

[0003] When a person drives a vehicle, he or she usually observes the surrounding traffic participants and predicts their future state before starting a new driving motivation. The future trajectory can represent the future state of the traffic participant, which is used to detect potential dangers in advance and to design decision-making or planning algorithms. However, due to the diversity of traffic participant motivations, the complex interaction between traffic participants and the environment, the uncertainty of sensory information, the computational burden and time requirements of autonomous vehicles, how to accurately predict the future trajectory of traffic participants has attracted much attention and become one of the key points to improve the safety of autonomous driving.

[0004] In recent years, the traffic prediction method based on deep learning trains the network to make the model have autonomous learning characteristics, which greatly improves the performance of vehicle trajectory prediction, including: CS-LSTM(M) model based on convolution social pool long short-term memory network, MHA-LSTM model based on attention mechanism long short-term memory network, etc.; VectorNet model based on vectorization network, GISNet model based on graph information sharing network, etc. However, the map is a graph structure with complex topology, and the grid process inevitably leads to information loss. In addition, although two pairs of lanes are close to each other in space, the lane pairs in the same or opposite directions have completely different semantic meanings and dependencies. Moreover, in order to obtain long-distance lane information, the receptive field sometimes not only covers the expected area, but also covers a large area outside the lane. Therefore, vehicle trajectory prediction is still a challenging task, and the existing vehicle trajectory prediction methods still have room for improvement in prediction accuracy and stability. SUMMARY

[0005] The purpose of the present application is to provide a vehicle trajectory prediction method, device, system and storage medium, which can extract complex road information and continuous vehicle trajectory information with high quality, so as to improve the vehicle trajectory prediction accuracy.

[0006] To achieve the above object, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a vehicle trajectory prediction method, comprising:

[0008] constructing a vectorized map and using the vectorized map to construct a city road network structure;

[0009] According to the city road network structure, complex road information and continuous vehicle trajectory information are captured;

[0010] According to the complex road information and continuous vehicle trajectory information, multi-line level features are calculated and obtained;

[0011] According to the multi-line level features, a global graph model is constructed;

[0012] According to the global graph model, a recurrent neural network decoder is used to decode the target node to obtain a vehicle trajectory prediction result.

[0013] In combination with the first aspect, further, the vectorized map is constructed, comprising:

[0014] Sampling the line to obtain a sampling point;

[0015] Connecting the sampling points to form a polyline;

[0016] Vectorizing the polyline to obtain a vector set;

[0017] Taking the vectors in the vector set as nodes, a vectorized map is constructed on the vector set through the connection of the nodes;

[0018] Wherein, the line includes a vehicle trajectory and a city road;

[0019] The expression of the node is shown in formula (1):

[0020] v=[d s , d e , a, r] (1)

[0021] In formula (1), v is a node v, i.e. a vector v in the vector set, d m，j，k is the starting point coordinate of the vector v, d m∈{pre，suc，left，rig ht} is the end point coordinate of the vector v, a is the attribute feature corresponding to the vector v, and r is the integer ID of the polyline to which the vector v belongs.

[0022] In combination with the first aspect, further, the city road network structure is constructed using the vectorized map, comprising:

[0023] constructing a city road network structure according to a feature matrix of each node in the vectorized map and an adjacency matrix representing connectivity between the nodes;

[0024] The feature matrix is composed of attribute features of the nodes, and the attribute features include a timestamp of a vehicle trajectory, a feature type of a city road, and a lane speed limit.

[0025] The expression of the adjacency matrix is shown in formula (2):

[0026] A = {A m，j，k} m∈{pre，suc，left，rig ht} (2)

[0027] In formula (2), A is the adjacency matrix, k is the kth node, j is the jth node, m is the relationship index between the kth node and the jth node, pre, suc, left, and right are predecessor, successor, left neighbor, and right neighbor relationships, A m，j，k is a connectivity variable between the kth node and the jth node, A m，j，k = 1 indicates that the kth node is an m-type relationship of the jth node, and A m，j，k = 0 indicates that the kth node is not an m-type relationship of the jth node, wherein for any node, its predecessor node is defined as an adjacent node that can go to the node, its successor node is defined as an adjacent node that can start from the node, and its left neighbor and right neighbor nodes are defined as the leftmost and rightmost nodes of the node.

[0028] In combination with the first aspect, further, capturing complex road information and continuous vehicle trajectory information according to the city road network structure includes:

[0029] connecting all nodes belonging to the same polyline in the city road network structure to each other to construct a subgraph at a vector level;

[0030] capturing complex road information by using a pre-obtained extended graph convolution network according to the subgraph;

[0031] capturing continuous vehicle trajectory information by using a pre-obtained recurrent neural network according to the subgraph;

[0032] The capturing of complex road information by using the pre-obtained extended graph convolution network includes:

[0033] constructing a filter acting on each node in the subgraph in a Fourier domain;

[0034] performing convolution processing on the feature matrix of each node in the subgraph and the adjacency matrix representing the connectivity between the nodes, and capturing complex road information around each node in the subgraph by using the filter;

[0035] The expression for the complex road information is shown in formula (3):

[0036]

[0037] In formula (3), Y is the output of the filter, i.e., complex road information, X is the feature matrix, and A is the adjacency matrix. I is the identity matrix. for The degree matrix, W is the weight matrix used to perform symmetric normalization on the adjacency matrix A;

[0038] The expression for the temporal characteristics of the continuous vehicle trajectory information is shown in formula (4):

[0039]

[0040] In formula (4), t is time t, O t Let P be the output of the recurrent neural network at time t. t Let S be the input of the recurrent neural network at time t. t Let S be the hidden value of the recurrent neural network at time t. t-1 Let t be the hidden value of the recurrent neural network at time t-1, V be the parameter matrix from the hidden layer to the output layer in the recurrent neural network, U be the parameter matrix from the input layer to the hidden layer in the recurrent neural network, B be the weight matrix between each time point, and f be the recurrent formula.

[0041] In conjunction with the first aspect, further, based on the complex road information and continuous vehicle trajectory information, the calculation and acquisition of multi-line level features includes:

[0042] For the polyline and its nodes in the subgraph, a single-level subgraph propagation operation is defined;

[0043] According to the single-layer subgraph propagation operation, the complex road information and continuous vehicle trajectory information are input into the subgraph, and multi-line level features are calculated and obtained by each node in the subgraph.

[0044] The expression for the single-layer subgraph propagation operation is shown in formula (5):

[0045]

[0046] In formula (5), v is node v, v (l+1) Let v be the node feature of the (l+1)th layer. (l) Let u be the feature of the l-th layer node v. (l) For nodes near node v, g enc (·) To enable recurrent neural networks to exchange features of individual nodes. to aggregate information of all neighboring nodes by using the extended graph convolutional network, for a simple connection between the node v and its adjacent nodes;

[0047] The calculation formula of the multi-line level feature is shown as formula (6):

[0048]

[0049] In formula (6), L v is the multi-line level feature of the node v, is a max-pooling operation.

[0050] In combination with the first aspect, further, constructing the global graph model according to the multi-line level feature includes:

[0051] According to the multi-line level feature, a pre-acquired graph neural network and a self-attention mechanism are used to model high-order interaction of node features of all nodes belonging to the same polyline in the subgraph, so as to acquire the global graph model.

[0052] In formula (7), the expression of the global graph model is shown as formula (7):

[0053]

[0054] In formula (7), the expression of the global graph model is shown as formula (7): is the multi-line level node feature of the n th node v belonging to the same polyline after the multi-line level feature is transmitted through the l+1 layer graph neural network, is the l+1 layer multi-line level node feature set of the nodes belonging to the same polyline, is the multi-line level node feature of the n th node v belonging to the same polyline after the multi-line level feature is transmitted through the l layer graph neural network, is the l layer multi-line level node feature set of the nodes belonging to the same polyline, is the adjacency matrix corresponding to the node set belonging to the same polyline, is a fully connected graph, is the feature matrix of the node after the multi-line level feature extraction, are linear projections of the feature matrix of the node after the multi-line level feature extraction , and GNN(·) is a single-layer graph neural network.

[0055] In combination with the first aspect, further, the formula for decoding the target node by using the recurrent neural network decoder is shown as formula (8):

[0056]

[0057] In formula (8), v futureL is a vehicle trajectory prediction result of a node v v L is a multi-line level feature of a node v L is a total number of layers of a graph neural network L is a multi-line level feature of a node v v L is a feature obtained after passing through all layers of a graph neural network L is a recurrent neural network decoder.

[0058] In a second aspect, the present application provides a vehicle trajectory prediction device, comprising:

[0059] a network construction module, configured to construct a vectorized map, and construct a city road network structure by using the vectorized map;

[0060] an information capturing module, configured to capture complex road information and continuous vehicle trajectory information according to the city road network structure;

[0061] a multi-line level feature calculation module, configured to calculate and obtain a multi-line level feature according to the complex road information and the continuous vehicle trajectory information;

[0062] a global graph model construction module, configured to construct a global graph model according to the multi-line level feature;

[0063] a prediction module, configured to decode a target node by using a recurrent neural network decoder according to the global graph model, and obtain a vehicle trajectory prediction result.

[0064] In a third aspect, the present application provides a system, comprising a processor and a storage medium;

[0065] the storage medium is configured to store instructions;

[0066] the processor is configured to operate according to the instructions to perform steps of the method according to any one of the first aspect.

[0067] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps of the method according to any one of the first aspect.

[0068] Compared with the prior art, the present application has the following beneficial effects:

[0069] The vehicle trajectory prediction method provided by the application uses a new map interaction, directly vectorizes the original map to construct a map, can avoid information loss, extracts complex road information and continuous vehicle trajectory information with high quality, calculates multi-line level features, constructs a global graph model for vehicle trajectory prediction, and can improve the vehicle trajectory prediction accuracy. The extended graph convolution network is used to perform convolution processing on the feature matrix and the adjacency matrix, which can capture lane features with complex topological structure and spatial dependence, thereby obtaining complex road information. Because the vehicle trajectory information is a continuous sequence in time, the recurrent neural network is used to capture the vehicle trajectory features, so as to obtain the dynamic change information of the vehicle trajectory, and the complex road information is combined to realize higher precision prediction. The hierarchical graph network is constructed, the feature information obtained by the extended graph convolution network and the recurrent neural network in the local subgraph is transmitted to the global graph, the self-attention mechanism is used for high-order interaction between lanes, lanes and vehicles, and vehicles and vehicles, and then high-accuracy and high-stability trajectory prediction is realized. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is a vehicle trajectory prediction method flowchart provided by an embodiment of the application;

[0071] Figure 2 is a whole architecture schematic diagram of the vehicle trajectory prediction provided by an embodiment of the application;

[0072] Figure 3 is a schematic diagram for capturing complex road information by using an extended graph convolution network provided by an embodiment of the application, wherein (a) is an extended graph convolution process schematic diagram of a complex road node, and (b) is an extended graph convolution network structure schematic diagram;

[0073] Figure 4 is a global graph model schematic diagram provided by an embodiment of the application;

[0074] Figure 5 is a comparison result schematic diagram of average displacement error ADE and final displacement error FDE on an Argoverse data set provided by an embodiment of the application;

[0075] Figure 6 is a root mean square error RMSE comparison result schematic diagram on an NGSIM I-80 data set provided by an embodiment of the application. DETAILED DESCRIPTION

[0076] The technical solutions of the application will be further described in detail below in combination with specific embodiments.

[0077] Embodiments of the present application are described below in detail, examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only, for the purpose of explanation, and are not to be understood as limiting the present application. The embodiments of the present application and the technical features in the embodiments can be combined with each other without conflict.

[0078] Embodiment one:

[0079] Figure 1 is a flowchart of a vehicle trajectory prediction method provided by the first embodiment of the present application, the flowchart only shows the logical order of the method of the present embodiment, and the steps shown or described can be completed in an order different from that shown in other possible embodiments of the present application without conflict. Figure 1

[0080] The vehicle trajectory prediction method provided by the present embodiment can be applied to a terminal and can be executed by a vehicle trajectory prediction device, which can be implemented in software and / or hardware. The device can be integrated in a terminal, such as any tablet computer or computer device with communication function. Referring to Figure 1 , the method of the present embodiment specifically includes the following steps:

[0081] Step one: construct a vectorized map and use the vectorized map to construct a city road network structure;

[0082] The vectorized map is a map composed of vectorized representations of vehicle trajectories, city roads, etc. The vehicle trajectory is a directed spline form with respect to time, which can be approximated as a vector sequence. For a vehicle trajectory, we can sample the points in the vehicle trajectory at fixed time intervals and connect them into vectors. Given a small enough spatial or temporal interval, the resulting polyline can be used as an approximation of the original map and vehicle trajectory. The vectorization process is a one-to-one mapping between continuous trajectories, mapping annotations, and vector sets, which allows us to form a graph representation on the vector set, which can be encoded using a graph neural network. Specifically, we regard each vector belonging to the map polyline as a node in the graph, and the node represents the vehicle trajectory and city road in the form of a vector. We regard the vector as a node in the graph, and form a vectorized map on the vector set through the connection of the nodes.

[0083] Constructing a vectorized map includes the following steps:

[0084] Step ①: sample the route to obtain the sampling points;

[0085] In the present embodiment, the route includes vehicle trajectories and city roads.

[0086] ​Step 2: connecting the sampling points to form a polyline;

[0087] Step 3: vectorizing the polyline to obtain a vector set;

[0088] Step 4: taking the vectors in the vector set as nodes, and constructing a vectorized map on the vector set through the connection of the nodes.

[0089] The expression of the node is shown in formula (1):

[0090] v=[d s ,d e ,a,r](1)

[0091] In formula (1), v is a node v, i.e., a vector v in the vector set, d s is the starting point coordinate of the vector v, d e is the ending point coordinate of the vector v, a is the attribute feature corresponding to the vector v, and r is the integer ID of the polyline to which the vector v belongs.

[0092] Each node in the vectorized map contains rich attribute features, such as the timestamp of the vehicle trajectory, the characteristic type of the urban road, and the lane speed limit, which helps us to construct the urban road network structure in the form of a graph by obtaining the attribute features of the nodes and the connection between the nodes.

[0093] Using the vectorized map, the construction of the urban road network structure includes: constructing the urban road network structure according to the feature matrix of each node in the vectorized map and the adjacency matrix representing the connectivity between each node.

[0094] The feature matrix is composed of the attribute features of each node, and in this embodiment, the attribute features include: the timestamp of the vehicle trajectory, the characteristic type of the urban road, and the lane speed limit.

[0095] The expression of the adjacency matrix is shown in formula (2):

[0096] A={A m，j，k} m∈{pre，suc，left，rig ht} (2)

[0097] In formula (2), A is the adjacency matrix, k is the kth node, j is the jth node, m is the relationship index between the kth node and the jth node, pre, suc, left, and right are the predecessor, successor, left neighbor, and right neighbor relationships, A m，j，k is the connectivity variable between the kth node and the jth node, A m，j，k = 1 indicates that the kth node is the m-type relationship of the jth node, and A m，j，k=0 indicates an m-type relationship between the k-th node and the j-th node. For any node, its predecessor node is defined as a node that can reach the node, its successor node is defined as a node that can start from the node, and its left and right neighbors are defined as the nodes that are closest to the node on the left and right, i.e., the nodes that are closest in space, which are measured by the distance on the left and right adjacent lanes, respectively.

[0098] The adjacency matrix further improves the urban road network structure from the perspective of connectivity, which helps subsequent extended graph convolutional networks to capture complex road information using the urban road network structure.

[0099] Step 2: Capture complex road information and continuous vehicle trajectory information based on the urban road network structure;

[0100] After constructing the urban road network structure, a hierarchical approach is adopted to leverage the spatial and semantic locality of nodes. This involves connecting all nodes belonging to the same polyline to construct a subgraph at the vector level. Complex road information includes a range of factors affecting vehicle trajectories, such as road feature types, lane speed limits, and connectivity between lanes. Ordinary graph convolutional networks can only capture the spatial features of the adjacent parts of nodes, while extended graph convolutional networks can utilize four adjacency matrices to capture a broader range of further relationships. For example... Figure 3 The diagram illustrates the use of an extended graph convolutional network to capture complex road information in this embodiment. (a) shows the extended graph convolution process for complex road nodes, and (b) shows the structure of the extended graph convolutional network. In (b), each convolutional module consists of an extended graph convolution (1, 2, 4, 8, 16, 32) and a linear layer with residual connections. All layers have 128 feature channels. Based on the complex topological structure of lanes, the extended graph convolutional network is used to convolve the adjacency matrix and the feature matrix composed of node attribute features in the urban road network structure to obtain spatial information. Specifically, a filter is constructed in the Fourier domain to act on the node, capturing the complex road information around that node. Because vehicle trajectory information is a continuous temporal sequence, a recurrent neural network is used to capture the temporal features of vehicle trajectory information.

[0101] Based on the urban road network structure, capturing complex road information and continuous vehicle trajectory information includes the following steps:

[0102] Step A: Connect all nodes belonging to the same polyline in the urban road network structure to each other, and construct a subgraph at the vector level;

[0103] Step B: Based on the subgraph, use the pre-acquired extended graph convolutional network to capture complex road information;

[0104] Capturing complex road information using a pre-acquired extended graph convolutional network involves the following steps:

[0105] Step i: Construct filters that act on each node in the subgraph in the Fourier domain;

[0106] Step ii: Perform convolution processing on the feature matrix of each node in the subgraph and the adjacency matrix representing the connectivity between nodes, and use filters to capture the complex road information around each node in the subgraph;

[0107] The expression for complex road information is shown in formula (3):

[0108]

[0109] In formula (3), Y is the output of the filter, i.e., complex road information, X is the feature matrix, and A is the adjacency matrix. I is the identity matrix. for The degree matrix, To perform symmetric normalization on the adjacency matrix A, W is the weight matrix. In this embodiment, all connections share the same weight.

[0110] Step C: Based on the subgraph, capture continuous vehicle trajectory information using the pre-acquired recurrent neural network;

[0111] The expression for the temporal characteristics of continuous vehicle trajectory information is shown in formula (4):

[0112]

[0113] In formula (4), t is time t, O t Let P be the output of the recurrent neural network at time t. t Let S be the input of the recurrent neural network at time t. t Let S be the hidden value of the recurrent neural network at time t. t-1 Let t be the hidden value of the recurrent neural network at time t-1, V be the parameter matrix from the hidden layer to the output layer in the recurrent neural network, U be the parameter matrix from the input layer to the hidden layer in the recurrent neural network, B be the weight matrix between each time point, and f be the recursive formula representing the continuous transmission of the hidden value.

[0114] Step 3: Calculate and obtain multi-line level features based on complex road information and continuous vehicle trajectory information;

[0115] Since the nodes are interconnected, the complex road information and continuous vehicle trajectory information extracted in step two are input into the subgraph. The nodes in the subgraph obtain multi-line level features through calculation. The multi-line level features include multiple features extracted from the complex road information and continuous vehicle trajectory information.

[0116] According to the complex road information and the continuous vehicle trajectory information, the multi-line level feature is calculated by the following steps:

[0117] Step a: defining a single-layer subgraph propagation operation for the polyline and its nodes in the subgraph;

[0118] The expression of the single-layer subgraph propagation operation is shown in formula (5):

[0119]

[0120] In formula (5), v is a node v, v (l+1) is an (l+1)th layer node feature of the node v, v (l) is an lth layer node feature of the node v, u (l) is a node near the node v, g enc (·) is a function of making a recurrent neural network exchange a single node feature, is a function of aggregating information of all adjacent nodes by using an extended graph convolutional network, is a simple connection between the node v and its adjacent nodes.

[0121] Step b: inputting the complex road information and the continuous vehicle trajectory information into the subgraph according to the single-layer subgraph propagation operation, and calculating the multi-line level feature from each node in the subgraph;

[0122] The calculation formula of the multi-line level feature is shown in formula (6):

[0123]

[0124] In formula (6), L v is a multi-line level feature of the node v, is a max-pooling operation.

[0125] Step four: constructing a global graph model according to the multi-line level feature;

[0126] In this embodiment, when the global graph model is constructed according to the multi-line level feature, the node features of all nodes belonging to the same polyline in the subgraph are modeled by high-order interaction by using a pre-obtained graph neural network and a self-attention mechanism according to the multi-line level feature, so as to obtain the global graph model.

[0127] The expression of the global graph model is shown in formula (7):

[0128]

[0129] In formula (7), is a multi-line level node feature of an nth node v belonging to the same polyline after the multi-line level feature is transmitted by an (l+1)th layer graph neural network, the (l+1)th multi-line level node feature set of the nodes belonging to the same polyline, the multi-line level node feature of the nth node v belonging to the same polyline after the multi-line level feature passes through the l-layer graph neural network, the lth multi-line level node feature set of the nodes belonging to the same polyline, the adjacency matrix corresponding to the node set belonging to the same polyline, is a fully connected graph, is a feature matrix of the nodes after multi-line level feature extraction, are linear projections of the feature matrix of the nodes after multi-line level feature extraction , and GNN(·) is a single-layer graph neural network.

[0130] As shown in Figure 4 , it is a global graph model schematic diagram provided by the embodiment, by constructing a hierarchical graph network, the feature information obtained by the extended graph convolution network and the recurrent neural network in the local subgraph is transmitted to the global graph, the self-attention mechanism is used for high-order interaction of lane and lane, lane and vehicle, and vehicle and vehicle, and then high-accuracy and high-stability trajectory prediction is realized.

[0131] Step five: according to the global graph model, the target node is decoded by using the recurrent neural network decoder, and the vehicle trajectory prediction result is obtained;

[0132] The formula for decoding the target node by using the recurrent neural network decoder is shown in formula (8):

[0133]

[0134] In formula (8), v future is the vehicle trajectory prediction result of the node v, L v is the multi-line level feature of the node v, is the total number of layers of the graph neural network, is the feature obtained after the multi-line level feature L v of the node v passes through all layers of the graph neural network, is the recurrent neural network decoder.

[0135] As shown in Figure 2 , it is a whole architecture schematic diagram of vehicle trajectory prediction provided by the embodiment, Figure 2In the method, firstly, sub-graphs are constructed respectively, a graph constructed by vehicle trajectory nodes is used to extract continuous vehicle trajectory information by using a recurrent neural network, a graph constructed by road nodes is used to extract complex road information by using an extended graph convolutional neural network, and other information indirectly affecting the vehicle trajectory, such as a pedestrian crossing, is additionally reserved, then all nodes with information are constructed into a global graph, and features are further obtained by using a graph neural network and a self-attention mechanism, and finally, a recurrent neural network decoder is used to predict the future trajectory of the vehicle.

[0136] The vehicle trajectory prediction method provided in the embodiment uses a new map interaction, directly vectorizes the original map to construct a map, and can avoid information loss; the extended graph convolutional network is used to perform convolutional processing on the feature matrix and the adjacency matrix, so as to capture lane features and spatial dependencies with complex topological structures, thereby obtaining complex road information; the use of multiple adjacency matrices can help us get rid of the limitation that the graph convolutional network can only capture the information of adjacent nodes, so that the current node can obtain the road information of multiple adjacent nodes, complete the extended graph convolutional network, and improve the road spatial information acquisition capability; because the vehicle trajectory information is a continuous sequence in time, the recurrent neural network is used to capture the vehicle trajectory features, so as to obtain the dynamic change information of the vehicle trajectory, and combine the complex road information to realize higher-precision prediction; the local sub-graphs are classified and integrated in the form of the extended graph convolutional network, the recurrent neural network and the vectorized map, and the rich information of the lane graph, the vehicle running trajectory and the high-precision map, so as to help the model complete better high-order interaction, thereby making the vehicle trajectory prediction method have better prediction performance; the hierarchical graph network is constructed, the feature information obtained by the extended graph convolutional network and the recurrent neural network in the local sub-graphs is transmitted to the global graph, the self-attention mechanism is used for high-order interaction between lanes, between lanes and vehicles, and between vehicles, and high-accuracy and high-stability trajectory prediction is realized.

[0137] To verify the beneficial effects of the vehicle trajectory prediction method provided in the embodiment, the following scientific demonstration is performed through economic benefit calculation and simulation experiment.

[0138] Firstly, for the vehicle trajectory prediction method provided in the embodiment, the average displacement error ADE and the final displacement error FDE are used as indexes for evaluating the accuracy of the trajectory prediction method, the ADE represents the average distance between the predicted trajectory and the actual trajectory, and the FDE represents the distance between the final prediction result and the actual real position, and the root mean square error RMSE is used as an index for stability of the trajectory prediction method, representing the fluctuation range of the prediction result. The expressions of the ADE, the FDE and the RMSE are as follows.

[0139]

[0140] where i is the i-th prediction object, N q is the total number of prediction objects, t is the prediction time sample t, T is the total prediction duration, Y t is the predicted trajectory, actual trajectory of the i-th prediction object at the prediction time sample t, Y T is the final prediction result, actual position of the i-th prediction object, and σ is the number of prediction object samples, Y t is the predicted trajectory, actual trajectory at the prediction time sample t.

[0141] The experiment was run in an environment with an Intel Core i7 processor, an NVIDA GeForce RTX2070 SUPER graphics card, and 16GB of RAM, and the operating system used was 64-bit Windows 10.

[0142] The data set used in the experimental data includes Argoverse and NGSIM I-80; the data in Argoverse comes from a subset of Argo AI's autonomous driving test vehicles operating in the Miami and Pittsburgh areas, which are two American cities with different urban driving challenges and driving habits. The data covers the recording of sensor data and log fragments, including different seasons, different weather conditions and different times of the day, to provide a wide range of real-world driving scenarios. The total lane length covers 204 kilometers in Miami and 86 kilometers in Pittsburgh; the NGSIM I-80 data set is data collected and processed on April 13, 2005 on Interstate 80 in Emeryville, California, and there are a total of 45 minutes of data available in the complete data set, divided into three 15-minute time periods: 16:00-16:15, 17:00-17:15, and 17:15-17:30, containing detailed trajectory data, wide-area probe data. The vehicle trajectory data provides the precise position of each vehicle within the study area every tenth of a second, resulting in detailed lane position and position relative to other vehicles.

[0143] As Figure 5 shown, the comparative results of the average displacement error ADE and the final displacement error FDE provided by the present embodiment on the Argoverse data set are shown in the schematic diagram, which is Figure 5It can be seen that the vehicle trajectory prediction method (GRA-MLN in this embodiment) provided in this embodiment has obvious reduction in average displacement error ADE and final displacement error FDE on the Argoverse dataset compared with Constant Velocity, Nearest Neighbor, LSTM ED based on long short-term memory network, uulm-mrm, VectorNet based on vectorization network and LaneGCN based on graph convolutional neural network, and the reduction rates are 58.07%, 57.10%, 31.16%, 22.11%, 18.23%, 13.45% and 62.74%, 62.69%, 40.61%, 29.83%, 26.68%, 22.22% respectively, indicating that the prediction accuracy of the experimental results of the vehicle trajectory prediction method provided in this embodiment is relatively high.

[0144] As shown in Figure 6 , it is a comparison result diagram of root mean square error RMSE on NGSIM I-80 dataset provided in this embodiment, Figure 6 In the figure, different colors respectively represent the root mean square error produced when predicting the vehicle trajectory in the future 1 second, 2 seconds, 3 seconds, 4 seconds and 5 seconds, and Figure 6 It can be seen that the vehicle trajectory prediction method (GRA-MLN in this embodiment) provided in this embodiment has 7.50%-50.00%, 15.15%-52.54%, 17.71%-53.85%, 20.74%-55.32% and 18.59%-53.37% improvement in future trajectory prediction RMSE index within τ seconds (τ = 1, 2, 3, 4, 5) on NGSIM I-80 dataset compared with Constant Velocity, IMM-KF based on Kalman filter, C-VGMM+VIM of unified framework of maneuver classification and motion prediction, MFP-1 of multi-step future prediction, CS-LSTM(M) based on convolution social pool long short-term memory network and MHA-LSTM based on attention mechanism, which is very close to the current best GISNet model based on graph information sharing network, indicating that the prediction result error of the vehicle trajectory prediction method provided in this embodiment is relatively small and the stability is relatively high.

[0145] Therefore, the vehicle trajectory prediction method provided in this embodiment can effectively reduce the average displacement error ADE and the final displacement error FDE of the vehicle trajectory prediction data, and can effectively reduce the root mean square error RMSE, indicating that the vehicle trajectory prediction method provided in this embodiment can effectively improve the accuracy of vehicle trajectory prediction while maintaining stability, and thus achieve better vehicle trajectory prediction effect.

[0146] Embodiment Two:

[0147] The embodiment provides a vehicle trajectory prediction device, comprising:

[0148] a network construction module, configured to construct a vectorized map, and construct a city road network structure by using the vectorized map;

[0149] an information capturing module, configured to capture complex road information and continuous vehicle trajectory information according to the city road network structure;

[0150] a multi-line level feature calculation module, configured to calculate and obtain multi-line level features according to the complex road information and the continuous vehicle trajectory information;

[0151] a global graph model construction module, configured to construct a global graph model according to the multi-line level features;

[0152] a prediction module, configured to decode a target node by using a recurrent neural network decoder according to the global graph model, and obtain a vehicle trajectory prediction result.

[0153] The vehicle trajectory prediction device provided by the embodiment can execute the vehicle trajectory prediction method provided by any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0154] Embodiment three:

[0155] The embodiment provides a system, comprising a processor and a storage medium.

[0156] The storage medium is used for storing instructions.

[0157] The processor is used for operating according to the instructions to execute the steps of the method in the embodiment one.

[0158] Embodiment four:

[0159] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the method in the embodiment one.

[0160] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system or a computer program product. Therefore, the application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0161] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0162] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0163] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0164] The above merely provides the preferred embodiment of the present application, and it should be noted that for those of ordinary skill in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should also be considered as falling within the scope of the present application.

Claims

1. A vehicle trajectory prediction method, characterized by, The application relates to a method for predicting vehicle trajectories based on a vectorized map. The method comprises the following steps: constructing a vectorized map and using the vectorized map to construct a city road network structure; capturing complex road information and continuous vehicle trajectory information according to the city road network structure; calculating and obtaining multi-line level features according to the complex road information and the continuous vehicle trajectory information; constructing a global graph model according to the multi-line level features; decoding a target node by using a recurrent neural network decoder according to the global graph model to obtain a vehicle trajectory prediction result; constructing a city road network structure by using a vectorized map comprises: constructing the city road network structure according to a feature matrix of each node in the vectorized map and an adjacency matrix representing the connectivity between the nodes; The expression of the adjacency matrix is shown in Equation (2): (2); In formula (2), It is an adjacency matrix. For the first 1 node For the first 1 node For the first The node and the first Index of relationships between nodes , , , Relationships are defined as predecessor, successor, left neighbor, and right neighbor. For the first The node and the first Connectivity variables between nodes Indicates the first The node is the _th Each node Type relationship, Indicates the first The node is not the first Each node The relationship is of the type where, for any node, its predecessor node is defined as the neighboring node that can reach the node, its successor node is defined as the neighboring node that can be started from the node, and its left and right neighbors are defined as the nodes that are closest to the node on the left and right. wherein the feature matrix is composed of attribute features of the nodes, and the attribute features include a timestamp of a vehicle trajectory, a feature type of a city road and a lane speed limit; capturing complex road information and continuous vehicle trajectory information according to the city road network structure comprises: connecting all nodes belonging to the same polyline in the city road network structure to each other to construct a subgraph at a vector level; capturing complex road information by using a pre-obtained extended graph convolutional network according to the subgraph; capturing continuous vehicle trajectory information by using a pre-obtained recurrent neural network according to the subgraph; wherein capturing complex road information by using the pre-obtained extended graph convolutional network comprises: constructing a filter acting on each node in the subgraph in a Fourier domain; The expression of complex road information is shown in equation (3): (3); In formula (3), The output of the filter, i.e., complex road information, For the characteristic matrix, It is an adjacency matrix. , It is the identity matrix. for The degree matrix, For the adjacency matrix Perform symmetric normalization. The weight matrix; performing convolution processing on the feature matrix of each node in the subgraph and the adjacency matrix representing the connectivity between the nodes to capture complex road information around each node in the subgraph by using the filter; （4）； In Equation (4), is the time, is the output of the recurrent neural network at time t, is the input of the recurrent neural network at time t, is the hidden value of the recurrent neural network at time t, is the hidden value of the recurrent neural network at time t, is the parameter matrix from the hidden layer to the output layer in the recurrent neural network, is the parameter matrix from the input layer to the hidden layer in the recurrent neural network, is the weight matrix between each time point, is the recurrent recurrence formula.

2. The vehicle trajectory prediction method of claim 1, wherein, the expression of the time feature of the continuous vehicle trajectory information is shown in formula (4): constructing a vectorized map comprises: sampling a line to obtain a sampling point; connecting the sampling points to form a polyline; vectorizing the polyline to obtain a vector set; taking vectors in the vector set as nodes to form a vectorized map on the vector set by connecting the nodes; The expression of the node is shown as formula (1): (1); In formula (1), is a node , i.e. a vector in the vector set , is a start point coordinate of the vector , is an end point coordinate of the vector , is a corresponding attribute feature of the vector , is an integer ID of a polyline to which the vector belongs.

3. The vehicle trajectory prediction method of claim 1, wherein, wherein the line comprises a vehicle trajectory and a city road; calculating and obtaining multi-line level features according to the complex road information and the continuous vehicle trajectory information comprises: defining a single-layer subgraph propagation operation for the polyline and the nodes thereof in the subgraph; inputting the complex road information and the continuous vehicle trajectory information into the subgraph according to the single-layer subgraph propagation operation, and calculating and obtaining multi-line level features from each node in the subgraph; （5）； In formula (5), is a node , is a node of the first layer node feature, is a node of the first layer node feature, is a node nearby the node, is to make the recurrent neural network exchange for a single node feature, is to aggregate information of all adjacent nodes by using an extended graph convolutional network, is a simple connection between the node and its adjacent nodes; The calculation formula of the multi-line level feature is shown as formula (6): (6); In formula (6), For nodes Multi-level features, This is a max pooling operation.

4. The vehicle trajectory prediction method of claim 1, wherein, wherein the expression of the single-layer subgraph propagation operation is shown in formula (5): constructing a global graph model according to the multi-line level features comprises: The expression of the global graph model is shown as formula (7): (7); In formula (7), is the multi-line level feature of the i-th node belonging to the same polyline after the multi-line level feature of the node is transmitted through the layer graph neural network, is the multi-line level node feature after the multi-line level feature of the i-th node belonging to the same polyline is transmitted through the layer graph neural network, is the multi-line level node feature set of the node belonging to the same polyline, is the multi-line level feature of the i-th node belonging to the same polyline after the multi-line level feature of the node is transmitted through the layer graph neural network, is the multi-line level node feature set of the node belonging to the same polyline, is the adjacency matrix corresponding to the node set belonging to the same polyline, is a fully connected graph, is the feature matrix of the node after the multi-line level feature extraction, are linear projections of the feature matrix of the node after the multi-line level feature extraction, is a single-layer graph neural network.​​​​​​​​​​ 5. The vehicle trajectory prediction method of claim 1, wherein, The formula for decoding the target node using the recurrent neural network decoder is shown as formula (8): (8); In formula (8), is a vehicle trajectory prediction result of the node , is a multi-line level feature of the node , is a total number of layers of the graph neural network, is a multi-line level feature of the node , is a feature obtained after the node passes through all layers of the graph neural network, is a recurrent neural network decoder.

6. A vehicle trajectory prediction device characterized by comprising: modeling high-order interactions of node features of all nodes belonging to the same polyline in the subgraph by using a pre-obtained graph neural network and a self-attention mechanism according to the multi-line level features to obtain a global graph model; The application further provides a vehicle trajectory prediction method based on a vectorized map. The method comprises the following steps: constructing a vectorized map and using the vectorized map to construct a city road network structure; capturing complex road information and continuous vehicle trajectory information according to the city road network structure; calculating and obtaining multi-line level features according to the complex road information and the continuous vehicle trajectory information; constructing a global graph model according to the multi-line level features; decoding a target node by using a recurrent neural network decoder according to the global graph model to obtain a vehicle trajectory prediction result; A global graph model construction module is configured to construct a global graph model according to the multi-line level features; A prediction module is configured to decode the target node by using a recurrent neural network decoder according to the global graph model, and obtain a vehicle trajectory prediction result. The vectorized map is used to construct the urban road network structure, including: The urban road network structure is constructed according to the feature matrix of each node in the vectorized map and the adjacency matrix representing the connectivity between the nodes. The feature matrix is composed of attribute features of each node, and the attribute features include a timestamp of the vehicle trajectory, a feature type of the urban road, and a lane speed limit; The expression of the adjacency matrix is shown in Equation (2): (2); In formula (2), It is an adjacency matrix. For the first 1 node For the first 1 node For the first The node and the first Index of relationships between nodes , , , Relationships are defined as predecessor, successor, left neighbor, and right neighbor. For the first The node and the first Connectivity variables between nodes Indicates the first The node is the _th Nodes Type relationship, Indicates the first The node is not the first Each node The relationship is of the type where, for any node, its predecessor node is defined as the neighboring node that can reach the node, its successor node is defined as the neighboring node that can be started from the node, and its left and right neighbors are defined as the nodes that are closest to the node on the left and right. The complex road information and the continuous vehicle trajectory information are captured according to the urban road network structure, including: All nodes belonging to the same polyline in the urban road network structure are connected to each other to construct a subgraph at the vector level; The complex road information is captured by using a pre-obtained extended graph convolution network according to the subgraph; The continuous vehicle trajectory information is captured by using a pre-obtained recurrent neural network according to the subgraph; The complex road information is captured by using the pre-obtained extended graph convolution network, including: A filter acting on each node in the subgraph is constructed in the Fourier domain; The feature matrix of each node in the subgraph and the adjacency matrix representing the connectivity between the nodes are convoluted to capture the complex road information around each node in the subgraph by using the filter. The expression of complex road information is shown in equation (3): (3); In formula (3), The output of the filter, i.e., complex road information, The characteristic matrix, It is an adjacency matrix. , It is the identity matrix. for The degree matrix, For the adjacency matrix Perform symmetric normalization. The weight matrix; The expression of the time feature of the continuous vehicle trajectory information is shown in formula (4): （4）； In Equation (4), is the time, is the output of the recurrent neural network at time t, is the input of the recurrent neural network at time t, is the hidden value of the recurrent neural network at time t, is the hidden value of the recurrent neural network at time t, is the parameter matrix from the hidden layer to the output layer in the recurrent neural network, is the parameter matrix from the input layer to the hidden layer in the recurrent neural network, is the weight matrix between each time point, is the recurrent recurrence formula.

7. A system, characterized by The program is executed by the processor to implement the steps of the method according to any one of claims 1-5. The program is executed by the processor to implement the steps of the method according to any one of claims 1-5. ​ 8. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images