Efficient multi-target vehicle trajectory prediction method based on multi-scale image convolution

Abstract

The invention provides an efficient multi-target vehicle trajectory prediction method based on multi-scale image convolution, and belongs to the technical field of intelligent traffic and automatic driving. The technical problems that an existing prediction method in a complex traffic environment is insufficient, and especially the efficiency and the precision are insufficient under the multi-target condition are solved. According to the technical scheme, the method comprises the steps that firstly, sub-graphs of vehicles and lanes are constructed, and vehicle tracks and lane features are extracted through a multi-scale graph convolutional network; secondly, dynamically adjusting the weights of different target vehicles by using a self-adaptive dynamic weight module; and then, clustering the vehicle motion modes based on the motion feature library. The method has the advantages that feature extraction and prediction performance is improved by combining a high-definition semantic map and adopting a space-time multi-head attention mechanism and a self-adaptive dynamic weight module, and the method shows performance superior to that of the prior art in multiple traffic scenes and has remarkable practical application potential.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation and autonomous driving technology, and in particular to an efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution. Background Art

[0002] As autonomous driving technology matures, vehicle trajectory prediction plays a vital role in intelligent transportation systems. By accurately predicting the future trajectories of multiple target vehicles, the autonomous driving system can effectively avoid potential dangers, avoid collisions, and improve driving safety and efficiency. At the same time, multi-target vehicle trajectory prediction can help autonomous vehicles make better path planning in complex traffic environments, thereby reducing traffic congestion and improving road capacity.

[0003] At present, most mainstream vehicle trajectory prediction methods are based on single-target prediction, usually using single-scale spatiotemporal feature extraction techniques such as convolutional neural networks (CNN) and recurrent neural networks (RNN). However, multi-target vehicle trajectory prediction in complex traffic scenarios often requires capturing the dynamic interactions between different vehicles, and existing single-target models cannot fully cope with this challenge. Therefore, a more efficient vehicle trajectory prediction method that can capture multi-scale graph convolution is needed. Summary of the invention

[0004] The purpose of the present invention is to provide an efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution, which can effectively handle the complex interaction relationship between vehicles and lane lines, thereby improving the accuracy and real-time performance of target vehicle trajectory prediction.

[0005] The inventive concept of the present invention is: the present invention provides an efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution, which improves the feature extraction and prediction performance by combining high-definition semantic maps, adopting spatiotemporal multi-head attention mechanism and adaptive dynamic weight module. The specific steps include: first, constructing a subgraph of vehicles and lanes, extracting vehicle trajectories and lane features through a multi-scale graph convolution network; second, dynamically adjusting the weights of different target vehicles using an adaptive dynamic weight module; then, clustering vehicle motion patterns based on a motion feature library to more accurately reflect the complexity of traffic scenes. This method has shown performance superior to existing technologies in multiple traffic scenarios and has significant practical application potential.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: an efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution, comprising the following steps:

[0007] S1: Build a high-definition semantic map to obtain the characteristics of vehicles and their surroundings in the traffic scene;

[0008] S2: Use multi-scale graph convolutional networks to extract features from vehicle trajectories and lane subgraphs;

[0009] S3: Modeling the spatiotemporal correlation of vehicle trajectory and lane information through spatiotemporal multi-head attention mechanism;

[0010] S4: Introduce an adaptive dynamic weight module to dynamically adjust the trajectory weights of multiple target vehicles;

[0011] S5: Based on the constructed motion feature library, common vehicle motion patterns are extracted and trajectory prediction is optimized.

[0012] Furthermore, in step S1, a high-definition semantic map is constructed to obtain features of vehicles and their surroundings in the traffic scene as input features of the network.

[0013] Furthermore, the step S2 specifically includes the following steps:

[0014] S3.1: The first channel of the model is the historical trajectory data of the vehicle, and each trajectory is represented by [p 1 ,p 2 ,…,p t ], where p represents the horizontal and vertical coordinates corresponding to the vehicle at time step t. The trajectory input data is processed by the CNN-ResUnit network based on the feature pyramid structure. The one-dimensional convolution in the network is operated in parallel to output a time feature map. Finally, the vehicle trajectory features are represented as a feature matrix;

[0015] S3.2: The second channel of the model is the vectorized lane data. The vectorized lane data is constructed into a spatial graph through a multi-scale graph convolution network. The lanes are covered with nodes and quantized into small black dots. The dotted lines represent lane lines. In order to capture the feature information of the lane subgraph on a larger scale, the basic operator of the multi-scale graph convolution is designed to capture long-term dependencies. The formula is expressed as:

[0016]

[0017] Among them, X is the trajectory input data of the model, Y is the output data of the model, i∈{left,right} indicates that the current lane is the left lane or the right lane, W0 is the initialization weight matrix of the model, and A i and W i are the adjacency matrix and weight matrix corresponding to the i-th connection type, A suc and A pred Propagate the information of predecessor and successor lane nodes, and is the adjacency matrix A suc and A predThe k-th matrix power, c is the expansion coefficient, k c is the size of the cth expansion, W pred,kc represents the weight matrix of the expansion of the cth successor car to the node, W suc,kc Represents the weight matrix of the expansion of the c-th forward vehicle to the node.

[0018] Furthermore, in step S3, the vehicle trajectory and the transmitted node information provided in the previous step are fused through the spatiotemporal multi-head attention mechanism. In order to obtain the spatiotemporal correlation between the vehicle trajectory and the context environment, the spatial multi-head attention and multi-scale graph convolution are used to capture the complete correlation between the vehicle and the lane; wherein the scene is divided into four feature representations, namely, vehicle to lane (AL), lane to lane (LL), lane to vehicle (LA) and vehicle to vehicle (AA). The above four features are used to attach all the correlations between the vehicle and lane nodes. The specific formula of the spatial multi-head attention layer is expressed as:

[0019]

[0020] Among them, MHA(f q ,f k ) is the output value of the multi-head attention mechanism, and the input value f q and f k are the feature representations of query and key respectively. Q is the query matrix, which represents the part of the input data where relevant information needs to be found. K is the key matrix, which represents the part of the data that can match the query. k is the dimension of the key vector, which is used to avoid the gradient disappearance caused by excessive similarity, and V is the value matrix, which is used to calculate the final output value in combination with the attention weights.

[0021] A spatial multi-head attention operation is performed on the output of the previous step to obtain the AL feature tensor map, which introduces real-time traffic information into the lane node. The lane sub-map is processed with a multi-scale graph convolution model, and an attention operation is performed on the output to obtain LL, which propagates traffic information on the lane sub-map to update the lane information features. A spatial attention operation is performed on AL and LL to obtain LA, which transmits the updated map features and real-time traffic information to the vehicle. A spatial attention mechanism is performed on AL, LL, and LA to obtain AA, which transmits information between vehicles and generates vehicle features.

[0022] Furthermore, in step S4, a multi-layer perceptron and learnable weights are used to construct a dynamic adaptive head, and the trajectory weights of different vehicles are dynamically learned, so that the modal module can adapt to the trajectory of each vehicle. The specific formula is as follows:

[0023]

[0024]

[0025]

[0026]

[0027] Among them, f is the AA matrix feature obtained by the fusion module, W d1 and W d2 are trainable weight parameters, which are weight matrices used to perform linear transformations on input features. These weights are constantly adjusted during the model training process. W1 and W2 are calculated intermediate results, respectively. d1 and W d2 It is obtained by matrix multiplication with the input feature f, norm is the normalization layer, ReLU is the activation function, and F d1 Represents the nonlinear transformation result of the input feature f, y pred is the predicted coordinate sequence.

[0028] Furthermore, in step S5, the traffic scenes in the data set are classified into T-junctions, crossroads, straight sections and Y-junctions, and the motion features of the vehicles are clustered using the K-means algorithm. The four scenes and the motion features obtained by clustering are constructed into a vehicle motion feature library. When the model matches the corresponding motion features, the motion features are mapped into a learnable state using a multi-layer perceptron, and the mapped motion features are used as multimodal inputs and input into the last layer of the multi-layer perceptron of the model. The formula is as follows:

[0029]

[0030] Among them, E i represents the output features after the multi-layer perceptron, specifically corresponding to the feature representation of i motion features, n represents n traffic scenes in a large scene, k represents k motion features, d represents different traffic directions, i∈{1,...,k} represents which motion feature is currently being processed, It contains the trajectory prediction information corresponding to each motion feature. Each motion feature can help the model predict a specific trajectory, stabilize the model training process, and make the prediction process more efficient.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention extracts vehicle trajectories and lane features by introducing a multi-scale graph convolutional network, which can capture the complex dynamics of traffic scenes at different spatial and temporal scales. This method can not only model the independent motion mode of each vehicle, but also capture the mutual influence between multiple vehicles, which significantly improves the accuracy of multi-target trajectory prediction, especially in high-density traffic environments.

[0033] 2. The present invention adopts a spatiotemporal multi-head attention mechanism to model the spatiotemporal relationship between vehicles and lanes in traffic scenes. This mechanism can effectively capture the complex interactive relationship between vehicles and between vehicles and lanes, so as to better understand the dynamic changes in traffic scenes and further improve the accuracy of vehicle trajectory prediction. Through the adaptive dynamic weight module, the present invention can dynamically adjust the weights of different target vehicles according to the changes in traffic scenes. This module can adaptively adjust the model's attention to different vehicles according to the motion state of each vehicle, thereby optimizing multi-target prediction and improving the robustness of the system in complex traffic environments.

[0034] 3. The present invention introduces a motion feature library based on actual traffic data to perform clustering modeling on vehicle motion patterns in common traffic scenarios. The feature library can help the model quickly match appropriate motion patterns, effectively reduce computational overhead, and improve the prediction efficiency of the model, especially in large-scale traffic scenarios.

[0035] 4. The present invention not only solves the accuracy and efficiency problems of multi-target vehicle trajectory prediction in complex traffic scenarios, but also has good real-time and robustness, and is widely applicable to autonomous driving, intelligent transportation and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0037] Figure 1 Schematic diagram of the overall architecture of an efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution according to an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of a vectored lane node according to an embodiment of the present invention.

[0039] Figure 3 Schematic diagram of a motion feature library according to an embodiment of the present invention.

[0040] Figure 4 Schematic diagram of vehicle trajectory prediction results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0042] Example 1

[0043] See also Figures 1 to 4The technical solution provided in this embodiment is an efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution. The overall architecture of the method is shown in the attached Figure 1 As shown, the following steps are included:

[0044] S1: Build a high-definition semantic map to obtain the characteristics of vehicles and their surroundings in the traffic scene;

[0045] S2: Use multi-scale graph convolutional networks to extract features from vehicle trajectories and lane subgraphs;

[0046] S3: Modeling the spatiotemporal correlation of vehicle trajectory and lane information through spatiotemporal multi-head attention mechanism;

[0047] S4: Introduce an adaptive dynamic weight module to dynamically adjust the trajectory weights of multiple target vehicles;

[0048] S5: Based on the constructed motion feature library, common vehicle motion patterns are extracted and trajectory prediction is optimized.

[0049] Specifically, in step S1, a high-definition semantic map is constructed to obtain features of vehicles and their surroundings in the traffic scene as input features of the network.

[0050] Specifically, the step S2 includes the following steps:

[0051] S3.1: The first channel of the model is the historical trajectory data of the vehicle, and each trajectory is represented by [p 1 ,p 2 ,…,p t ], where p represents the horizontal and vertical coordinates corresponding to the vehicle at time step t. The trajectory input data is processed by the CNN-ResUnit network based on the feature pyramid structure. The one-dimensional convolution in the network is operated in parallel to output a time feature map. Finally, the vehicle trajectory features are represented as a feature matrix;

[0052] S3.2: The second channel of the model is the vectorized lane data. The vectorized lane data is constructed into a spatial graph through a multi-scale graph convolutional network. The lanes are covered with nodes and quantized into small black dots. The dashed lines represent lane lines, as shown in the attached figure. Figure 2 In order to capture the feature information of the lane sub-graph on a larger scale, a basic operator of multi-scale graph convolution is designed to capture long-term dependencies. The formula is expressed as follows:

[0053]

[0054] Among them, X is the trajectory input data of the model, Y is the output data of the model, i∈{left,right} indicates that the current lane is the left lane or the right lane, W0 is the initialization weight matrix of the model, and A i and W i are the adjacency matrix and weight matrix corresponding to the i-th connection type, A suc and A pred Propagate the information of predecessor and successor lane nodes, and is the adjacency matrix A suc and A pred The k-th matrix power, c is the expansion coefficient, k c is the size of the cth expansion, W pred,kc represents the weight matrix of the expansion of the cth successor car to the node, W suc,kc Represents the weight matrix of the expansion of the c-th forward vehicle to the node.

[0055] Specifically, in step S3, the vehicle trajectory and the node information transmitted provided in the previous step are integrated through the spatiotemporal multi-head attention mechanism. In order to obtain the spatiotemporal correlation between the vehicle trajectory and the context environment, the spatial multi-head attention and multi-scale graph convolution are used to capture the complete correlation between the vehicle and the lane; wherein the scene is divided into four feature representations, namely, vehicle to lane AL, lane to lane LL, lane to vehicle LA and vehicle to vehicle AA. The above four features are used to attach all the correlations between the vehicle and lane nodes. The specific formula of the spatial multi-head attention layer is expressed as:

[0056]

[0057] Among them, MHA(f q ,f k ) is the output value of the multi-head attention mechanism, and the input value f q and f k are the feature representations of query and key respectively. Q is the query matrix, which represents the part of the input data where relevant information needs to be found. K is the key matrix, which represents the part of the data that can match the query. k is the dimension of the key vector, which is used to avoid the gradient disappearance caused by excessive similarity, and V is the value matrix, which is used to calculate the final output value in combination with the attention weights.

[0058] A spatial multi-head attention operation is performed on the output of the previous step to obtain the AL feature tensor map, which introduces real-time traffic information into the lane node. The lane sub-map is processed with a multi-scale graph convolution model, and an attention operation is performed on the output to obtain LL, which propagates traffic information on the lane sub-map to update the lane information features. A spatial attention operation is performed on AL and LL to obtain LA, which transmits the updated map features and real-time traffic information to the vehicle. A spatial attention mechanism is performed on AL, LL, and LA to obtain AA, which transmits information between vehicles and generates vehicle features.

[0059] Specifically, in step S4, a multi-layer perceptron and learnable weights are used to construct a dynamic adaptive head, and the trajectory weights of different vehicles are dynamically learned, so that the modal module can adapt to the trajectory of each vehicle. The specific formula is as follows:

[0060]

[0061]

[0062]

[0063]

[0064] Among them, f is the AA matrix feature obtained by the fusion module, W d1 and W d2 are trainable weight parameters, which are weight matrices used to perform linear transformations on input features. These weights are constantly adjusted during the model training process. W1 and W2 are calculated intermediate results, respectively. d1 and W d2 It is obtained by matrix multiplication with the input feature f, norm is the normalization layer, ReLU is the activation function, and F d1 Represents the nonlinear transformation result of the input feature f, y pred is the predicted coordinate sequence.

[0065] Specifically, in step S5, the traffic scenes in the data set are classified into T-shaped intersections, crossroads, straight sections and Y-shaped intersections, and the motion features of the vehicles are clustered using the K-means algorithm. The four scenes and the motion features obtained by clustering are constructed into a vehicle motion feature library, as shown in the attached figure. Figure 3 As shown in the figure, when the model matches the corresponding motion features, the multi-layer perceptron is used to map the motion features into a learnable state, and the mapped motion features are used as multimodal inputs to the last layer of the multi-layer perceptron in the model. The formula is as follows:

[0066]

[0067] Among them, Ei represents the output features after the multi-layer perceptron, specifically corresponding to the feature representation of i motion features, n represents n traffic scenes in a large scene, k represents k motion features, d represents different traffic directions, i∈{1,...,k} represents which motion feature is currently being processed, It contains the trajectory prediction information corresponding to each motion feature. Each motion feature can help the model predict a specific trajectory, stabilize the model training process, and make the prediction process more efficient. The results are shown in the attached Figure 4 As shown in the figure, the red dot represents the vehicle, the blue dot represents the actual trajectory of the vehicle, the green and red trajectories are the 6 predicted trajectory data, and the red trajectory is the trajectory with the highest probability.

[0068] Example 2

[0069] Based on Example 1, in order to verify the performance of the model, this example selected AutoBot, D-TNT, HiVT and PAGA models as comparative experimental models. Using input data constructed with a single target vehicle as the center of the scene, the proposed model was verified on the latest training set and test set of Argoverse. Official indicator brier-mFDE k The performance of the most advanced PAGA model is close to that of the HiVT model, and the number of parameters is only 60% of that of the HiVT model. It also achieves high performance in other indicators. On the validation set, when the number of predicted trajectories is 6, the model method proposed in this embodiment performs better than the mADE k and mFDE k In terms of indicators, it performs best, and the results are shown in Table 1 below.

[0070] Table 1 Single target trajectory prediction performance index table of Argoverse dataset

[0071]

[0072] Example 3

[0073] Based on Example 1, for the multi-target vehicle trajectory prediction on Argoverse, this example constructs input data with the center of the scene as the origin. The scenes with less than five single targets in Argoverse are grouped into a synthetic multi-target scene to obtain a sub-dataset for training. In order to compare the specific differences between different methods, the batchsize is adjusted to 1. The model proposed in this example is verified on the test subset of Argoverse. In the official indicator mFDE, the performance of the model proposed in this example ranks second, and the experimental results are shown in Table 2.

[0074] Table 2 Multi-target trajectory prediction performance indicators of Argoverse dataset

[0075]

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient multi-target vehicle trajectory prediction method based on multi-scale graph convolution, characterized in that: The following steps are involved: S1: Build a high-definition semantic map to obtain the characteristics of vehicles and their surroundings in the traffic scene; S2: Use multi-scale graph convolutional networks to extract features from vehicle trajectories and lane subgraphs; S3: Modeling the spatiotemporal correlation of vehicle trajectory and lane information through spatiotemporal multi-head attention mechanism; S4: Introduce an adaptive dynamic weight module to dynamically adjust the trajectory weights of multiple target vehicles; S5: Based on the constructed motion feature library, common vehicle motion patterns are extracted and trajectory prediction is optimized.

2. The multi-target vehicle trajectory prediction method based on multi-scale graph convolution according to claim 1 is characterized in that: In step S1, a high-definition semantic map is constructed to obtain features of vehicles in the traffic scene and their surroundings as input features of the network.

3. The multi-target vehicle trajectory prediction method based on multi-scale graph convolution according to claim 1 is characterized in that: The step S2 includes the following steps: S3.1: The first channel of the model is the historical trajectory data of the vehicle, and each trajectory is represented by [p 1 ,p 2 ,…,p t ], where p represents the horizontal and vertical coordinates corresponding to the vehicle at time step t. The trajectory input data is processed by the CNN-ResUnit network based on the feature pyramid structure. The one-dimensional convolution in the network is operated in parallel to output a time feature map. Finally, the vehicle trajectory features are represented as a feature matrix; S3.2: The second channel of the model is the vectorized lane data. The vectorized lane data is constructed into a spatial graph through a multi-scale graph convolution network. The lanes are covered with nodes and quantized into small black dots. The dotted lines represent lane lines. In order to capture the feature information of the lane subgraph on a larger scale, the basic operator of the multi-scale graph convolution is designed to capture long-term dependencies. The formula is expressed as: Among them, X is the trajectory input data of the model, Y is the output data of the model, i∈{left,right} indicates that the current lane is the left lane or the right lane, W0 is the initialization weight matrix of the model, and A i and W i are the adjacency matrix and weight matrix corresponding to the i-th connection type, A suc and A pred Propagate the information of predecessor and successor lane nodes, and is the adjacency matrix A suc and A pred The k-th matrix power, c is the expansion coefficient, k c is the size of the cth expansion, W pred,kc represents the weight matrix of the expansion of the cth successor car to the node, W suc,kc Represents the weight matrix of the expansion of the c-th predecessor to the node.

4. The multi-target vehicle trajectory prediction method based on multi-scale graph convolution according to claim 1 is characterized in that: In step S3, the vehicle trajectory and the node information provided in the previous step are integrated through the spatiotemporal multi-head attention mechanism. In order to obtain the spatiotemporal correlation between the vehicle trajectory and the context environment, the spatial multi-head attention and multi-scale graph convolution are used to capture the complete correlation between the vehicle and the lane; wherein, the scene is divided into four feature representations, namely, vehicle to lane AL, lane to lane LL, lane to vehicle LA and vehicle to vehicle AA. The above four features are used to attach all the correlations between the vehicle and lane nodes. The formula of the spatial multi-head attention layer is expressed as: Among them, MHA(f q ,f k ) is the output value of the multi-head attention mechanism, and the input value f q and f k are the feature representations of query and key respectively. Q is the query matrix, which represents the part of relevant information that needs to be found in the input data. K is the key matrix, which represents the part of information in the data that matches the query. k is the dimension of the key vector, which is used to avoid the gradient vanishing due to excessive similarity, and V is the value matrix, which is used to calculate the final output value in combination with the attention weights; A spatial multi-head attention operation is performed on the output of the previous step to obtain the AL feature tensor map, which introduces real-time traffic information into the lane node. The lane sub-map is processed with a multi-scale graph convolution model, and an attention operation is performed on the output to obtain LL, which propagates traffic information on the lane sub-map to update the lane information features. A spatial attention operation is performed on AL and LL to obtain LA, which transmits the updated map features and real-time traffic information to the vehicle. A spatial attention mechanism is performed on AL, LL, and LA to obtain AA, which transmits information between vehicles and generates vehicle features.

5. The multi-target vehicle trajectory prediction method based on multi-scale graph convolution according to claim 1 is characterized in that: In step S4, a multi-layer perceptron and learnable weights are used to construct a dynamic adaptive head, and the trajectory weights of different vehicles are dynamically learned, so that the modal module can adapt to the trajectory of each vehicle. The formula is as follows: Among them, f is the AA matrix feature obtained by the fusion module, W d1 and W d2 is a trainable weight parameter, which is a weight matrix used to linearly transform the input features. These weights will be adjusted continuously during the training process of the model. W1 and W2 are the intermediate results of the calculation, respectively. d1 and W d2 It is obtained by matrix multiplication with the input feature f, norm is the normalization layer, ReLU is the activation function, and F d1 Represents the nonlinear transformation result of the input feature f, y pred is the predicted coordinate sequence.

6. The multi-target vehicle trajectory prediction method based on multi-scale graph convolution according to claim 1 is characterized in that: In step S5, the traffic scenes in the data set are classified into T-shaped intersections, crossroads, straight sections and Y-shaped intersections, and the motion features of the vehicles are clustered using the K-means algorithm. The four scenes and the motion features obtained by clustering are constructed into a vehicle motion feature library. When the model matches the corresponding motion features, the motion features are mapped into a learnable state using a multi-layer perceptron, and the mapped motion features are used as multi-modal inputs and input into the last layer of the multi-layer perceptron of the model. The formula is as follows: Among them, E i represents the output features after the multi-layer perceptron, corresponding to the feature representation of i motion features, n represents that there are n traffic scenes in the large scene, k represents k motion features, d represents different traffic directions, i∈{1,...,k} represents which motion feature is currently being processed, It contains the trajectory prediction information corresponding to each motion feature. Each motion feature can help the model predict a specific trajectory, stabilize the model training process, and make the prediction process more efficient.

Images