A method and system for autonomous driving trajectory prediction based on streaming Transformer

Abstract

The present invention discloses a method for predicting autonomous driving trajectories based on a streaming Transformer, comprising: obtaining independent scene data of an autonomous driving vehicle, and preprocessing the independent scene data to obtain preprocessed continuous scene data, and inputting the preprocessed continuous scene data into a pre-trained streaming Transformer trajectory prediction model to obtain a predicted future trajectory. The present invention can solve the technical problems that the existing trajectory prediction methods can only effectively utilize information in local windows, which conflicts with actual applications, resulting in a large difference between the model and the deployment; the technical problems that the model has a weak ability to cope with emergencies due to the lack of construction of a time-series scene context relationship; and the technical problems that the trajectory prediction of the model is not accurate enough and has mutations due to the lack of attention to the consistency of trajectory prediction between time series.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and sequence prediction, and more specifically, to a method and system for autonomous driving trajectory prediction based on streaming Transformer. Background Art

[0002] Trajectory prediction for traffic participants plays a vital role in the field of autonomous driving research. Accurate trajectory prediction has a key impact on downstream vehicle path planning tasks. Therefore, trajectory prediction research is an important part of deep learning in the field of autonomous driving, showing great research potential and application value.

[0003] Existing trajectory prediction methods mainly include methods focusing on scene representation, methods focusing on scene modeling, and methods focusing on precise prediction. The methods focusing on scene representation mainly use vectorized representation, which effectively represents the structure of the high-precision map and the motion status of traffic participants in the scene, thereby greatly facilitating the learning and modeling of the model. The methods focusing on scene modeling mostly use deep learning methods based on the Transformer model. Such methods can not only comprehensively model the internal features of high-precision maps and traffic participants, but also efficiently learn the spatial relationship between scene elements, thereby ensuring that the model gives accurate predicted trajectories. The methods focusing on precise prediction further optimize the preliminary predicted trajectory through pre-training, additional fine-tuning modules or post-processing methods to obtain more accurate prediction results.

[0004] However, existing trajectory prediction methods still have some defects that cannot be ignored: First, limited by the existing training data sets of independent scenes, such methods can only effectively utilize the information in local windows, which conflicts with actual applications and leads to large differences between models and deployments; Second, since such methods ignore the connection between the context of time series scenes, they limit the model's perception of invisible information in the current scene, resulting in a weaker ability of the model to cope with emergencies; Third, since such methods can only predict each scene independently and ignore the consistency of trajectory predictions between time series, the trajectory prediction will be inaccurate and there will be problems with mutations. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a highly automated driving trajectory prediction method and system based on a streaming Transformer, which aims to solve the technical problems that the existing trajectory prediction method can only effectively utilize the information in the local window, which conflicts with the actual application, thus resulting in a large difference between the model and the deployment; the technical problem that the model has a weak ability to cope with emergencies due to the lack of construction of the scene context relationship of the time series; and the technical problem that the trajectory prediction of the model is not accurate enough and there are mutations due to the lack of attention to the consistency of trajectory prediction between time series.

[0006] To achieve the above object, according to one aspect of the present invention, a method for autonomous driving trajectory prediction based on streaming Transformer is provided, comprising the following steps:

[0007] (1) Obtain independent scene data of the autonomous driving vehicle, and preprocess the independent scene data to obtain preprocessed continuous scene data.

[0008] (2) The continuous scene data preprocessed in step (1) is input into the pre-trained streaming Transformer trajectory prediction model to obtain the predicted future trajectory.

[0009] Preferably, step (1) is specifically as follows: first, independent scene data of each autonomous driving vehicle is obtained, which includes historical trajectories and high-precision maps of traffic participants; then, multiple time segmentation points are selected according to the historical trajectories of traffic participants in the independent scene data; and a section of trajectory is selected from each time segmentation point on the historical trajectory as the future predicted trajectory, and a section of trajectory is selected from the time segmentation point as the known historical trajectory; the future predicted trajectories and the known historical trajectories corresponding to all time segmentation points constitute a continuous trajectory segment; then, the high-precision map in the independent scene data is aggregated according to each time segmentation point, and the obtained processing results and the continuous trajectory segment together constitute the continuous scene data.

[0010] Preferably, the streaming Transformer trajectory prediction model includes a scene encoder and a future trajectory decoder for single processing, as well as a scene context stream and a predicted trajectory stream for cross-scene feature interaction.

[0011] The scene encoder includes the map encoder, history track encoder and interaction encoder.

[0012] The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N mRepresents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally and globally, and the output dimension is N m ×C map feature tensor F m , where C represents the dimension of the map feature tensor;

[0013] The history track encoder uses a multi-layer one-dimensional neighbor attention module, whose input is a dimension of N a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps in the historical trajectory, and feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectories are processed by attention weighted encoding, and the output is a N-dimensional a ×C object feature tensor F a ;

[0014] The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene feature tensor F s .

[0015] Preferably, the input of the future trajectory decoder is the feature tensor F from the current scene s The extracted dimension is N fa ×C’s traffic participant feature tensor F fa , the future trajectory encoder focuses on the traffic participant feature tensor F fa Perform decoding operation, and the final output is the multimodal prediction trajectory Y, the corresponding probability P and its corresponding feature F Y , the dimension of the multimodal prediction trajectory Y is N fa ×N m ×T×2, where N fa Represents the number of traffic participants, N m represents the number of predicted trajectory modes, T is the number of steps in the predicted future trajectory, and the feature dimension 2 refers to the coordinate position.

[0016] The input of the scene context information flow is the current scene feature tensor F output by the interaction encoder s , and the acquired historical scene feature tensor F s ′, which uses the decomposition attention module to process the two, and finally outputs the updated current scene feature tensor. The decomposition attention module includes a map-map attention submodule and an object-scene attention submodule;

[0017] The predicted trajectory information flow is placed after the future trajectory decoder to connect the historical prediction with the current prediction. Its input is the multimodal predicted trajectory Y output by the future trajectory decoder and its corresponding feature F Y , the two are processed through the cross attention module, and the final output is the updated multimodal prediction trajectory Y r .

[0018] Preferably, the streaming Transformer trajectory prediction model is trained by the following steps:

[0019] (2-1) Obtain an autonomous driving dataset for trajectory prediction, preprocess the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and use the vectorization method in the VectorNet model to convert the preprocessed autonomous driving dataset into a vector format to obtain a format-converted autonomous driving dataset, and divide the format-converted autonomous driving dataset into a training set and a validation set at a ratio of 9:1, wherein the autonomous driving dataset includes historical and future trajectories of traffic participants and high-precision maps.

[0020] (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M (whose dimension is N) in the autonomous driving scene segment is m ×L×2) and the historical trajectory vector A of the traffic participants (whose dimension is N a ×H×4) input the map encoder, history trajectory encoder, and interaction encoder in the streaming Transformer trajectory prediction model to obtain N dimensions respectively. m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and the dimension is (N a +N m )×C scene feature tensor F s .

[0021] (2-3) The scene context information flow uses a decomposed attention module (which includes a map-map attention submodule and an object-scene attention submodule) to calculate the current scene feature tensor F sAnd the obtained historical scene feature tensor F s ’ is processed to obtain the updated current scene feature tensor.

[0022] (2-4) Extract the dimension N from the updated current scene feature tensor obtained in step (2-3) fa ×C is the feature tensor F of the traffic participant of interest fa , and input it into the future trajectory decoder to obtain the multimodal trajectory Y, the corresponding probability P and its corresponding feature F Y .

[0023] (2-5) The multimodal prediction trajectory Y obtained in step (2-4) and its corresponding feature F Y Input the predicted trajectory information stream to obtain the updated multimodal predicted trajectory Y r It should be noted that a memory trajectory library is maintained in the prediction trajectory information flow to store and record the historical multimodal prediction trajectory Y' and the corresponding features F Y′ .

[0024] (2-6) The updated multimodal predicted trajectory Y obtained in step (2-5) r The loss function is calculated using the multimodal predicted trajectory Y and its corresponding probability P obtained in step (2-4), and the streaming Transformer trajectory prediction model is iteratively trained using the loss function until the streaming Transformer trajectory prediction model converges, thereby obtaining a trained streaming Transformer trajectory prediction model.

[0025] (2-7) Use the verification set obtained in step (2-1) to verify the streaming Transformer trajectory prediction model preliminarily trained in step (2-6) to obtain the final trained streaming Transformer trajectory prediction model.

[0026] Preferably, the autonomous driving datasets used in step (2-1) are Argoverse 1 and Argoverse 2 datasets, the former of which is 323,557 autonomous driving scene clips collected in Miami and Pittsburgh, each of which contains 3 seconds of known historical trajectory, 2 seconds of future predicted trajectory and high-precision map, while the latter is a total of 250,000 autonomous driving scene clips collected in up to 6 cities, each of which contains 5 seconds of known historical trajectory, 6 seconds of future predicted trajectory and high-precision map. Both datasets use a sampling frequency of 10 Hz and only use high-precision maps and traffic participants within a radius of 150 meters for each scene.

[0027] The map encoder in step (2-2) adopts the PointNet network. Specifically, step (2-2) first maps the high-precision map vector M into a high-dimensional feature through a multi-layer perceptron. Then, the high-precision map vector M is processed by a maximum pooling operation to obtain the global feature corresponding to the high-precision map vector M. Then, the global feature is connected to each point feature in the high-precision map vector M to obtain a point feature containing global information. Then, the above mapping operation and maximum pooling operation are repeated for all point features to obtain a high-dimensional feature with a dimension of N. m ×C map feature tensor F m The object encoder uses a neighbor attention network, which compresses the historical trajectory vector A of the traffic participants in the autonomous driving scene fragment in time series and increases the dimension of the features through multi-layer stacked neighbor attention modules to obtain a dimension of N a ×C object feature tensor F a , and the object feature tensor F a With the map feature tensor F m The connection operation is performed to obtain a preliminary scene feature tensor. Subsequently, the interactive encoder uses a standard Transformer encoder to process the preliminary scene feature tensor to obtain a scene feature tensor F s .

[0028] Preferably, step (2-3) is specifically as follows: the decomposition attention module first performs an operation on the current scene feature tensor F s And the obtained historical scene feature tensor F s ′ is split to obtain the current object feature tensor, the current map feature tensor, the historical object feature tensor, and the historical map feature tensor respectively. Afterwards, the map-map attention module in the decomposition attention module takes the current map feature tensor and the historical map feature tensor as input, the current map feature tensor as the query feature, and the historical map feature tensor as the key feature to perform attention operation processing to obtain the updated current map feature tensor; at the same time, the object-scene attention module takes the current object feature tensor and the historical scene feature tensor as input, the current object feature tensor as the query feature, and the historical scene feature as the key feature to perform attention operation processing to obtain the updated current scene feature tensor; finally, the decomposition attention module connects the updated current object feature tensor with the map feature tensor to obtain the updated current scene feature tensor.

[0029] Preferably, step (2-5) is specifically as follows: first, the predicted trajectory information flow first obtains the historical multimodal predicted trajectory Y' and the corresponding feature F stored in the memory trajectory library in the predicted trajectory information flow Y′, and the historical multimodal prediction trajectory Y' is subjected to rotation and translation transformation to obtain the transformed historical multimodal prediction trajectory as follows:

[0030]

[0031] Where R(θ) and y represent the rotation matrix and trajectory center corresponding to the historical multimodal prediction trajectory Y', respectively.

[0032] Then, the prediction trajectory information flow combines the multimodal prediction trajectory Y and its corresponding features F Y As query features, the transformed multimodal prediction trajectory and its corresponding features are input into the attention module as key-value features to obtain the updated features F Yr :

[0033]

[0034] Where MHA represents the attention operation, Q, K, and V represent the query features, key features, and value features used in the operation, respectively, and TE represents the trajectory encoding operation.

[0035] Finally, the updated feature F Yr Input into the multi-layer perceptron module to obtain the updated multimodal prediction trajectory Y r And as the final network output.

[0036] Preferably, the loss function Loss used in step (2-6) is:

[0037] Loss=Loss reg +Loss cls +Loss refine

[0038] in:

[0039] Loss reg =L smoothL1 (Y,Y)

[0040] Loss cls =L CE (P,P)

[0041] Loss refine =L smoothL1 (Y r ,Y)

[0042] Where L smoothL1 represents the SMOOTH-L1 function, L CE represents the cross entropy function, Y represents the true value of the multimodal prediction trajectory, and P represents the true value of the probability corresponding to the multimodal prediction trajectory.

[0043] According to another aspect of the present invention, a method for predicting an autonomous driving trajectory based on a streaming Transformer is provided, comprising:

[0044] The first module is used to obtain independent scene data of the autonomous driving vehicle and preprocess the independent scene data to obtain preprocessed continuous scene data.

[0045] The second module is used to input the continuous scene data preprocessed by the first module into the pre-trained streaming Transformer trajectory prediction model to obtain the predicted future trajectory.

[0046] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0047] (1) Since the present invention adopts step (1) in the streaming Transformer trajectory prediction model, it reorganizes the existing data set, thereby transforming the opposing driving scenes into continuous driving sub-scenes, so that it can simulate the real driving situation to a certain extent, thereby alleviating the technical problem of suboptimal and inefficient existing methods in real application scenarios;

[0048] (2) Since the present invention adopts steps (2-3) in the training process of the streaming Transformer trajectory prediction model, it realizes the temporal interaction between the current scene features and the historical scene features through the scene context information flow, and makes better use of the historical information, thereby effectively expanding the perception range and predicting some currently invisible situations. Therefore, it can solve the technical problems that the existing methods cannot use the historical scene context and are difficult to deal with possible emergencies;

[0049] (3) Since the present invention adopts steps (2-5) in the training process of the streaming Transformer trajectory prediction model, it uses the trajectory prediction information flow and the historical trajectory to constrain and optimize the current prediction, thereby effectively improving the accuracy of the current trajectory, thereby solving the technical problem that the current predicted trajectory is not accurate enough and has mutations due to the inconsistency of the existing time series trajectory;

[0050] (4) Since the present invention adopts steps (2-5) in the training process of the streaming Transformer trajectory prediction model, it retains historical trajectories by adopting a trajectory encoding attention module and a memory trajectory library, thereby explicitly introducing restrictions on trajectory temporal consistency and simplifying the prediction difficulty of the current scene by using the historical motion paradigm. Therefore, it can enhance the generalization ability of the model for sudden motion situations, thereby improving the accuracy of trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the automatic driving trajectory prediction method based on streaming Transformer of the present invention;

[0052] Figure 2 is a detailed schematic diagram of step (1) in the method of the present invention;

[0053] Figure 3 It is a structural diagram of the streaming Transformer trajectory prediction model of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] In view of the problems of independent scene processing and neglect of cross-scene temporal relationships in existing autonomous driving trajectory prediction methods, the present invention proposes an autonomous driving trajectory prediction method based on streaming Transformer. The two information streams proposed in the present invention, namely the scene context information stream and the predicted trajectory information stream, are used to realize cross-scene information transmission, and the Transformer module is used in the information stream for efficient temporal information interaction. Among them, the scene context information stream is used to realize the interaction of scene features in time series, thereby enriching the receptive field range of the current scene and helping to deal with possible unknown occlusion problems; while the predicted trajectory information stream maintains several predicted trajectory repositories for interaction between current predictions and historical predictions, thereby optimizing the prediction accuracy of the current trajectory and ensuring the consistency of time series predictions. The above designs can quickly and effectively model the scene relationship of time series and significantly improve the trajectory prediction performance.

[0056] like Figure 1 As shown, the present invention provides an automatic driving trajectory prediction method based on streaming Transformer, comprising the following steps:

[0057] (1) Obtain independent scene data of the autonomous driving vehicle and preprocess the independent scene data to obtain preprocessed continuous scene data (used to simulate the continuity and dynamics of driving scenes in the real world).

[0058] like Figure 2As shown, this step is specifically as follows: first, the independent scene data of each autonomous driving vehicle is obtained (which includes the historical trajectory and high-precision map of the traffic participants), and then, multiple time division points are selected according to the historical trajectory of the traffic participants in the independent scene data (the number of the time division points can be adjusted manually according to the actual situation), and a section of the trajectory from each time division point forward is selected on the historical trajectory as the future predicted trajectory, and a section of the trajectory from the time division point backward is selected as the known historical trajectory, and the future predicted trajectories and the known historical trajectories corresponding to all time division points constitute a continuous trajectory segment; then, the high-precision map in the independent scene data is aggregated according to each time division point, and the obtained processing results and the continuous trajectory segments together constitute continuous scene data (the continuous scene data not only contains the historical and future trajectories of the traffic participants, but also includes the spatiotemporal information of their surrounding environment and other traffic elements, so the reorganized continuous scene can simulate the real continuous traffic scene to a certain extent).

[0059] This data set reorganization not only improves the spatiotemporal continuity of the scene, but also allows the existing model to fully utilize the rich information in the original data. While keeping the existing model architecture unchanged, better trajectory prediction performance can be achieved only through this new data structure. Therefore, the innovative data set reorganization method proposed in the present invention can significantly enhance the correlation of temporal and spatial information at the scene level while maintaining the accuracy of the original data, thereby providing more comprehensive and coherent data support for driving scene prediction and optimization models, and promoting the development of related technical fields.

[0060] The advantage of this step is that it transforms the existing independent scene data set into a continuous form, thereby simulating the real application scenario to a certain extent.

[0061] (2) The continuous scene data preprocessed in step (1) is input into the pre-trained streaming Transformer trajectory prediction model to obtain the predicted future trajectory.

[0062] like Figure 3 As shown, the streaming Transformer trajectory prediction model of the present invention includes a scene encoder and a future trajectory decoder for single processing, as well as a scene context stream and a predicted trajectory stream for cross-scene feature interaction.

[0063] The scene encoder includes the map encoder, history track encoder and interaction encoder.

[0064] The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N mRepresents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally (Local) and globally (Global) in sequence, and the output dimension is N m ×C map feature tensor F m , where C represents the dimension of the map feature tensor;

[0065] The history track encoder uses a multi-layer one-dimensional neighbor attention module, whose input is a dimension of N a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps in the historical trajectory, and feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectories are processed by attention weighted encoding, and the output is a N-dimensional a ×C object feature tensor F a ;

[0066] The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene feature tensor F s .

[0067] The input of the future trajectory decoder is the feature tensor F from the current scene s The extracted dimension is N fa ×C’s focused agents feature tensor F fa , the future trajectory encoder focuses on the traffic participant feature tensor F fa Perform decoding operation, and the final output is the multimodal prediction trajectory Y, the corresponding probability P and its corresponding feature F Y , the dimension of the multimodal prediction trajectory Y is N fa ×N m ×T×2, where N fa Represents the number of traffic participants, N m represents the number of predicted trajectory modes, T is the number of steps in the predicted future trajectory, and the feature dimension 2 refers to the coordinate position.

[0068] The scene context information stream is placed after the interaction encoder to realize the temporal scene feature interaction. The input of the scene context information stream is the current scene feature tensor F output by the interaction encoder. s , and the acquired historical scene feature tensor F s ′, which uses a decomposition attention module (which includes a map-map attention submodule and an object-scene attention submodule) to process the two, and finally outputs the updated current scene feature tensor.

[0069] The predicted trajectory information flow is placed after the future trajectory decoder to connect the historical prediction with the current prediction. Its input is the multimodal predicted trajectory Y output by the future trajectory decoder and its corresponding feature F Y , the two are processed through the cross attention module, and the final output is the updated multimodal prediction trajectory Y r .

[0070] Considering that the future trajectories at different moments may have great differences as the movement proceeds, in order to measure the correlation between such trajectories, the present invention uses a cross-attention module based on trajectory embedding (TE), which can effectively associate trajectories with consistent motion mode directions and distinguish trajectories with large differences in motion modes, thereby relying on historical trajectories to simplify the current prediction difficulty and ensure the consistency of trajectories between time series.

[0071] Specifically, the streaming Transformer trajectory prediction model of the present invention is trained by the following steps:

[0072] (2-1) Obtain an autonomous driving dataset for trajectory prediction (which includes historical and future trajectories of traffic participants and high-precision maps), preprocess the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and use the vectorization method in the VectorNet model to convert the preprocessed autonomous driving dataset into a vector format to obtain a format-converted autonomous driving dataset, and divide the format-converted autonomous driving dataset into a training set and a validation set in a ratio of 9:1.

[0073] Specifically, the autonomous driving datasets used in this step are Argoverse 1 and Argoverse 2 datasets. The former is 323,557 autonomous driving scene clips collected in Miami and Pittsburgh, each of which contains 3 seconds of known historical trajectory, 2 seconds of future predicted trajectory and high-precision map, while the latter is a total of 250,000 autonomous driving scene clips collected in up to 6 cities, each of which contains 5 seconds of known historical trajectory, 6 seconds of future predicted trajectory and high-precision map. Both datasets use a sampling frequency of 10 Hz and only use high-precision maps and traffic participants with a radius of 150 meters for each scene.

[0074] (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M (whose dimension is N) in the autonomous driving scene segment is m ×L×2) and the historical trajectory vector A of the traffic participants (whose dimension is N a ×H×4) input the map encoder, history trajectory encoder, and interaction encoder in the streaming Transformer trajectory prediction model to obtain N dimensions respectively. m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and the dimension is (N a +N m )×C scene feature tensor F s .

[0075] Specifically, the map encoder uses the PointNet network, which first maps the HD map vector M into high-dimensional features through a multi-layer perceptron. Then, the HD map vector M is processed by the maximum pooling operation to obtain the global features corresponding to the HD map vector M. Then, the global features are connected to each point feature in the HD map vector M to obtain point features containing global information. Then, the above mapping operation and maximum pooling operation are repeated for all point features to obtain a high-dimensional feature with a dimension of N. m ×C map feature tensor F m The object encoder uses a neighbor attention network, which compresses the historical trajectory vector A of the traffic participants in the autonomous driving scene fragment in time sequence and increases the dimension of the feature through multi-layer stacked neighbor attention modules (similar to convolution modules, which can realize attention modeling in a certain local area) to obtain a dimension N a ×C object feature tensor F a , and the object feature tensor F a With the map feature tensor F mThe connection operation is performed to obtain a preliminary scene feature tensor. Subsequently, the interactive encoder uses a standard Transformer encoder to process the preliminary scene feature tensor to obtain a scene feature tensor F s .

[0076] (2-3) The scene context information flow uses a decomposed attention module (which includes a map-map attention submodule and an object-scene attention submodule) to calculate the current scene feature tensor F s And the obtained historical scene feature tensor F s ’ is processed to obtain the updated current scene feature tensor.

[0077] Specifically, the decomposition attention module first decomposes the current scene feature tensor F s And the obtained historical scene feature tensor F s ′ is split and processed to obtain the current object feature tensor, the current map feature tensor, the historical object feature tensor, and the historical map feature tensor respectively. Afterwards, the map-map attention module in the decomposition attention module takes the current map feature tensor and the historical map feature tensor as input, the current map feature tensor as the query feature, and the historical map feature tensor as the key feature to perform attention operation processing to obtain the updated current map feature tensor (this operation expands the map range from the feature level); at the same time, the object-scene attention module takes the current object feature tensor and the historical scene feature tensor as input, the current object feature tensor as the query feature, and the historical scene feature as the key feature to perform attention operation processing to obtain the updated current scene feature tensor (this operation enables traffic participants to effectively perceive information outside the field of vision); finally, the decomposition attention module connects the updated current object feature tensor with the map feature tensor to obtain the updated current scene feature tensor.

[0078] The updated scene feature tensor is used to output to two subsequent branches, namely, the decoder branch is used to decode the multimodal prediction trajectory and the scene information stream as the historical scene feature of the next frame. In addition, for the historical features input into the above-mentioned attention module, the present invention also introduces Motion Layer Normalization for processing (this operation explicitly measures the differences in time and motion state of the feature tensors at different moments).

[0079] The advantage of this step is that it models the scene context relationship between time sequences, thereby expanding the perception scope of the current scene context and enhancing the model's ability to deal with emergencies.

[0080] (2-4) Extract the dimension N from the updated current scene feature tensor obtained in step (2-3) fa×C is the feature tensor F of the traffic participant of interest fa , and input it into the future trajectory decoder to obtain the multimodal trajectory Y, the corresponding probability P and its corresponding feature F Y .

[0081] (2-5) The multimodal prediction trajectory Y obtained in step (2-4) and its corresponding feature F Y Input the predicted trajectory information stream (which relies on the historical trajectory to update the current trajectory prediction) to obtain the updated multimodal predicted trajectory Y r It should be noted that a memory trajectory library is maintained in the prediction trajectory information flow to store and record the historical multimodal prediction trajectory Y' and the corresponding features F Y′ .

[0082] Specifically, this step first predicts the trajectory information flow and first obtains the historical multimodal predicted trajectory Y' and the corresponding feature F stored in the memory trajectory library in the predicted trajectory information flow Y′ , and the historical multimodal prediction trajectory Y' is subjected to rotation and translation transformation to obtain the transformed historical multimodal prediction trajectory as follows:

[0083]

[0084] Where R(θ) and y represent the rotation matrix and trajectory center corresponding to the historical multimodal prediction trajectory Y', respectively.

[0085] Then, the prediction trajectory information flow combines the multimodal prediction trajectory Y and its corresponding features F Y As query features, the transformed multimodal prediction trajectory and its corresponding features are input into the attention module as key-value features to obtain the updated features F Yr :

[0086]

[0087] Among them, MHA represents the attention operation (Q, K, and V represent the query features, key features, and value features used in the operation, respectively), and TE (Trajectory Embedding) represents the trajectory encoding operation.

[0088] Finally, the updated feature F Yr Input into the multi-layer perceptron module to obtain the updated multimodal prediction trajectory Y r And as the final network output, and use the updated multimodal prediction trajectory Y rThe memory trajectory library is updated with its features, and the multimodal prediction trajectories and their features that are far apart in time (the present invention uses a 3-frame interval) are deleted according to the first-in-first-out principle, thereby completing the flow update process of the current scene.

[0089] The advantage of this step is that it models the predicted trajectory relationship between time series, thereby improving the accuracy of trajectory prediction and effectively solving the problem of trajectory mutation.

[0090] (2-6) The updated multimodal predicted trajectory Y obtained in step (2-5) r The loss function is calculated using the multimodal predicted trajectory Y and its corresponding probability P obtained in step (2-4), and the streaming Transformer trajectory prediction model is iteratively trained using the loss function until the streaming Transformer trajectory prediction model converges, thereby obtaining a preliminarily trained streaming Transformer trajectory prediction model.

[0091] Among them, the loss function includes Loss for supervising the multimodal prediction trajectory Y reg , which uses the SMOOTH-L1 function. In addition, for the probability P corresponding to the multimodal prediction trajectory Y, the cross entropy function is used as the supervision Loss cls , for the updated multimodal prediction trajectory Y r The same supervision method as the initial multimodal prediction trajectory is still used, which can be expressed as:

[0092]

[0093]

[0094]

[0095] Therefore, the total loss function Loss is:

[0096] Loss=Loss reg +Loss cls +Loss refine

[0097] Where L smoothL1 represents the SMOOTH-L1 function, L CE represents the cross entropy function, represents the true value of the multimodal prediction trajectory, Represents the true value of the probability corresponding to the multimodal predicted trajectory.

[0098] (2-7) Use the verification set obtained in step (2-1) to verify the streaming Transformer trajectory prediction model preliminarily trained in step (2-6) to obtain the final trained streaming Transformer trajectory prediction model.

[0099] Test Results

[0100] In order to illustrate the effectiveness of the method of the present invention and the improvement of trajectory prediction accuracy, comparative experiments were conducted with several mainstream network models on the Argoverse 1 and Argoverse 2 autonomous driving datasets. The evaluation indicators involved in the test process of this method include: (1) Minimum Average Displacement Error (minADE): This value evaluates the error distance between the predicted trajectory and the actual trajectory. The smaller the value, the better the method effect; (2) Minimum Final Displacement Error (minFDE): This value evaluates the error distance between the predicted trajectory and the actual trajectory. The smaller the value, the better the method effect.

[0101] Tables 1 and 2 below show the comparison between the present invention and other methods on the Av1 and Av2 datasets:

[0102] Table 1

[0103] Model minFDE@1 minADE@1 minFDE@6 minADE@6 THOMAS 4.71 1.95 1.51 0.88 GANet 4.48 1.78 1.35 0.73 ProphNet 4.74 1.80 1.33 0.68 QCNet 4.30 1.69 1.29 0.65 The present invention 3.93 1.59 1.24 0.66

[0104] Table 2

[0105] Model minFDE@6 minADE@6 LaneGCN 0.71 1.08 HiVT 0.66 0.96 R-pred 0.66 0.95 HPNet 0.64 0.87 The present invention 0.61 0.91

[0106] It can be seen from Tables 1 and 2 above that the method of the present invention far exceeds most previous autonomous driving trajectory prediction methods. Specifically, the method of the present invention is significantly better than other methods in terms of minFDE@1 and minADE@1 indicators, which are 8.60% and 5.91% higher than the previous best method QCNet, respectively. In addition, the method of the present invention also achieves good performance in other indicators.

[0107] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for autonomous driving trajectory prediction based on streaming Transformer, characterized in that: The following steps are involved: (1) acquiring independent scene data of the autonomous driving vehicle, and preprocessing the independent scene data to obtain preprocessed continuous scene data; (2) Inputting the continuous scene data preprocessed in step (1) into the pre-trained streaming Transformer trajectory prediction model to obtain the predicted future trajectory; The streaming Transformer trajectory prediction model includes a scene encoder and a future trajectory decoder for single-shot processing, as well as a scene context stream and a predicted trajectory stream for cross-scene feature interaction; The scene encoder includes a map encoder, a history track encoder, and an interaction encoder; The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N m Represents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally and globally, and the output dimension is N m ×C map feature tensor F m , where C represents the dimension of the map feature tensor; The history track encoder uses a multi-layer one-dimensional neighbor attention module, whose input is a dimension of N a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps in the historical trajectory, and feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectories are processed by attention weighted encoding, and the output is a N-dimensional a ×C object feature tensor F a ; The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene feature tensor F s .

2. The automatic driving trajectory prediction method based on streaming Transformer according to claim 1 is characterized in that: Step (1) is specifically as follows: first, independent scene data of each autonomous driving vehicle is obtained, which includes the historical trajectory and high-precision map of the traffic participants; then, multiple time segmentation points are selected according to the historical trajectory of the traffic participants in the independent scene data; and a segment of trajectory is selected from each time segmentation point forward on the historical trajectory as the future predicted trajectory, and a segment of trajectory is selected from the time segmentation point backward as the known historical trajectory; the future predicted trajectories and the known historical trajectories corresponding to all the time segmentation points constitute a continuous trajectory segment; Then, the high-precision maps in the independent scene data are aggregated according to each time segmentation point, and the processing results and continuous trajectory segments together constitute continuous scene data.

3. The automatic driving trajectory prediction method based on streaming Transformer according to claim 2 is characterized in that: The input of the future trajectory decoder is the feature tensor F from the current scene s The extracted dimension is N fa ×C’s traffic participant feature tensor F fa , the future trajectory encoder focuses on the traffic participant feature tensor F fa Perform decoding operation, and the final output is the multimodal prediction trajectory Y, the corresponding probability P and its corresponding feature F Y , the dimension of the multimodal prediction trajectory Y is N fa ×N m ×T×2, where N fa Represents the number of traffic participants, N m represents the number of predicted trajectory modes, T is the number of steps in the predicted future trajectory, and the feature dimension 2 refers to the coordinate position; The input of the scene context information flow is the current scene feature tensor F output by the interaction encoder s , and the acquired historical scene feature tensor F s ′, which uses the decomposition attention module to process the two, and finally outputs the updated current scene feature tensor. The decomposition attention module includes a map-map attention submodule and an object-scene attention submodule; The predicted trajectory information flow is placed after the future trajectory decoder to connect the historical prediction with the current prediction. Its input is the multimodal predicted trajectory Y output by the future trajectory decoder and its corresponding feature F Y , the two are processed through the cross attention module, and the final output is the updated multimodal prediction trajectory Y r .

4. The automatic driving trajectory prediction method based on streaming Transformer according to claim 3 is characterized in that: The streaming Transformer trajectory prediction model is trained through the following steps: (2-1) obtaining an autonomous driving dataset for trajectory prediction, preprocessing the autonomous driving dataset to obtain a preprocessed autonomous driving dataset, and converting the preprocessed autonomous driving dataset into a vector format using a vectorization method in a VectorNet model to obtain a format-converted autonomous driving dataset, and dividing the format-converted autonomous driving dataset into a training set and a validation set at a ratio of 9:1, wherein the autonomous driving dataset includes historical and future trajectories of traffic participants and high-precision maps; (2-2) For each autonomous driving scene segment in the training set obtained in step (2-1), the high-precision map vector M (whose dimension is N) in the autonomous driving scene segment is m ×L×2) and the historical trajectory vector A of the traffic participants (whose dimension is N a ×H×4) input the map encoder, history trajectory encoder, and interaction encoder in the streaming Transformer trajectory prediction model to obtain N dimensions respectively. m ×C map feature tensor F m , dimension is N a ×C object feature tensor F a , and the dimension is (N a +N m )×C scene feature tensor F s ; (2-3) The scene context information flow uses a decomposed attention module (which includes a map-map attention submodule and an object-scene attention submodule) to analyze the current scene feature tensor F s And the obtained historical scene feature tensor F s ′ is processed to obtain the updated current scene feature tensor; (2-4) Extract the dimension N from the updated current scene feature tensor obtained in step (2-3) fa ×C is the feature tensor F of the traffic participant of interest fa , and input it into the future trajectory decoder to obtain the multimodal trajectory Y, the corresponding probability P and its corresponding feature F Y ; (2-5) The multimodal prediction trajectory Y obtained in step (2-4) and its corresponding feature F Y Input the predicted trajectory information stream to obtain the updated multimodal predicted trajectory Y r ; It should be noted that a memory trajectory library is maintained in the prediction trajectory information flow to store and record the historical multimodal prediction trajectory Y' and the corresponding feature F Y′ ; (2-6) The updated multimodal predicted trajectory Y obtained in step (2-5) r The loss function is calculated using the multimodal predicted trajectory Y and its corresponding probability P obtained in step (2-4), and the streaming Transformer trajectory prediction model is iteratively trained using the loss function until the streaming Transformer trajectory prediction model converges, thereby obtaining a trained streaming Transformer trajectory prediction model; (2-7) Use the verification set obtained in step (2-1) to verify the streaming Transformer trajectory prediction model preliminarily trained in step (2-6) to obtain the final trained streaming Transformer trajectory prediction model.

5. The automatic driving trajectory prediction method based on streaming Transformer according to claim 4 is characterized in that: The autonomous driving datasets used in step (2-1) are Argoverse 1 and Argoverse 2 datasets. The former is 323,557 autonomous driving scene clips collected in Miami and Pittsburgh, each of which contains 3 seconds of known historical trajectory, 2 seconds of future predicted trajectory and high-precision map, while the latter is a total of 250,000 autonomous driving scene clips collected in up to 6 cities, each of which contains 5 seconds of known historical trajectory, 6 seconds of future predicted trajectory and high-precision map; both datasets use a sampling frequency of 10 Hz and each scene only uses high-precision maps and traffic participants with a radius of 150 meters; The map encoder in step (2-2) adopts the PointNet network. Specifically, step (2-2) first maps the high-precision map vector M into a high-dimensional feature through a multi-layer perceptron. Then, the high-precision map vector M is processed by a maximum pooling operation to obtain the global feature corresponding to the high-precision map vector M. Then, the global feature is connected to each point feature in the high-precision map vector M to obtain a point feature containing global information. Then, the above mapping operation and maximum pooling operation are repeated for all point features to obtain a high-dimensional feature with a dimension of N. m ×C map feature tensor F m The object encoder uses a neighbor attention network, which compresses the historical trajectory vector A of the traffic participants in the autonomous driving scene fragment in time series and increases the dimension of the feature through multi-layer stacked neighbor attention modules to obtain a dimension N a ×C object feature tensor F a , and the object feature tensor F a With the map feature tensor F m The connection operation is performed to obtain a preliminary scene feature tensor. Subsequently, the interactive encoder uses a standard Transformer encoder to process the preliminary scene feature tensor to obtain a scene feature tensor F s .

6. The method for autonomous driving trajectory prediction based on streaming Transformer according to claim 5, characterized in that: Step (2-3) is as follows: the decomposition attention module first decomposes the current scene feature tensor F s And the obtained historical scene feature tensor F s ' is split to obtain the current object feature tensor, the current map feature tensor, the historical object feature tensor, and the historical map feature tensor respectively. After that, the map-map attention module in the decomposition attention module takes the current map feature tensor and the historical map feature tensor as input, the current map feature tensor as the query feature, and the historical map feature tensor as the key feature to perform attention operation processing to obtain the updated current map feature tensor; At the same time, the object-scene attention module takes the current object feature tensor and the historical scene feature tensor as input, uses the current object feature tensor as the query feature, and uses the historical scene feature as the key feature to perform attention operation processing to obtain the updated current scene feature tensor; Finally, the decomposition attention module concatenates the updated current object feature tensor with the map feature tensor to obtain the updated current scene feature tensor.

7. The automatic driving trajectory prediction method based on streaming Transformer according to claim 6 is characterized in that: Step (2-5) is as follows: first, the predicted trajectory information flow first obtains the historical multimodal predicted trajectory Y' and the corresponding feature F stored in the memory trajectory library in the predicted trajectory information flow. Y′ , and the historical multimodal prediction trajectory Y' is subjected to rotation and translation transformation to obtain the transformed historical multimodal prediction trajectory as follows: Where R(θ) and y represent the rotation matrix and trajectory center corresponding to the historical multimodal prediction trajectory Y', respectively; Then, the prediction trajectory information flow combines the multimodal prediction trajectory Y and its corresponding features F Y As query features, the transformed multimodal prediction trajectory and its corresponding features are input into the attention module as key-value features to obtain the updated features F Yr : Where MHA represents the attention operation, Q, K, and V represent the query features, key features, and value features used in the operation, respectively, and TE represents the trajectory encoding operation; Finally, the updated feature F Yr Input into the multi-layer perceptron module to obtain the updated multimodal prediction trajectory Y r And as the final network output.

8. The method for autonomous driving trajectory prediction based on streaming Transformer according to claim 7, characterized in that: The loss function Loss used in step (2-6) is: Loss=Loss reg +Loss cls +Loss refine in: Where L smoothL1 represents the SMOOTH-L1 function, L CE represents the cross entropy function, represents the true value of the multimodal prediction trajectory, Represents the true value of the probability corresponding to the multimodal predicted trajectory.

9. An autonomous driving trajectory prediction system based on streaming Transformer, characterized in that: include: The first module is used to obtain independent scene data of the autonomous driving vehicle and preprocess the independent scene data to obtain preprocessed continuous scene data; The second module is used to input the continuous scene data preprocessed by the first module into the pre-trained streaming Transformer trajectory prediction model to obtain the predicted future trajectory; The streaming Transformer trajectory prediction model includes a scene encoder and a future trajectory decoder for single-shot processing, as well as a scene context stream and a predicted trajectory stream for cross-scene feature interaction; The scene encoder includes a map encoder, a history track encoder, and an interaction encoder; The map encoder uses the PointNet network, whose input is of dimension N m ×L×2 high-precision map vector, where N m Represents the total number of HD map vectors, L represents the number of points contained in each HD map vector, and feature dimension 2 represents the coordinate value of the map under the bird's-eye view; its dimension is N m ×L×2 high-precision map vectors are processed locally and globally, and the output dimension is N m ×C map feature tensor F m , where C represents the dimension of the map feature tensor; The history track encoder uses a multi-layer one-dimensional neighbor attention module, whose input is a dimension of N a ×H×4 historical trajectory vectors of traffic participants, where N a represents the total number of traffic participants, H represents the number of steps in the historical trajectory, and feature dimension 4 represents the horizontal coordinate, vertical coordinate, speed on the horizontal coordinate, and speed on the vertical coordinate of the traffic participant, which is N a ×H×4 traffic participants’ historical trajectories are processed by attention weighted encoding, and the output is a N-dimensional a ×C object feature tensor F a ; The interactive encoder uses a standard Transformer network, whose input is of dimension N m ×C map feature tensor F m With dimension N a ×C object feature tensor F a , after connecting the two, the output dimension is (N a +N m )×C current scene feature tensor F s .

Images