Dual-mode collaborative end-to-end automatic driving trajectory prediction method based on momentum sensing

Abstract

The invention discloses a dual-mode collaborative end-to-end automatic driving trajectory prediction method based on momentum sensing, and the method comprises the steps: collecting environment information, extracting features, removing noise, and obtaining a bird's-eye view feature tensor; inputting the information into a target detection tracking module and a real-time mapping module to obtain an information tensor; performing trajectory prediction on the detected target, establishing a target kinematic model, and predicting a prediction trajectory point of the target according to historical trajectory data; synchronously inputting the information tensor into a trajectory prediction module to obtain candidate trajectories of a target generated according to the current information, and selecting the candidate trajectory closest to a predicted trajectory point; and carrying out feature level fusion on the obtained candidate trajectory and the prediction tensor to obtain an optimized prediction trajectory. According to the method, noise is eliminated through the lightweight network, the prediction result is used as a prediction generation basis, so that the stability of the prediction trajectory is improved, historical information is fused, normal work under the high-dynamic non-intervisibility condition is ensured, and robustness is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving trajectory prediction, and particularly to a dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception. Background Technique

[0002] Trajectory prediction is a crucial component in autonomous driving systems, directly affecting the vehicle's decision-making and planning capabilities. Currently, trajectory prediction methods are mainly divided into rule-based methods and deep learning-based methods. Rule-based methods rely on pre-set physical models and kinematic formulas, with the advantages of high computational efficiency and strong interpretability, but they have limitations in dealing with complex and dynamic traffic scenarios. Deep learning-based methods use a large amount of data to train models, which can capture implicit patterns and non-linear relationships in the data and improve prediction accuracy. However, training complex deep learning models requires high-performance computing resources, which may lead to overly long training times, and the internal mechanisms of the models are complex and are usually regarded as "black boxes", making it difficult to understand their decision-making processes and lacking interpretability of the prediction results.

[0003] Interacting between the two prediction modes can often make up for their respective disadvantages, enabling the prediction results to handle complex scenarios while also having a certain degree of interpretability and credibility. However, how to effectively combine the two is an urgent problem to be solved. Directly fusing the prediction results of the two at the result level often lacks stability and may lead to low model prediction accuracy. Therefore, exploring methods for fusion at the feature level may be more helpful in improving the stability and prediction accuracy of the model.

[0004] Using the rule method as an intermediate constraint for the deep learning model to optimize the prediction results. This fusion strategy aims to utilize the interpretability and stability of the rule method to make up for the deficiencies of the deep learning model in dealing with complex constraint conditions. By introducing physical constraints or prior knowledge into the deep learning model, the training stability and prediction accuracy of the model can be improved. However, in practical applications, factors such as computational cost, model complexity, and prediction accuracy need to be weighed to ensure the practicality and effectiveness of the fusion model. Therefore, when designing a dual-mode collaborative end-to-end trajectory prediction method, it is necessary to satisfy the original achievable effects while effectively constraining the process using the rule-based method. Summary of the Invention

[0005] The object of the present invention is to overcome the deficiencies of the prior art. To achieve the above object, a dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception is adopted to solve the problems raised in the above background technique.

[0006] A dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception includes the following steps:

[0007] Step S1: Collect the vehicle surrounding environment information through in-vehicle sensing devices, extract features using the bird's-eye view module, and remove noise through a lightweight network to obtain a bird's-eye view feature tensor.

[0008] Step S2: Input the denoised bird's-eye view feature tensor into the target detection and tracking module and the real-time mapping module in sequence to obtain the target detection result at the current moment and the information tensor of the environmental map.

[0009] Step S3: Use the Kalman filter algorithm to predict the trajectory of the detected target, establish a target kinematic model, and predict the predicted trajectory points of the target based on historical trajectory data.

[0010] Step S4: Synchronously input the obtained bird's-eye view feature tensor, target detection result, and information tensor of the environmental map into the trajectory prediction module to obtain candidate trajectories of the target generated according to the current information, and select the candidate trajectory closest to the predicted trajectory points.

[0011] Step S5: Perform feature-level fusion on the obtained candidate trajectory and the historical trajectory prediction tensor of the current target to obtain an optimized predicted trajectory.

[0012] As a further solution of the present invention: The specific steps in Step S1 include:

[0013] Step S11: Collect the vehicle perimeter visual raw data based on an in-vehicle multi-view camera and input it into a pre-trained ResNet-101 network. Through multi-layer residual convolution, obtain the multi-scale feature tensor at the current moment. where C is the number of cameras;

[0014] Step S12: Use a pyramid convolutional neural network to fuse the multi-scale feature tensors. Obtain the multi-scale feature tensor at the current moment under multiple perspectives.

[0015] Step S13: Input the multi-view multi-scale feature tensor V t , the historical BEV bird's-eye view feature information and the query tensor into the BEVFormer bird's-eye view module to obtain the bird's-eye view feature of the vehicle surrounding environment at the current moment.

[0016] Step S14: Based on the lightweight Tranformer structure, perform three self-attention networks, residual summation, and normalization operations on the obtained bird's-eye view feature to filter out the useless noise information in the feature tensor and obtain the denoised bird's-eye view feature tensor.

[0017] As a further solution of the present invention: The specific steps in step S2 include:

[0018] Step S21: Input the denoised bird's-eye view feature tensor into the target detection and tracking module, and after position encoding processing, input it into the Transformer encoding layer to extract the feature information containing the detection target;

[0019] Then perform decoding in sequence to obtain the information tensor containing the target recognition and positioning at the current moment, and the pose information of the corresponding object;

[0020] Step S22: Input the denoised bird's-eye view feature vector into the real-time mapping module, and use the graph neural network to process the extracted feature information to generate a simplified map and the information tensor containing the environmental map

[0021] As a further solution of the present invention: The specific steps in step S3 include:

[0022] Step S31: Establish the target motion state vector x and covariance matrix p, and transfer the target motion state vector to the current vehicle coordinate system x according to the pose transformation matrix of the vehicle in continuous time t-1 ;

[0023]

[0024] wherein, is the position, heading, speed, acceleration, and angular acceleration information of the target at the previous moment;

[0025] Step S32: Use the target motion model and previous state estimation to predict the pose information at the current moment and the covariance matrix

[0026]

[0027] wherein, F is the state transition matrix, w t is the process noise; the predicted covariance matrix is:

[0028]

[0029] wherein, Q k-1 is the process noise covariance;

[0030] Step S33: The pose information z corresponding to the target tAs the observed data, a Kalman filter update process is constructed to obtain the motion state of the target after observation. Based on this, a prediction for the next moment is made to obtain the trajectory prediction result based on the current moment.

[0031] As a further solution of the present invention: The specific steps in step S4 include:

[0032] Step S41: Input the information tensor obtained at the current moment obtained in step S3 and the bird's-eye view feature into the fully connected network layer. By combining the target-level feature and the environmental information, K candidate trajectories of the target are predicted.

[0033]

[0034] Among them, AAInt(·) is the event and event interactors, AMInt(·) is the event and local map interactors, AGInt(·) is the event and global interactors, and MLP(·) is the linear fully connected network.

[0035] Step S42: Use the Hausdorff distance to calculate the distances of the K candidate trajectories to the trajectory prediction point Select the maximum value among the minimum distances of the K sets of point sets to the prediction point as the predicted trajectory of the target.

[0036] As a further solution of the present invention: The specific steps in step S5 include:

[0037] Step S51: According to the candidate trajectories determined in step S4 the corresponding tensors and mix the target historical trajectory prediction information tensor and the score information tensor to obtain generated based on the predicted trajectory

[0038] Step S52: Sum the generated with the current bird's-eye view feature and perform different-level feature map combinations with the vehicle's own reference coordinate system information. The obtained results are respectively input into two fully connected networks to generate the predicted trajectory of the target and the corresponding score information tensor

[0039] Compared with the prior art, the present invention has the following technical effects:

[0040] With the above technical solution, noise is eliminated through a lightweight network, and the rule-based prediction result is used as the generation basis for trajectory prediction, thereby improving the stability of the predicted trajectory, and historical information is fused to ensure that the method can still work properly in the case of high dynamic non-line-of-sight, enhancing the robustness of the algorithm. Brief Description of the Drawings

[0041] The following will describe in detail the specific embodiments of the present invention with reference to the accompanying drawings:

[0042] Figure 1 It is a schematic diagram of the steps of the autonomous driving trajectory prediction method according to the disclosed embodiment of the present application;

[0043] Figure 2 It is a flowchart of the autonomous driving trajectory prediction method according to the disclosed embodiment of the present application. Specific Embodiments

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0045] Please refer to Figure 1 and Figure 2 In the embodiments of the present invention, a dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception includes the following steps:

[0046] Step S1: Collect the vehicle surrounding environment information through an in-vehicle sensing device, extract features using a bird's-eye view module, and remove noise through a lightweight network to obtain a bird's-eye view feature tensor. The specific steps include:

[0047] Step S11: Collect the vehicle perimeter visual raw data based on an in-vehicle multi-view camera and input it into a pre-trained ResNet-101 network, and obtain the multi-scale feature tensor at the current moment through multi-layer residual convolution where C is the number of cameras;

[0048] In this embodiment, the vehicle perimeter visual raw data is collected by an in-vehicle multi-view camera After passing through a convolution process with a stride of 2, a 7*7 convolution kernel, and a 64-dimensional output, and a convolution process with a stride of 2, a 3*3 convolution kernel, and a 64-dimensional output in sequence, a tensor with 6 channels and a size that is only the input is obtained;

[0049] Next, it is input into four groups of convolutional layers, and residual processing is performed on the input and output of each convolutional set respectively:

[0050]

[0051] Among them, is the output of the convolutional set, x is the input tensor, and h(x) is the residual term.

[0052] In each subsequent layer, the number of input channels is doubled, while the height and width dimensions are halved, and the feature tensors of the image at 256 dimensions, 512 dimensions, 1024 dimensions, and 2048 dimensions are obtained in sequence, marked as C2, C3, C4, and C5 respectively, and saved to At this time, the multi-scale feature tensor is obtained, where C is the number of cameras.

[0053] Step S12: Use the pyramid convolutional neural network to fuse the multi-scale feature tensor to obtain the multi-scale feature tensor at the current moment from multiple perspectives

[0054] In this embodiment, first, a 1*1 convolutional operation is applied to the 2048-dimensional feature tensor, and the number of channels of the output result is unified with that of the C4 layer to obtain the P5 layer feature tensor. Then, the P5 layer is upsampled by a factor of 2 and added to the C4 layer in terms of features to form the P4 layer feature tensor. Similar operations are performed to fuse P4 and C3 into layer P3, and P3 and C2 into layer P2 respectively.

[0055] After the above operations, a 3*3 convolutional operation is applied to the feature layers of different scales to improve the degree of feature fusion, and the tensor after fusing the multi-scale features from this perspective is saved to ;

[0056] Step S13: Input the multi-perspective multi-scale feature tensor V t , the historical BEV bird's-eye view feature information and the query tensor into the BEVFormer bird's-eye view module, and sequentially adopt the spatial cross-attention mechanism and the temporal attention mechanism to obtain the bird's-eye view feature of the vehicle surrounding environment at the current moment

[0057] Specifically, first, a query tensor is randomly generated, and the spatial cross-attention mechanism is adopted to perform feature interaction with the multi-perspective multi-scale feature tensor V t to achieve spatial feature fusion.

[0058] Next, the historical BEV bird's-eye view feature information is introduced. Through the ego-vehicle historical pose transformation matrix and displacement vector Γ t-1 , align the coordinates with the current moment:

[0059]

[0060] Adopt a temporal attention mechanism to fuse the BEV features of the current frame and historical frames in the temporal dimension, capture the temporal dependence relationship, enhance the feature representation ability, and finally obtain the BEV bird's-eye view feature of the vehicle's surrounding environment at the current moment

[0061] Step S14: Based on the lightweight Tranformer structure, apply the obtained bird's-eye view features through three self-attention networks, residual summation, and normalization operations to filter out the useless noise information in the feature tensor and obtain the denoised bird's-eye view feature tensor

[0062] In this embodiment, the BEV bird's-eye view features obtained in step S13 are passed through three self-attention networks and residual summation and normalization operations to filter out the useless noise information in the feature tensor and obtain the encoded BEV bird's-eye view feature tensor

[0063]

[0064] Then, input it into the fully connected neural network and activation function respectively to obtain the Q, K, and V vectors, fully fuse the features inside and outside the sequence, and perform activation processing on the result to obtain the denoised BEV bird's-eye view feature tensor

[0065]

[0066] Step S2: Input the denoised bird's-eye view feature tensor into the target detection and tracking module and the real-time mapping module in sequence to obtain the target detection result at the current moment and the information tensor of the environmental map. The specific steps include:

[0067] Step S21: Input the denoised bird's-eye view feature tensor into the target detection and tracking module. First, perform positional encoding on it, and then input it into the Transformer encoding layer to extract the feature information containing the detection target;

[0068] Then decode it, adopt the self-attention mechanism and the cross-attention mechanism to obtain the information tensor containing the target recognition and localization at the current moment and the pose information P of the corresponding object t ;

[0069] First, flatten the tensor into a one-dimensional sequence, perform positional encoding on it, and then input it into the Transformer encoder. Through the multi-head self-attention mechanism, global interaction is performed on all features in the sequence, and through the feed-forward network, non-linear transformation is performed to generate high-level semantic representations.

[0070] Next, send the encoded information tensor to the decoder, and adopt the self-attention mechanism and the cross-attention mechanism to capture the relationships between targets and global context information, focus on the regions in the image that are most relevant to the query, and thus extract the key information for target prediction.

[0071] Finally, after being processed by the attention mechanism, each query vector passes through a feed-forward neural network to generate the corresponding target category and recognition box.

[0072] Step S22: Input the denoised bird's-eye view feature vector into the real-time mapping module. By vectorizing the representation of the environmental map, convert the map elements and the trajectories of moving objects into vector forms. For each vector set in the map, construct a subgraph and build local features through the subgraph network.

[0073] Then integrate all the vector sets, use all the subgraph features as nodes to construct a global feature map. Utilize the graph neural network to process the extracted feature information to generate a simplified version of the map and an information tensor containing the environmental map

[0074] Step S3: Use the Kalman filtering algorithm to predict the trajectories of the detected targets, establish a target kinematic model, and predict the predicted trajectory points of the targets based on historical trajectory data. Its specific steps include:

[0075] Step S31: Establish the target motion state vector x and covariance matrix p, and transfer the target motion state vector to the current vehicle coordinate system x according to the pose transformation matrix of the vehicle in continuous time t-1 ;

[0076]

[0077] Among them, are the position, heading, speed, acceleration, and angular acceleration information of the target at the previous moment;

[0078] Step S32: Utilize the target motion model and previous state estimates to predict the pose information at the current moment and the covariance matrix

[0079]

[0080] Among them, F is the state transition matrix, and w t is the process noise; the prior covariance matrix of the prediction is:

[0081]

[0082] Among them, Q k-1 is the process noise covariance, is the posterior covariance matrix at the previous moment;

[0083] Step S33: Use the pose information z t corresponding to the target as the observation data, calculate the Kalman gain, construct the Kalman filter update process, and obtain the motion state of the target after observation and make a prediction for the next moment based on this, to obtain the trajectory prediction result based on the current moment

[0084] First, determine the mathematical expression of the target observation model:

[0085] z t = H t x t + v t

[0086] Among them, H t is the observation matrix, and v t is the observation noise. Then, it is necessary to continuously iteratively calculate the motion state of the target at the current moment using the observed values And the Kalman gain K t in each iteration process can be calculated as:

[0087]

[0088] Among them, r t is the observation noise covariance. Then, calculate the posterior motion state of the target at the current moment according to the Kalman gain of this iteration

[0089]

[0090] and the corresponding posterior covariance matrix

[0091]

[0092] Among them, I is the identity matrix. Through continuous iteration, the motion state of the target is obtained, and the trajectory point of the target at the next moment is estimated through the velocity, acceleration, and angular acceleration in the motion state

[0093] Step S4. Synchronously input the obtained bird's-eye view feature tensor, object detection result, and information tensor of the environmental map into the trajectory prediction module to obtain candidate trajectories of the object generated according to the current information: And use the Hausdorff distance to select the candidate trajectory closest to the predicted trajectory points. The specific steps include:

[0094] Step S41. Input the information tensor obtained at the current moment obtained in Step S3 and the bird's-eye view feature into the fully connected network layer. By combining the object-level features and environmental information, predict K candidate trajectories of the object

[0095]

[0096] where AAInt(·) is the event and event interaction unit, AMInt(·) is the event and local map interaction unit, AGInt(·) is the event and global interaction unit, and MLP(·) is the linear fully connected network;

[0097] Step S42. Calculate the distances of the K candidate trajectories to the trajectory prediction point using the Hausdorff distance, and select the maximum value among the minimum distances of the K sets of point sets to the prediction point as the predicted trajectory of the object

[0098] In this embodiment, first, divide the candidate trajectory set T t into K sets of point sets, and each set of point sets represents the preview point of the predicted trajectory. Then calculate the Hausdorff distance from the points in each set of point sets to the trajectory prediction point :

[0099]

[0100] Then select the trajectory point set with the minimum Hausdorff distance to the trajectory prediction point as the candidate trajectory of the object

[0101]

[0102] Step S5. Perform feature-level fusion on the obtained candidate trajectory and the historical trajectory prediction tensor of the current object to obtain an optimized predicted trajectory. The specific steps include:

[0103] Step S51. Look up the candidate trajectory determined according to Step S4 through indexing The corresponding tensor and mix the target historical trajectory prediction information tensor and the score information tensor to obtain the generated on the predicted trajectory Based on

[0104]

[0105] where σ(·) represents the sigmoid function. It is a feature tensor that fuses and extracts historical trajectory information. Through the cross-attention mechanism, the historical prediction trajectory feature information is fused, and the historical information is used as a process constraint to alleviate the problem of trajectory prediction floating caused by the target being occluded, and finally a feature tensor of the final predicted trajectory information is obtained

[0106] Step S52: Add the generated to the current bird's-eye view feature Perform summation processing, and perform feature map combinations at different levels with the vehicle's own reference coordinate system information. The obtained results are then respectively input into two fully connected networks to generate the predicted trajectory of the target and the corresponding score information tensor

[0107]

[0108] where Ego_anchor is the vehicle's own coordinate information, and concat(·) is the feature fusion operation.

[0109] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the protection scope of the present invention.

Claims

1. A dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception, characterized in that: The following steps are involved: Step S1, collecting vehicle surrounding environment information through the vehicle-mounted sensor device, extracting features using the bird's-eye view module, and removing noise through a lightweight network to obtain a bird's-eye view feature tensor; Step S2: input the denoised bird's-eye view feature tensor into the target detection and tracking module and the real-time mapping module in sequence to obtain the target detection result at the current moment and the information tensor of the environment map; Step S3: Use the Kalman filter algorithm to predict the trajectory of the detected target, establish a target kinematic model, and predict the predicted trajectory points of the target based on historical trajectory data; Step S4, synchronously inputting the obtained bird's-eye view feature tensor, target detection result, and information tensor of the environment map into the trajectory prediction module, obtaining a candidate trajectory for generating the target according to the current information, and selecting the candidate trajectory closest to the predicted trajectory point; Step S5: perform feature-level fusion on the obtained candidate trajectory and the historical trajectory prediction tensor of the current target to obtain an optimized predicted trajectory.

2. According to the momentum sensing-based dual-mode collaborative end-to-end autonomous driving trajectory prediction method of claim 1, it is characterized in that: The specific steps in step S1 include: Step S11: Collect the original visual data around the vehicle based on the on-board multi-view camera and input it into the pre-trained ResNet-101 network. Through multi-layer residual convolution, obtain the multi-scale feature tensor at the current moment. Where C is the number of cameras; Step S12: Use pyramid convolutional neural network to fuse multi-scale feature tensors Get the multi-scale feature tensor under multiple perspectives at the current moment Step S13: transform the multi-view multi-scale feature tensor V t , Historical BEV Bird's-Eye View Feature Information and query tensor Input to the BEVFormer bird's-eye view module to obtain the bird's-eye view features of the vehicle's surrounding environment at the current moment Step S14: Based on the lightweight Tranformer structure, the obtained bird's-eye view features After three self-attention networks, residual summation, and normalization operations, the useless noise information in the feature tensor is filtered out to obtain the denoised bird's-eye view feature tensor.

3. The dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception according to claim 1 is characterized in that: The specific steps in step S2 include: Step S21: The denoised bird's-eye view feature tensor The input is sent to the target detection and tracking module, and after position encoding, it is input to the Transformer encoding layer to extract the feature information of the detected target. Then decode it in sequence to obtain the information tensor containing the target recognition and positioning at the current moment, as well as the corresponding object posture information; Step S22: The denoised bird's-eye view feature vector Input the real-time mapping module, use the graph neural network to process the extracted feature information, generate a simplified map and an information tensor containing the environment map 4. The dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception according to claim 1 is characterized in that: The specific steps in step S3 include: Step S31: Establish the target motion state vector x and covariance matrix p, and transfer the target motion state vector to the current vehicle coordinate system x according to the vehicle's continuous time posture transformation matrix. t-1 ; in, The position, heading, speed, acceleration, and angular acceleration information of the target at the last moment; Step S32: using the target's motion model and previous state estimation, according to the motion state Predict the current position information And the covariance matrix Where F is the state transfer matrix, w t is the process noise; the covariance matrix of the prediction for: Among them, Q k-1 is the process noise covariance; Step S33: the position information z corresponding to the target t As observation data, construct the Kalman filter update process to obtain the motion state of the target after observation And make a prediction for the next moment on this basis to get the trajectory prediction result based on the current moment 5. The dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception according to claim 1 is characterized in that: The specific steps in step S4 include: Step S41: The information tensor obtained at the current moment obtained in step S3 With bird's eye view feature Input into the fully connected network layer, and predict K candidate trajectories of the target by combining target-level features with environmental information Among them, AAInt(·) is an event-event interactor, AMInt(·) is an event-local map interactor, AGInt(·) is an event-global interactor, and MLP(·) is a linear fully connected network; Step S42: Calculate K candidate trajectories using Hausdorff distance To the predicted trajectory point Distance, select K groups of points to the predicted point The maximum value among the minimum distances is used as the predicted trajectory of the target 6. The dual-mode collaborative end-to-end autonomous driving trajectory prediction method based on momentum perception according to claim 1 is characterized in that: The specific steps in step S5 include: Step S51: candidate trajectories determined in step S4 The corresponding tensor And mix the target history trajectory prediction information tensor And the score information tensor Get the predicted trajectory Generated on the basis of Step S52: generate the obtained With the current bird's eye view feature The summation is performed and the feature maps of different levels are combined with the vehicle's own reference coordinate system information. The results are then input into two fully connected networks to generate the predicted trajectory of the target. And the corresponding score information tensor

Images