Trajectory prediction method for autonomous vehicle and model training method thereof

Abstract

This application discloses a trajectory prediction method and a model training method for autonomous vehicles. The model training method includes: acquiring raw sensor data, vehicle state, and future trajectory information collected by the autonomous vehicle; constructing a Gaussian bird's-eye view BEV heatmap of the autonomous vehicle based on the future trajectory information and the vehicle speed in the vehicle state; determining the trajectory-guided BEV features output by the first network architecture based on the raw sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture; and training the first and second network architectures to be trained based on the Gaussian BEV heatmap, the predicted BEV heatmap, the trajectory-guided BEV features, and the vehicle state. This scheme utilizes the future trajectory itself to generate a self-supervised Gaussian BEV heatmap as a supervision signal, eliminating the dependence on perception annotations and predefined motion priors.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a trajectory prediction method for autonomous vehicles and a model training method thereof. Background Technology

[0002] Currently, in autonomous driving systems, end-to-end systems typically learn the mapping from raw sensor inputs (such as cameras and LiDAR) to driving actions or future trajectories directly through deep learning models. However, end-to-end methods attempting to bypass perception labeling often rely on complex future frame prediction or latent feature reconstruction as auxiliary tasks. For example, future frame prediction involves predicting future sensor data (such as images or point cloud frames), while latent feature reconstruction involves reconstructing latent representations of the input data or learning general environmental representations.

[0003] However, the aforementioned auxiliary tasks introduce additional computational complexity and design bottlenecks, making it difficult to cover the complex and ever-changing driving scenarios in the real world. This results in poor generalization and low efficiency of the prediction models used to generate trajectories. Summary of the Invention

[0004] This application addresses the aforementioned shortcomings by providing a trajectory prediction method and model training method for autonomous vehicles. This approach utilizes the vehicle's future trajectory itself to generate a self-supervised Gaussian bird's-eye-view (BEV) heatmap as a supervisory signal, guiding the trajectory prediction network to learn BEV features containing potential driving intentions and scene information. This significantly reduces the reliance on perception-labeled data, achieving efficient, self-supervised end-to-end trajectory generation and improving prediction accuracy and generalization ability.

[0005] In a first aspect, a trajectory prediction method for autonomous vehicles is provided, comprising: acquiring real-time sensor data and real-time vehicle state information collected by the autonomous vehicle; inputting the real-time sensor data and the real-time vehicle state information into a target model to obtain a predicted trajectory of the autonomous vehicle output by the target model, wherein the target model includes a first network architecture and a second network architecture; wherein the first network architecture is used to determine second trajectory-guided BEV features, and the first network structure includes a first network, a second network, and a third network.

[0006] The first network is used to determine a second BEV feature based on the real-time sensor data. The second network is used to determine a second coded feature based on the real-time vehicle status information. The third network is used to determine a second predicted BEV heatmap based on the second BEV feature, the second coded feature, and the second BEV heatmap query vector. The fusion of the second BEV feature and the second predicted BEV heatmap is used to determine the second trajectory-guided BEV feature. The second network architecture is used to determine the prediction noise in the predicted trajectory based on the second trajectory-guided BEV feature and the real-time vehicle status information, so as to determine the denoised predicted trajectory.

[0007] This approach enables end-to-end trajectory generation without annotations or predefined motion priors, reducing system complexity while enhancing the ability to model driving uncertainties.

[0008] In conjunction with the first aspect, in a possible implementation of the first aspect, the real-time sensor data and the real-time vehicle state information are input into a target model to obtain the predicted trajectory of the autonomous vehicle output by the target model, including: processing the real-time sensor data through a first network to obtain the second BEV feature, wherein the first network includes a Transfuser encoder, and the real-time sensor data includes real-time images and / or real-time LiDAR input; processing the real-time vehicle state information through a second network to obtain the second encoded feature, wherein the real-time vehicle state information includes one or more of the real-time speed, real-time acceleration, and real-time navigation commands of the autonomous vehicle; obtaining the initialized third BEV heatmap query vector; processing the third BEV heatmap query vector, the second BEV feature, and the second encoded feature through a transformer encoder in the third network to determine a fourth BEV heatmap query vector; processing the fourth BEV heatmap query vector through a deconvolutional network in the third network to obtain the second predicted BEV heatmap; and determining the second trajectory-guided BEV feature based on the fusion of the second predicted BEV heatmap and the second BEV feature.

[0009] This scheme utilizes a Transfuser encoder to efficiently fuse multimodal sensor data to generate a second BEV feature. Simultaneously, a dedicated network encodes real-time vehicle status information. A Transformer encoder, combining the initial query vector, the second BEV feature, and the real-time vehicle status encoded features, learns to generate a second query vector more focused on heatmap prediction. This second query vector is then decoded into a second predicted BEV heatmap using a deconvolutional network. Finally, the original BEV feature and the predicted heatmap are fused to obtain a trajectory-guided BEV feature that incorporates both original environmental perception information and predicted future location probability information.

[0010] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the second trajectory-guided BEV feature based on the fusion of the second predicted BEV heatmap and the second BEV feature includes: inputting the second predicted BEV heatmap and the second BEV feature into a channel fusion network to determine the second trajectory-guided BEV feature.

[0011] This scheme uses a dedicated channel fusion network to deeply fuse the predicted heatmap and the original BEV features, which can more effectively extract and combine key information from the two types of features, generating more informative and directional trajectory-guided BEV features.

[0012] Secondly, a model training method for trajectory prediction of autonomous vehicles is provided, comprising: acquiring raw sensor data, vehicle state, and future trajectory information collected by the autonomous vehicle; constructing a Gaussian bird's-eye view BEV heatmap of the autonomous vehicle based on the future trajectory information and the vehicle speed in the vehicle state, wherein the Gaussian BEV heatmap is used to indicate the probability distribution of the vehicle's position in the BEV space in the future time period; determining the trajectory-guided BEV features output by the first network architecture based on the raw sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture, wherein the Gaussian BEV heatmap is supervisory information, the first network architecture includes a first network, a second network, and a third network, wherein the first network is used to determine the first BEV feature based on the raw sensor data, the second network is used to determine the first encoded feature based on the vehicle state, and the third network... The system is used to determine a predicted BEV heatmap based on the first BEV feature, the first encoded feature, and the first BEV heatmap query vector. The fusion of the first BEV feature and the predicted BEV heatmap is used to determine the trajectory-guided BEV feature. Based on the Gaussian BEV heatmap, the predicted BEV heatmap, the trajectory-guided BEV feature, and the vehicle state, the first network architecture and the second network architecture to be trained are trained to obtain a target model for trajectory prediction of autonomous vehicles. The second network architecture is used to determine the predicted noise in the future trajectory based on the trajectory-guided BEV feature and the vehicle state. The loss function of the first network architecture and the second network architecture is determined based on a first deviation between the Gaussian BEV heatmap information and the predicted BEV heatmap feature, and a second deviation between the predicted noise determined by the second network architecture and the real noise added to the future trajectory information.

[0013] This approach utilizes real-world future trajectory information to generate self-supervised Gaussian BEV heatmaps as supervisory signals. These heatmaps guide the first network architecture to learn and generate trajectory-guided BEV features rich in driving intent and scene context information. A second network architecture is then used for denoising training. Consequently, without relying on external perception annotations or predefined motion patterns, this approach achieves self-supervised, end-to-end generation of diverse and accurate future trajectory predictions, significantly improving the model's generalization and predictive performance.

[0014] Combining the first and second aspects, in the possible implementations of the first and second aspects, a Gaussian bird's-eye view BEV heatmap of the autonomous vehicle is constructed based on the future trajectory information and the vehicle speed in the vehicle state. This includes: for each moment in the future time domain, using the BEV position of the autonomous vehicle at each moment as the mean of the Gaussian distribution, and constructing a Gaussian distribution based on the standard deviation corresponding to the vehicle speed; and determining the Gaussian BEV heatmap based on the maximum value obtained by the Gaussian distribution at each BEV grid point at each moment.

[0015] This scheme uses the vehicle's future real position as the mean and combines it with the current speed to determine the standard deviation, thus constructing a Gaussian probability distribution heatmap that conforms to the vehicle's motion characteristics. This provides accurate and physically meaningful supervision information for subsequent network training.

[0016] In conjunction with the first and second aspects, in possible implementations of the first and second aspects, determining the trajectory-guided BEV features output by the first network architecture based on the original sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture includes: processing the original sensor data through the first network to obtain the first BEV features, wherein the first network includes a Transfuser encoder, and the original sensor data includes images and / or LiDAR input; processing the vehicle state through the second network to obtain the first encoded features, wherein the vehicle state includes one or more of the autonomous vehicle's speed, acceleration, and navigation commands; obtaining an initialized first BEV heatmap query vector; processing the first BEV heatmap query vector, the first BEV features, and the first encoded features through a transformer encoder in the third network to determine a second BEV heatmap query vector; processing the second BEV heatmap query vector through a deconvolutional network in the third network to obtain the predicted BEV heatmap; and determining the trajectory-guided BEV features based on the fusion of the predicted BEV heatmap and the first BEV features.

[0017] This scheme utilizes a Transfuser encoder to efficiently fuse multimodal sensor data to generate the first BEV feature. Simultaneously, a dedicated network encodes vehicle state information. A Transformer encoder, combining the initial query vector, the first BEV feature, and the encoded vehicle state features, learns to generate a second query vector more focused on heatmap prediction. This second query vector is then decoded into a predicted BEV heatmap using a deconvolutional network. Finally, the original BEV features and the predicted heatmap are fused to obtain trajectory-guided BEV features that incorporate both original environmental perception information and predicted future location probability information.

[0018] In conjunction with the first and second aspects, in possible implementations of the first and second aspects, determining the trajectory-guided BEV feature based on the fusion of the predicted BEV heatmap and the first BEV feature includes: inputting the predicted BEV heatmap and the first BEV feature into a channel fusion network to determine the trajectory-guided BEV feature.

[0019] This scheme uses a dedicated channel fusion network to deeply fuse the predicted heatmap and the original BEV features, which can more effectively extract and combine key information from the two types of features, generating more informative and directional trajectory-guided BEV features.

[0020] In conjunction with the second aspect, in a possible implementation of the second aspect, the first network architecture and the second network architecture to be trained are trained based on the trajectory-guided BEV features and the vehicle state, including: adding Gaussian noise to the future trajectory information to obtain noisy trajectory information; inputting the noisy trajectory information, the trajectory-guided BEV features, and the vehicle state into the second network architecture to obtain the predicted noise output by the second network architecture; and training the second network architecture based on the deviation between the Gaussian noise and the predicted noise.

[0021] This approach employs a diffusion model training paradigm. By adding noise to the real trajectory and training a second network architecture to predict the added noise, the model learns to gradually recover the real trajectory from the noise distribution. Combining trajectory-guided BEV features and vehicle state as conditional information, the model is guided to learn and generate diverse trajectories that conform to the scene intent and vehicle state.

[0022] In conjunction with the first and second aspects, in possible implementations of the first and second aspects, inputting the noisy trajectory information, the trajectory-guided BEV feature, and the vehicle state into the second network architecture includes: mapping the noisy trajectory information into latent features; processing the trajectory-guided BEV feature through a compressor with a Q-Former structure to obtain a BEV query vector; determining a conditional embedding vector through diffusion time-distributed embedding and enhanced vehicle query in the vehicle state; and inputting the latent feature, the BEV query vector, and the conditional embedding vector into the second network architecture.

[0023] This scheme utilizes the Q-Former structure to effectively compress and refine high-dimensional trajectory-guided BEV features, generating information-dense BEV query vectors. Simultaneously, time information is introduced through diffusion time step embedding, and combined with enhanced vehicle state query vectors (such as vehicle state further encoded by MLP), together forming conditional embedding vectors. This provides more accurate and richer conditional information for the second network architecture (diffusion model), improving the accuracy of denoising prediction.

[0024] Thirdly, a trajectory prediction device for autonomous vehicles is provided, comprising: an acquisition module for acquiring real-time sensor data and real-time vehicle state information collected by the autonomous vehicle; and an inference module for inputting the real-time sensor data and the real-time vehicle state information into a target model to obtain a predicted trajectory of the autonomous vehicle output by the target model, wherein the target model includes a first network architecture and a second network architecture; wherein the first network architecture is used to determine a second trajectory-guided vehicle (BEV) feature, the first network structure includes a first network, a second network, and a third network, the first network is used to determine the second BEV feature based on the real-time sensor data, the second network is used to determine a second encoded feature based on the real-time vehicle state information, the third network is used to determine a second predicted BEV heatmap based on the second BEV feature, the second encoded feature, and a second BEV heatmap query vector, the fusion of the second BEV feature and the second predicted BEV heatmap is used to determine the second trajectory-guided BEV feature, and the second network architecture is used to determine the prediction noise in the predicted trajectory based on the second trajectory-guided BEV feature and the real-time vehicle state information to determine the denoised predicted trajectory.

[0025] Fourthly, a model training device for trajectory prediction of autonomous vehicles is provided, comprising: an acquisition module for acquiring raw sensor data, vehicle state, and future trajectory information collected by the autonomous vehicle; a processing module for constructing a Gaussian bird's-eye view BEV heatmap of the autonomous vehicle based on the future trajectory information and the vehicle speed in the vehicle state, the Gaussian BEV heatmap indicating the probability distribution of the vehicle's position in the BEV space within a future time period; and determining trajectory-guided BEV features output by the first network architecture based on the raw sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture, wherein the Gaussian BEV heatmap is supervisory information, the first network architecture includes a first network, a second network, and a third network, the first network is used to determine a first BEV feature based on the raw sensor data, and the second network is used to determine a first encoded feature based on the vehicle state. The third network is used to determine a predicted BEV heatmap based on the first BEV feature, the first encoded feature, and the first BEV heatmap query vector. The fusion of the first BEV feature and the predicted BEV heatmap is used to determine the trajectory-guided BEV feature. The first and second network architectures are trained based on the Gaussian BEV heatmap, the predicted BEV heatmap, the trajectory-guided BEV feature, and the vehicle state to obtain a target model for trajectory prediction of autonomous vehicles. The second network architecture is used to determine the predicted noise in the future trajectory based on the trajectory-guided BEV feature and the vehicle state. The loss functions of the first and second network architectures are determined based on a first deviation between the Gaussian BEV heatmap information and the predicted BEV heatmap feature, and a second deviation between the predicted noise determined by the second network architecture and the real noise added to the future trajectory information.

[0026] Fifthly, an electronic device is provided, including one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the method as in the first aspect and any possible implementation thereof to be performed, or cause the method as in the second aspect and any possible implementation thereof to be performed.

[0027] In a sixth aspect, a computer-readable storage medium is provided that stores computer instructions that, when executed on a computer, cause the method as described in the first aspect and any possible implementation thereof to be performed, or cause the method as described in the second aspect and any possible implementation thereof to be performed.

[0028] In a seventh aspect, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method as described in the first aspect and any possible implementation thereof, or causes the computer to perform the method as described in the second aspect and any possible implementation thereof. Attached Figure Description

[0029] Figure 1 This is a schematic flowchart of a model training method for trajectory prediction of autonomous vehicles provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of BEV heatmap construction provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of trajectory-guided BEV feature extraction provided in an embodiment of this application;

[0032] Figure 4 This is a structural schematic diagram of a device provided in an embodiment of this application;

[0033] Figure 5 This is a structural schematic diagram of a system on a chip (SoC) provided in an embodiment of this application. Detailed Implementation

[0034] The technical solution of this application is described below with reference to the accompanying drawings.

[0035] As mentioned in the background section, auxiliary tasks such as future frame prediction or latent feature reconstruction introduce additional computational complexity and design bottlenecks, making it difficult to cover the complex and ever-changing driving scenarios in the real world. This results in poor generalization and low efficiency of the prediction models used to generate trajectories.

[0036] To address the aforementioned issues, this application proposes a model training method 100 for trajectory prediction of autonomous vehicles. Figure 1 A schematic flowchart of a model training method 100 for trajectory prediction of autonomous vehicles provided in an embodiment of this application is shown. Figure 1 As shown, method 100 includes steps S110 to S140. In method 100, a self-supervised Gaussian bird's-eye view BEV heatmap generated using the vehicle's future trajectory itself can be used as a supervision signal to guide the trajectory prediction network to learn BEV features containing potential driving intentions and scene information, significantly reducing the dependence on perception annotation data. Furthermore, efficient, self-supervised end-to-end trajectory generation is achieved, improving prediction accuracy and generalization ability.

[0037] Step S110: Obtain raw sensor data, vehicle status, and future trajectory information collected by the autonomous vehicle.

[0038] Specifically, the raw sensor data includes environmental perception data collected in real time by the vehicle through sensors such as cameras and lidar; the vehicle status includes the vehicle's own motion state parameters; and the future trajectory information includes the coordinate sequence of the vehicle's planned driving path in the future.

[0039] For example, in embodiments of this application, the raw sensor data may include image data, point cloud data, or multimodal fusion data, such as road images captured by a forward-looking camera or 3D point clouds generated by LiDAR scanning. The vehicle status may include one or more of the following: vehicle speed, acceleration, steering angle, and navigation commands, such as a current vehicle speed of 60 km / h and an acceleration of 0.5 m / s². 2 And the left-turn command output by the navigation system. Future trajectory information can refer to the generated sequence of coordinates of one location point per second over the next few seconds.

[0040] Step S120: Based on the future trajectory information and the vehicle speed in the current vehicle state, construct a Gaussian bird's-eye view BEV heat map of the autonomous vehicle.

[0041] Specifically, this Gaussian BEV heatmap is used to indicate the probability distribution of the vehicle's position in the BEV space over future time periods. For example, the Gaussian BEV heatmap uses a two-dimensional grid to represent the probability density of the vehicle appearing at different positions in the BEV coordinate system at different future times.

[0042] Optionally, in step S120, this embodiment of the application can also construct a Gaussian distribution heatmap based on kinematic characteristics. Specifically, based on the future trajectory information and the vehicle speed in the vehicle state, a Gaussian bird's-eye view BEV heatmap of the autonomous vehicle is constructed, including: for each moment in the future time domain, using the BEV position of the autonomous vehicle at each moment as the mean of the Gaussian distribution, and constructing a Gaussian distribution based on the standard deviation corresponding to the vehicle speed; determining the Gaussian BEV heatmap based on the maximum value obtained by the Gaussian distribution at each BEV grid point at each moment.

[0043] For example, in an embodiment of this application, for the future planning time domain Every moment within The BEV position of the vehicle at that moment As the mean of the Gaussian distribution, and using the vehicle speed... An adaptive standard deviation is used to construct a Gaussian distribution in the BEV space. The standard deviation of the Gaussian distribution With the speed of the vehicle Proportional, that is .in It is an adjustable hyperparameter that controls the range of influence of speed. Ultimately, the target of the comprehensive Gaussian BEV heatmap for the entire future time period. By considering all future moments Gaussian distribution at each BEV grid point The maximum value is obtained by taking the upper limit. This scheme uses the adaptive standard deviation of speed to reflect motion uncertainty, ensuring that the heatmap conforms to the vehicle dynamics characteristics and providing a high-fidelity supervision signal for feature learning.

[0044] Step S130: Based on the original sensor data, the vehicle status, the Gaussian BEV heatmap, and the first network architecture, determine the trajectory-guided BEV features output by the first network architecture.

[0045] Specifically, the Gaussian BEV heatmap is supervisory information. The first network architecture includes a first network, a second network, and a third network. The first network is used to determine a first BEV feature based on the raw sensor data. The second network is used to determine a first coded feature based on the vehicle state. The third network is used to determine a predicted BEV heatmap based on the first BEV feature, the first coded feature, and the first BEV heatmap query vector. The fusion of the first BEV feature and the predicted BEV heatmap is used to determine the trajectory-guided BEV feature.

[0046] Specifically, in step S130, the trajectory-guided BEV features output by the first network architecture are determined based on the original sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture. This includes: processing the original sensor data through the first network to obtain the first BEV features, wherein the first network includes a Transfuser encoder, and the original sensor data includes images and / or LiDAR input; processing the vehicle state through a second network to obtain the first encoded features, wherein the vehicle state includes one or more of the autonomous vehicle's speed, acceleration, and navigation commands; obtaining an initialized first BEV heatmap query vector; processing the first BEV heatmap query vector, the first BEV features, and the first encoded features through a transformer encoder in a third network to determine a second BEV heatmap query vector; processing the second BEV heatmap query vector through a deconvolutional network in the third network to obtain the predicted BEV heatmap; and determining the trajectory-guided BEV features based on the fusion of the predicted BEV heatmap and the first BEV features.

[0047] For example, firstly, the image and LiDAR input at the current moment are processed through a shared first network (or sensor backbone network, such as a Transfuser encoder) to generate initial first BEV features. Simultaneously, the vehicle's status (such as speed, acceleration, and navigation commands) is encoded into first-level coded features through a second network (such as a multilayer perceptron). Furthermore, in order to decode the driving mode from these features, embodiments of this application initialize a set of sparse first BEV heatmap query vectors. (in Furthermore, through the Transformer encoder in the third network... This allows these queries to interact with BEV and vehicle features, thereby aggregating environmental and vehicle information to obtain a second BEV heatmap query vector. Subsequently, a lightweight deconvolutional network from the third network is used. The processed second BEV heatmap query vector Upsampling is performed to output the predicted single-channel BEV heatmap. Its size and The spatial dimensions are the same. Finally, the initial BEV features are... Compared with predicted BEV heatmap features The data is then fused to generate trajectory-guided BEV features (TrajBEV). The following examples will provide a detailed description of this process.

[0048] Optionally, in step S130, to address the information loss caused by simple feature splicing, this application may also employ a deep feature fusion strategy in its embodiments. Specifically, determining the trajectory-guided BEV feature based on the fusion of the predicted BEV heatmap and the first BEV feature includes: inputting the predicted BEV heatmap and the first BEV feature into a channel fusion network (such as several convolutional layers) to determine the trajectory-guided BEV feature.

[0049] Step S140: Based on the Gaussian BEV heatmap, the predicted BEV heatmap, the trajectory-guided BEV features, and the vehicle state, train the first network architecture and the second network architecture to be trained to obtain a target model for trajectory prediction of autonomous vehicles.

[0050] Specifically, the second network architecture is used to determine the predicted noise in the future trajectory based on the trajectory-guided BEV features and the vehicle state. The loss functions of the first network architecture and the second network architecture are determined based on a first deviation between the Gaussian BEV heatmap information and the predicted BEV heatmap features, and a second deviation between the predicted noise determined by the second network architecture and the real noise added to the future trajectory information.

[0051] Optionally, in step S140, the first network architecture and the second network architecture to be trained are trained according to the trajectory-guided BEV features and the vehicle state, including: adding Gaussian noise to the future trajectory information to obtain noisy trajectory information; inputting the noisy trajectory information, the trajectory-guided BEV features and the vehicle state into the second network architecture to obtain the predicted noise output by the second network architecture; and training the second network architecture according to the deviation between the Gaussian noise and the predicted noise.

[0052] For example, in an embodiment of this application, the second network architecture may include a diffusion model and a noise prediction network.

[0053] For example, in the diffusion process (or training process), towards the true future trajectory. Gaussian noise is gradually added to the trajectory to obtain the noisy track. Noise prediction network The goal is to predict the noise added to the trajectory. Noise prediction network The input may include noise trajectories. Conditional embedding and the aforementioned trajectory-guided BEV characteristics For example, noise trajectories. A learnable trajectory encoder can be used. Mapping to latent features Conditional embedding It can be embedded by diffusion time step And enhanced vehicle query Adding them together, we get: .in, Through an Ego-BEV interaction module Calculation. Trajectory-guided BEV features. High-resolution BEV features can be transformed into a compact set of BEV queries using a Q-Former-structured compressor. For example, in the denoising process (or inference process), from random noise... Initially, a pre-trained noise prediction network was used. Iterative denoising ultimately yields a clean trajectory. The following examples will provide a detailed description of this process.

[0054] For example, the total loss function of the first and second network architectures can be composed of a weighted average of the Gaussian BEV heatmap loss (i.e., the first bias) and the diffusion model loss (i.e., the second bias). For example, the Gaussian BEV heatmap loss... A focus-based loss approach can be used to encourage the network to accurately predict defined heatmap targets. (The loss in the diffusion model is also mentioned.) The mean squared error loss can be used to directly optimize the noise prediction network. The total loss can be a weighted average of the two losses mentioned above. The following examples will describe this process in detail.

[0055] Method 100 utilizes a self-supervised Gaussian BEV heatmap generated from future trajectories to guide feature learning, and combines it with a diffusion model for denoising training, achieving end-to-end trajectory generation without perceptual annotation. Furthermore, it employs heatmap construction, multi-stage feature processing, channel attention fusion, and Q-Former conditional compression to achieve efficient integration of environmental representation and motion intent, significantly improving the accuracy and diversity of trajectory prediction in complex scenarios.

[0056] The following is an introduction Figure 1 Detailed embodiments of each step shown.

[0057] Specifically, embodiments of this application provide an end-to-end autonomous driving trajectory generation method that requires no perception annotation and no predefined motion priors (such as trajectory anchor points). Exemplarily, embodiments of this application unify driving environment understanding and trajectory generation into a sequence generation problem based on a diffusion model. It innovatively constructs trajectory-guided BEV features and designs a trajectory-guided BEV diffusion Transformer to achieve a direct, diverse, and reasonable end-to-end mapping from raw sensor input to future trajectories.

[0058] In this embodiment, the future trajectory is first self-supervisedly encoded into a Gaussian BEV heatmap. This heatmap is then used as a self-supervised signal to guide a lightweight encoder in learning BEV features containing potential driving modes. Next, this feature, along with the vehicle's state, is used as a condition to perform denoising sampling on the continuous trajectory space using a diffusion model, thereby directly generating the final trajectory. This method completely eliminates the dependence on perceptual annotations and manual motion priors. Its unlabeled nature allows for improved model performance through trajectory data augmentation. The detailed process of each step is described below.

[0059] Corresponding to step S120, this application embodiment abandons the traditional approach of relying on perception annotations such as obstacle bounding boxes to learn BEV features, and proposes to construct a self-supervised Gaussian BEV heatmap target using the future trajectory itself. Figure 2 This illustration shows a schematic diagram of BEV heatmap construction provided in an embodiment of this application. Figure 2The core innovation of the heatmap shown lies in encoding a discrete sequence of future trajectory points into a continuous, probabilistic dynamic probability field for BEV space. This field not only represents the probability distribution of the drivable area, but also implicitly encodes the positional uncertainty under different motion states through a speed-adaptive mechanism, thereby directly reflecting the driving mode.

[0060] Specifically, such as Figure 2 As shown, for the future planning time domain Every moment within The BEV position of the vehicle at that moment As the mean of the Gaussian distribution, and using the vehicle speed... An adaptive standard deviation is constructed by creating a Gaussian distribution in the BEV space, such that the standard deviation of the Gaussian distribution is such that... With the speed of the vehicle Proportional, that is ,in It is an adjustable hyperparameter that controls the range of influence of speed. Then, in the BEV space, a Gaussian distribution is constructed for each time step. As shown in the following formula (1):

[0061]

[0062] Ultimately, the target of the comprehensive Gaussian BEV heatmap for the entire future time period. By considering all future moments Gaussian distribution at each BEV grid point Taking the maximum value above, its mathematical form is shown in equation (2) below:

[0063]

[0064] in Indicates the future moment of the vehicle The actual location coordinates. For the future of self-driving cars The speed. It is an adjustable hyperparameter that controls the range of influence of speed. The higher the speed, the more dispersed the Gaussian distribution, indicating that there is greater acceptable positional uncertainty at high speeds. This represents the spatial coordinate index on the BEV feature map. Indicates the location The final target value of the heat map is between (0,1]. The closer the value is to 1, the higher the probability that the location belongs to the vehicle's trajectory in the future.

[0065] Figure 2 The core physical significance of the design shown is: when a vehicle is moving at low speed or stationary (such as waiting at an intersection). Small, The small Gaussian distribution is sharply concentrated, and the heatmap shows clear high-probability points, reflecting high positional certainty at low speeds; when the vehicle is traveling at high speed... big, The large, flat, and diffuse Gaussian distribution of the heatmap indicates a broader, gradually varying probability region, reflecting greater potential location uncertainty at high speeds. This speed-adaptive characteristic allows the heatmap to dynamically encode uncertainties in driving. This heatmap aims to... As a self-monitoring signal, the network is forced to extract features from the raw sensor data that can explain the uncertainty of future trajectory position and its change with velocity in the process of trying to predict it, thereby learning a BEV spatial representation that can reflect both the probability of future position and implicit information about velocity and uncertainty.

[0066] Corresponding to step S130, this embodiment of the application can construct a trajectory-guided BEV encoder. The goal of the trajectory-guided BEV encoder is to extract BEV features (TrajBEV) for trajectory generation from the raw sensor data, without any perceptual annotation supervision throughout the process. Figure 3 This diagram illustrates the process of extracting trajectory-guided BEV features according to an embodiment of this application.

[0067] First, such as Figure 3 As shown, embodiments of this application can process the current image and LiDAR input through a shared sensor backbone network (such as a Transfuser encoder) to generate initial BEV features. Meanwhile, the vehicle's status (such as speed, acceleration, and navigation commands) is encoded as features through a multilayer perceptron. .

[0068] Continue to refer to Figure 3 In order to decode the driving mode from these features, embodiments of this application initialize a set of sparse BEV heatmap queries. (in This involves downsampling the original BEV mesh by a factor of 4 to improve efficiency. And this is done through a Transformer encoder. This allows these queries to interact with BEV and vehicle features, thereby aggregating environmental and vehicle information. The above process is shown in equation (3) below:

[0069]

[0070] Then, continue to refer to Figure 3 Using a lightweight deconvolutional network The processed query is upsampled, and the predicted single-channel BEV heatmap is output. Its size and They have the same spatial dimensions.

[0071] Finally, the initial BEV features Compared with predicted BEV heatmap features The data is then fused to generate trajectory-guided BEV features (TrajBEV). This serves as the core condition input for the subsequent diffusion model. The fusion process is shown in equation (4):

[0072]

[0073] in This represents the general BEV environmental characteristics extracted from raw sensor data. This represents the predictive heatmap features that encode potential driving patterns. This represents a lightweight channel fusion network (such as several convolutional layers) used to efficiently fuse two types of features. The final output trajectory-guided BEV features integrate environmental perception and planning intent.

[0074] Corresponding to step S140, this embodiment of the application can construct a trajectory-guided BEV diffusion Transformer. The trajectory-guided BEV diffusion Transformer is the core of this embodiment for generating diverse trajectories. It adopts a diffusion model framework, using the aforementioned TrajBEV features and vehicle state as conditions, to directly perform denoising on a continuous trajectory space without requiring predefined trajectory anchor points or target points.

[0075] Specifically, in the diffusion process, or in the training phase, it refers to the trajectory towards the real future. Gaussian noise is gradually added during the noise step. The trajectory after adding noise Satisfy the following equation (5):

[0076]

[0077] in, , , It is a predefined noise schedule. This can be sampled using reparameterization techniques. , .

[0078] Subsequently, a TB-DiT noise prediction network was introduced. The goal is to predict the noise added to the trajectory. Its input includes noise trajectories. Conditional embedding and TrajBEV features For example, noise tracks Through a learnable trajectory encoder Mapping to latent features Conditional embedding Embedded by diffusion time step And enhanced vehicle query Adding them together, we get: .in, Through an Ego-BEV interaction module The calculation process is shown in equation (6) below:

[0079]

[0080] in, It is a learnable vehicle status query vector. It includes a cross-attention layer to inject driving mode information from TrajBEV features into the query.

[0081] TrajBEV characteristics Then, a compressor with a Q-Former structure is used to transform high-resolution BEV features into a set of compact BEV queries. .

[0082] In the embodiments of this application, the TB-DiT network can be composed of multiple identical stacked blocks, each block performing the operation shown in equation (7) in sequence:

[0083]

[0084] in, It is a temporal self-attention layer that models the dependencies between trajectory points along the time dimension. This is a BEV cross-attention layer that integrates trajectory latent features with BEV queries. Interact with the environment to align the trajectory with the surroundings. It is the trajectory decoding layer, which maps the processed latent features back to the noise prediction. (Right now (output).

[0085] During the denoising process, or in the inference phase, from random noise Initially, a pre-trained noise prediction network was used. The noise is denoised iteratively according to the following formula, and finally a clean trajectory is obtained as shown in equation (8). :

[0086]

[0087] in, , This represents the noise variance.

[0088] Since the training in this embodiment is end-to-end, the total loss function of the aforementioned network architecture can be composed of a weighted sum of the Gaussian BEV heatmap loss and the diffusion model loss.

[0089] For example, Gaussian BEV heatmap loss A focus-based loss approach can be used to encourage the network to accurately predict the heatmap target defined by formula (1). Gaussian BEV heatmap loss. It can be shown in the following formula (9):

[0090]

[0091] in, BEV heatmap predicted by the network In position The value at that location. The hyperparameter representing the focus loss is used to balance the difficulty of samples.

[0092] For example, diffusion model loss The noise prediction network can be directly optimized using mean squared error loss, as shown in equation (10) below:

[0093]

[0094] This loss forces the network to learn from noisy trajectories. The actual noise added is predicted in the middle. .

[0095] The final total loss can be weighted by the Gaussian BEV heatmap loss and the diffusion model loss, as shown in equation (11) below:

[0096]

[0097] in and The weighting coefficients are used to balance the two losses.

[0098] Furthermore, thanks to its completely non-annotation design, this application can easily improve model performance through trajectory data augmentation. For example, by resampling the starting point of future trajectories in a given scene with perturbation, the diversity of training data can be increased at low cost, enabling the model to learn driving patterns with stronger generalization. The final trained target model can be used to generate diverse trajectories for autonomous vehicles.

[0099] Subsequently, during the inference phase, accelerated sampling strategies such as denoising diffusion implicit models (DDIM) can be employed to generate high-quality future trajectories from noise. By sampling multiple noise seeds in parallel, multiple diverse trajectory candidates can be obtained at once. The final planned trajectory can then be output by selecting the optimal one or through ensemble methods to address uncertainties in driving scenarios.

[0100] During model inference, real-time sensor data and real-time vehicle status information collected by the autonomous vehicle are first acquired. Then, this real-time sensor data and real-time vehicle status information are input into the target model to obtain the predicted trajectory of the autonomous vehicle output by the target model. Specifically, the target model includes a first network architecture (see reference). Figure 3 The network architecture shown is the second network architecture.

[0101] Among them, reference Figure 3 The first network architecture can be used to determine the second BEV characteristics. The first network structure includes a first network, a second network, and a third network, the first network (i.e., Figure 3 The first encoder shown from top to bottom is used to calculate the real-time sensor data (such as...). Figure 3 The second BEV feature is determined by the camera image data and LiDAR point cloud data shown. This second network (i.e. Figure 3 The second encoder shown from top to bottom is used to determine the second encoded feature based on the real-time vehicle status information. The third network is used to determine the second predicted BEV heatmap based on the second BEV feature, the second encoded feature, and the second BEV heatmap query vector. The fusion of the second BEV feature and the second predicted BEV heatmap is used to determine the second trajectory-guided BEV feature.

[0102] Furthermore, the second network architecture is used to determine the prediction noise in the predicted trajectory based on the second trajectory-guided BEV features and the real-time vehicle state information. Consequently, by removing the prediction noise from the predicted trajectory, a more accurate trajectory prediction result can be obtained.

[0103] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0104] This application also provides a readable storage medium containing instructions that, when executed by an electronic device, cause the electronic device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0105] This application also provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0106] The hardware module of this application is described below, which can be used to implement the aforementioned method 100.

[0107] Now for reference Figure 4 The diagram shows a block diagram of a device 400 according to one embodiment of this application. Device 400 may include one or more processors 401 coupled to a controller hub 403. In at least one embodiment, the controller hub 403 communicates with the processor 401 via a multi-branch bus such as a front side bus (FSB), a point-to-point interface such as a quickpath interconnect (QPI), or a similar connection 410. The processor 401 executes instructions controlling general types of data processing operations. In one embodiment, the controller hub 403 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0108] Device 400 may also include a coprocessor 402 and a memory 404 coupled to a controller hub 403. Alternatively, one or both of the memory and the GMCH may be integrated within the processor, with memory 404 and coprocessor 402 directly coupled to processor 401 and controller hub 403, which resides on a single chip with the IOH. Memory 404 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, coprocessor 402 is a dedicated processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of coprocessor 402 are indicated by dashed lines. Figure 4 middle.

[0109] Memory 404, as a computer-readable storage medium, may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 404 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0110] In one embodiment, device 400 may further include a network interface controller (NIC) 406. NIC 406 may include a transceiver for providing a radio interface to device 400, thereby enabling communication with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, NIC 406 may be integrated with other components of device 400. NIC 406 can implement the functionality of the communication unit in the above embodiments.

[0111] Device 400 may further include input / output (I / O) device 405. I / O 405 may include: a user interface designed to enable a user to interact with device 400; a peripheral component interface designed to enable peripheral components to also interact with device 400; and / or sensors designed to determine environmental conditions and / or location information related to device 400.

[0112] It is worth noting that, Figure 4 This is merely an example. That is, although... Figure 4 The diagram shows that device 400 includes multiple devices such as processor 401, controller hub 403, and memory 404. However, in actual applications, devices using the methods of this application may include only a portion of the devices in device 400. For example, it may include only processor 401 and NIC 406. Figure 4 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 404, which is a computer-readable storage medium, stores instructions that, when executed on a computer, cause the device 400 to perform the methods according to the above embodiments. Specific details can be found in the methods of the above embodiments, and will not be repeated here.

[0113] Now for reference Figure 5 The diagram shown is a block diagram of a SoC 500 according to an embodiment of this application. Figure 5 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 5In this SoC 500, the following are included: an interconnect unit 550 coupled to an application processor 510; a system proxy unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more coprocessors 520, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 530; and a direct memory access (DMA) unit 560. In one embodiment, the coprocessor 520 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0114] The static random-access memory (SRAM) unit 530 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 500 to perform the attention training method according to the above embodiments, as detailed in the methods described above, which will not be repeated here.

[0115] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0116] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0117] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0118] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0119] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0120] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0121] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A trajectory prediction method for autonomous vehicles, characterized in that, include: Acquire real-time sensor data and real-time vehicle status information collected by the autonomous vehicle; The real-time sensor data and the real-time vehicle status information are input into the target model to obtain the predicted trajectory of the autonomous vehicle output by the target model. The target model includes a first network architecture and a second network architecture. The first network architecture is used to determine the second trajectory-guided BEV feature. The first network structure includes a first network, a second network, and a third network. The first network is used to determine the second BEV feature based on the real-time sensor data. The second network is used to determine the second coded feature based on the real-time vehicle status information. The third network is used to determine the second predicted BEV heatmap based on the second BEV feature, the second coded feature, and the second BEV heatmap query vector. The fusion of the second BEV feature and the second predicted BEV heatmap is used to determine the second trajectory-guided BEV feature. The second network architecture is used to determine the prediction noise in the predicted trajectory based on the second trajectory-guided BEV feature and the real-time vehicle status information to determine the denoised predicted trajectory.

2. A model training method for trajectory prediction of autonomous vehicles, characterized in that, include: Acquire the raw sensor data, vehicle status, and future trajectory information collected by the autonomous vehicle; Based on the future trajectory information and the vehicle speed in the vehicle state, a Gaussian bird's-eye view BEV heat map of the autonomous vehicle is constructed. The Gaussian BEV heat map is used to indicate the probability distribution of the vehicle's position in the BEV space in the future time period. Based on the original sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture, the trajectory-guided BEV features output by the first network architecture are determined. The Gaussian BEV heatmap is supervisory information. The first network architecture includes a first network, a second network, and a third network. The first network determines the first BEV feature based on the original sensor data. The second network determines the first encoded feature based on the vehicle state. The third network determines the predicted BEV heatmap based on the first BEV feature, the first encoded feature, and the first BEV heatmap query vector. The fusion of the first BEV feature and the predicted BEV heatmap is used to determine the trajectory-guided BEV feature. Based on the Gaussian BEV heatmap, the predicted BEV heatmap, the trajectory-guided BEV features, and the vehicle state, the first network architecture and the second network architecture to be trained are trained to obtain a target model for trajectory prediction of autonomous vehicles. The second network architecture is used to determine the predicted noise in the future trajectory based on the trajectory-guided BEV features and the vehicle state. The loss functions of the first network architecture and the second network architecture are determined based on a first deviation between the Gaussian BEV heatmap and the predicted BEV heatmap, and a second deviation between the predicted noise determined by the second network architecture and the real noise added to the future trajectory information.

3. The method according to claim 2, characterized in that, The step of constructing a Gaussian bird's-eye view BEV heatmap based on the future trajectory information and the vehicle speed in the vehicle state includes: For each moment in the future time domain, the mean of the Gaussian distribution is taken as the BEV position of the autonomous vehicle at each moment, and a Gaussian distribution is constructed based on the standard deviation corresponding to the vehicle speed. The Gaussian BEV heatmap is determined based on the maximum value of the Gaussian distribution at each BEV grid point at each time step.

4. The method according to claim 2 or 3, characterized in that, The step of determining the trajectory-guided BEV features output by the first network architecture based on the original sensor data, the vehicle state, the Gaussian BEV heatmap, and the first network architecture includes: The raw sensor data is processed by a first network to obtain the first BEV feature, wherein the first network includes a Transfuser encoder and the raw sensor data includes images and / or LiDAR input; The autonomous vehicle status is processed by a second network to obtain the first encoded feature, wherein the autonomous vehicle status includes one or more of the autonomous vehicle's speed, acceleration, and navigation commands; Obtain the initialized first BEV heatmap query vector; The first BEV heatmap query vector, the first BEV feature, and the first encoded feature are processed by the transformer encoder in the third network to determine the second BEV heatmap query vector. The second BEV heatmap query vector is processed by the deconvolution network in the third network to obtain the predicted BEV heatmap; The trajectory-guided BEV features are determined by fusing the predicted BEV heatmap and the first BEV features.

5. The method according to claim 4, characterized in that, The step of determining the trajectory-guided BEV features based on the fusion of the predicted BEV heatmap and the first BEV features includes: The predicted BEV heatmap and the first BEV feature input value channel are fused into a network to determine the trajectory-guided BEV features.

6. The method according to claim 2 or 3, characterized in that, The step of training the first and second network architectures to be trained based on the trajectory-guided BEV features and the vehicle state includes: Gaussian noise is added to the future trajectory information to obtain noisy trajectory information; The noisy trajectory information, the trajectory-guided BEV features, and the vehicle state are input into the second network architecture to obtain the predicted noise output by the second network architecture; The second network architecture is trained based on the deviation between the Gaussian noise and the predicted noise.

7. The method according to claim 6, characterized in that, The step of inputting the noisy trajectory information, the trajectory-guided BEV features, and the vehicle status into the second network architecture includes: The noisy trajectory information is mapped to latent features; The trajectory-guided BEV features are processed by a compressor with a Q-Former structure to obtain the BEV query vector; The conditional embedding vector is determined by the diffusion time-distribution embedding and the enhanced vehicle query in the vehicle state; The latent features, the BEV query vector, and the conditional embedding vector are input into the second network architecture.

8. An electronic device, characterized in that, It includes one or more processors; one or more memories; said one or more memories storing one or more computer programs, said one or more computer programs including instructions that, when executed by said one or more processors, cause the method as described in any one of claims 1 to 6 to be performed.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the method as described in any one of claims 1 to 6 to be performed.

10. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 6.

Images