Vehicle trajectory prediction method, system and electronic device

Abstract

The embodiment of the application provides a vehicle driving track prediction method, system and electronic equipment, the method comprises the following steps: determining the vehicle driving information corresponding to the target vehicle; obtaining the bird's eye view feature information according to the vehicle driving information; determining the query feature information, the semantic segmentation information and the target object detection result according to the vehicle driving information and the bird's eye view feature information; determining the pre-defined anchor point information, and obtaining the first driving track information corresponding to the target vehicle according to the pre-defined anchor point information and the query feature information; obtaining the more accurate second driving track information corresponding to the target vehicle according to the bird's eye view feature information, the query feature information, the semantic segmentation information, the target object detection result and the first driving track information. In this way, based on the strategy from coarse to fine, the first driving track information can be converted to the second driving track information which is relatively accurate, so that the more accurate vehicle driving track is obtained, and the accuracy of the vehicle driving track is improved.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and in particular to a method, system and electronic device for predicting vehicle driving trajectory. Background Technology

[0002] Currently, the application of vehicle-assisted driving technology is becoming increasingly widespread in vehicles, and vehicle trajectory planning is a crucial aspect of this technology. By planning the vehicle trajectory, the vehicle can travel according to the planned path, thus achieving assisted driving. For trajectory planning, the planned trajectory must ensure that it avoids collisions with static obstacles along the path, and it also needs to be adjusted in real-time based on dynamic obstacles to ensure collision avoidance. Therefore, improving the accuracy of the planned vehicle trajectory is an important research direction in this field. Summary of the Invention

[0003] This application provides a vehicle trajectory prediction method, system, and electronic device that can obtain more accurate vehicle trajectories.

[0004] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a vehicle trajectory prediction method. The method includes: determining vehicle driving information corresponding to a target vehicle; obtaining bird's-eye view feature information based on the vehicle driving information; obtaining query feature information, semantic segmentation information, and target object detection result information based on the vehicle driving information and the bird's-eye view feature information; determining predefined anchor point information; obtaining first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and the query feature information; and obtaining second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information, and the first driving trajectory information.

[0005] The above technical solution first obtains bird's-eye view feature information based on vehicle driving information. Then, based on the vehicle driving information and bird's-eye view feature information, query feature information, semantic segmentation information, and target object detection results are obtained. A first driving trajectory is obtained based on predefined anchor point information and query feature information. Finally, the first driving trajectory information is refined based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection results, and the first driving trajectory information to obtain a second driving trajectory with higher accuracy. Thus, based on a coarse-to-fine strategy, the determined driving trajectory transitions from relatively coarse to relatively precise, resulting in a more accurate vehicle driving trajectory and improving the accuracy of the vehicle driving trajectory.

[0006] In one possible implementation of the first aspect described above, the vehicle driving information includes vehicle driving status information and vehicle navigation information. Based on the vehicle driving information and bird's-eye view feature information, query feature information, semantic segmentation information, and target object detection result information are obtained, including: encoding the vehicle driving status information and vehicle navigation information to obtain representation information; obtaining query feature information based on the representation information and bird's-eye view feature information, wherein the query feature information includes target object feature information; obtaining target object detection result information based on the target object feature information; and obtaining semantic segmentation information based on the bird's-eye view feature information.

[0007] The above technical solution considers vehicle driving status information and vehicle navigation information to plan the vehicle driving trajectory so that the vehicle driving trajectory conforms to the current vehicle status and meets the current navigation requirements.

[0008] In one possible implementation of the first aspect above, obtaining target object detection result information based on target object feature information includes performing detection processing on target object feature information based on a target object detection network to obtain target object detection result information, wherein the target object detection result information is obtained in the following manner:

[0009]

[0010] in, For the target object detection results information, For the target object detection head network, This refers to the characteristic information of the target object.

[0011] In one possible implementation of the first aspect described above, the semantic segmentation information is a semantic segmentation map. The semantic segmentation information is obtained based on the bird's-eye view feature information, including segmenting the bird's-eye view feature information using a semantic segmentation network. Specifically, the semantic segmentation information is obtained in the following manner:

[0012]

[0013] in, For semantic segmentation graphs, For semantic segmentation networks, This refers to the feature information of the bird's-eye view.

[0014] In one possible implementation of the first aspect above, the query feature information further includes trajectory feature information. Based on the predefined anchor point information and the query feature information, the first driving trajectory information corresponding to the target vehicle is obtained, including: obtaining the driving trajectory query feature information based on the predefined anchor point information and the trajectory feature information; determining the anchor point offset; and obtaining the first driving trajectory information based on the driving trajectory query feature information and the anchor point offset.

[0015] By adopting the above technical solution, coarse-grained trajectory prediction can be performed in advance based on predefined anchor points. Combined with trajectory feature information, driving trajectory query feature information that meets the current vehicle driving requirements can be obtained. The driving trajectory query feature information is then corrected based on the anchor point offset, so that the obtained first driving trajectory information is relatively accurate trajectory information.

[0016] In one possible implementation of the first aspect above, the driving trajectory query feature information is obtained based on predefined anchor point information and trajectory feature information, including: performing sinusoidal position encoding processing on the predefined anchor point information to obtain anchor point position encoding information; obtaining anchor point projection feature information based on the anchor point position encoding information using a multilayer perceptron; and obtaining driving trajectory query feature information based on the anchor point projection feature information and trajectory feature information using a cross-attention mechanism.

[0017] In one possible implementation of the first aspect above, the driving trajectory query feature information is obtained in the following way:

[0018] )

[0019] in, To query feature information for driving trajectory, This is a cross-attention function that processes anchor point projection features and trajectory features based on a cross-attention mechanism. This is a multilayer perceptron processing function that processes anchor point position encoding information to obtain anchor point projection feature information. To obtain the sinusoidal position encoding function for anchor point position encoding information by performing sinusoidal position encoding processing on predefined anchor point information, Here, Q represents the trajectory feature information, K represents the query for the cross-attention function, and V represents the key of the cross-attention function.

[0020] By adopting the above technical solution, predefined anchor point information is processed based on the sinusoidal position coding processing function, and time and space information is added to the anchor point information. This makes the driving trajectory query feature information a combination of the current driving time and driving space feature information, thereby improving the accuracy of the driving trajectory.

[0021] In one possible implementation of the first aspect above, the second driving trajectory information is obtained based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, and the first driving trajectory information, including: performing iterative denoising processing on the first driving trajectory information based on a diffusion model and a multilayer perceptron to obtain the first optimized query feature information; and obtaining the second driving trajectory information based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, the first driving trajectory information, and the first optimized query feature information.

[0022] Using the above technical solution, the first driving trajectory information is iteratively denoised to obtain refined and optimized query feature information. Then, by fully considering the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, first driving trajectory information, and first optimized query feature information, a second driving trajectory information with higher accuracy is obtained.

[0023] In one possible implementation of the first aspect above, iterative denoising processing of the first driving trajectory information based on a diffusion model and a multilayer perceptron is performed to obtain first optimized query feature information, including: performing sinusoidal position encoding processing on the first driving trajectory information based on a diffusion model to obtain trajectory position encoding feature information; and obtaining the first optimized query feature information based on the trajectory position encoding feature information using a multilayer perceptron; wherein the first optimized query feature information is obtained in the following manner:

[0024]

[0025] in, To optimize query feature information, This is the first driving trajectory information. A sinusoidal position coding function is used to process the first driving trajectory information to obtain trajectory position coding feature information. This is a multilayer perceptron processing function that processes trajectory position encoding feature information based on a multilayer perceptron.

[0026] Using the above technical solution, the first driving trajectory information is processed based on a multilayer perceptron and a sinusoidal position coding function to obtain the first optimized query feature information containing important feature information.

[0027] In one possible implementation of the first aspect above, the diffusion model is obtained as follows: determining historical driving trajectory information; adding Gaussian noise to the historical driving trajectory information based on forward diffusion to obtain noisy driving trajectory information; and training the initial diffusion model based on the noisy driving trajectory information to obtain the diffusion model.

[0028] By adopting the above technical solution, Gaussian noise is gradually added to the training set of the diffusion model, and then the model is trained based on the noisy driving trajectory information, so that the final diffusion model can be better processed for iterative denoising.

[0029] In one possible implementation of the first aspect above, Gaussian noise is added to the third driving trajectory information based on a forward diffusion method to obtain noisy driving trajectory information, including obtaining the noisy driving trajectory information in the following ways:

[0030] ,

[0031] in, For driving trajectory information with noise, This is the cumulative product of the noise scheduling table parameters. Historical driving trajectory information, It is Gaussian noise that follows a standard normal distribution.

[0032] In one possible implementation of the first aspect above, semantic bird's-eye view feature information is obtained based on target object detection result information, bird's-eye view feature information, and semantic segmentation information. This includes: projecting target object detection result information and semantic segmentation information onto bird's-eye view feature information to obtain target semantic feature information, which includes drivable area semantic feature information and target object semantic feature information; projecting drivable area semantic feature information onto bird's-eye view feature information to obtain drivable area semantic bird's-eye view feature information; and projecting target object semantic feature information onto bird's-eye view feature information to obtain target object area bird's-eye view feature information.

[0033] The target semantic feature information is obtained in the following way:

[0034]

[0035] in, For target semantic feature information, This refers to the information of the i-th segmentation region corresponding to the j-th semantic category;

[0036] Furthermore, the semantic bird's-eye view feature information is obtained in the following way:

[0037]

[0038] in, This refers to the semantic bird's-eye view feature information corresponding to the i-th segmented region. For bird's-eye view feature information, This is a function that processes the bird's-eye view feature information and the target semantic feature information corresponding to the j-th semantic category.

[0039] By employing the above technical solution, semantic segmentation information is extracted and processed to obtain target semantic feature information containing only important features. This reduces the amount of data and minimizes the interference of non-important features on driving trajectory prediction. Furthermore, by projecting the target semantic feature information onto bird's-eye view feature information, semantic bird's-eye view feature information is obtained, making it more representative of the vehicle's current driving scenario.

[0040] In one possible implementation of the first aspect above, road constraint trajectory feature information is obtained based on semantic bird's-eye view feature information and first optimized query feature information, including obtaining road constraint trajectory feature information in the following manner:

[0041] )

[0042] in, For road-constrained trajectory feature information, For cross-attention function, To optimize query feature information, Q represents the semantic bird's-eye view feature information, K represents the query of the cross-attention function, V represents the key of the cross-attention function, and V represents the value of the cross-attention function.

[0043] Using the above technical solution, cross-attention fusion processing is performed based on the feature information of the bird's-eye view of the drivable area and the feature information of the first optimized query to obtain road constraint trajectory feature information that can characterize the current road conditions.

[0044] In one possible implementation of the first aspect above, obstacle avoidance constraint trajectory feature information is obtained based on semantic bird's-eye view feature information, first driving trajectory information, and road constraint trajectory feature information, including obtaining obstacle avoidance constraint trajectory feature information in the following manner:

[0045] )

[0046] in, To constrain trajectory feature information for obstacle avoidance, It is a deformable attention function. This is semantic bird's-eye view feature information. This is the spatial reference information corresponding to the first driving trajectory information. For road-constrained trajectory feature information, Q is the query for the deformable attention function, K is the key of the deformable attention function, and V is the value of the deformable attention function.

[0047] Using the above technical solution, deformable attention fusion processing is performed based on semantic bird's-eye view feature information, spatial reference information corresponding to the first driving trajectory information, and road constraint trajectory feature information to obtain obstacle avoidance constraint trajectory feature information that can characterize the current target object's motion.

[0048] In one possible implementation of the first aspect above, the second optimized query feature information is obtained based on the road constraint trajectory feature information and the obstacle avoidance constraint trajectory feature information. This includes obtaining the second optimized query feature information based on the road constraint trajectory feature information and the obstacle avoidance constraint trajectory feature information using a gating mechanism. That is, the second optimized query feature information is obtained in the following way:

[0049]

[0050]

[0051] in, It is the sigmoid activation function. For the learnable linear function used to calculate the gating coefficients, To optimize query feature information, To constrain trajectory feature information for obstacle avoidance, This provides road-constrained trajectory feature information.

[0052] By employing the above scheme, and dynamically balancing the weights of road constraint trajectory feature information and obstacle avoidance constraint trajectory feature information based on a gating mechanism, optimized query feature information that better represents vehicle trajectory prediction information is obtained. Thus, based on an adaptive gating network, global scene representation and local semantic information are dynamically fused, balancing coarse-grained contextual understanding and precise spatial adaptation.

[0053] In one possible implementation of the first aspect above, obtaining the second driving trajectory information based on the second optimized query feature information and the query feature information includes: obtaining the third optimized query feature information based on the second optimized query feature information and the trajectory feature information; obtaining the fourth optimized query feature information based on the third optimized query feature information and the target object feature information; and obtaining the second driving trajectory information and the trajectory confidence information of the second driving trajectory information based on the fourth optimized query feature information.

[0054] The above technical solution integrates the second optimized query feature information and trajectory feature information based on a cross-attention mechanism, resulting in a third optimized query feature information that can perceive navigation intent, thus avoiding conflicts between the generated driving trajectory and navigation instructions. Furthermore, the third optimized query feature information and target object feature information are integrated based on the cross-attention mechanism, resulting in a fourth optimized query feature information that can capture the movement trends of dynamic obstacles, thus preventing collisions between the generated trajectory and dynamic obstacles. Based on the fourth optimized query feature information, a multilayer perceptron outputs second driving trajectory information that conforms to road rules and collision avoidance requirements, and also outputs trajectory confidence information to evaluate the safety of the second driving trajectory. This allows the driver to intuitively see the planned driving trajectory and its safety, facilitating driver judgment.

[0055] In one possible implementation of the first aspect above, the vehicle driving information includes a vehicle perception image, and bird's-eye view feature information is obtained based on the vehicle driving information, including: obtaining bird's-eye view feature information based on the vehicle perception image using a bird's-eye view encoder.

[0056] By using the above technical solution, the vehicle perception image is encoded into bird's-eye view feature information, which can more intuitively analyze the current road conditions of the vehicle and thus obtain more accurate driving trajectory information.

[0057] Secondly, this application also discloses a vehicle trajectory prediction system, including an acquisition module, an encoder module, a perception module, and a planning module. The acquisition module is used to acquire vehicle driving information; the encoder module is used to obtain bird's-eye view feature information based on the vehicle driving information; the perception module is used to obtain query feature information, semantic segmentation information, and target object detection result information based on the vehicle driving information and the bird's-eye view feature information; the planning module is used to determine predefined anchor point information, obtain the first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and the query feature information, and obtain the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information, and the first driving trajectory information.

[0058] In one implementation of the second aspect above, the encoder module includes a bird's-eye view encoder, which is used to obtain bird's-eye view feature information based on vehicle driving information.

[0059] In one implementation of the second aspect above, the perception module includes a decoder, a dense segmentation head, and a sparse target object detection head. The decoder is used to determine query feature information based on vehicle driving information and bird's-eye view feature information; the dense segmentation head is used to determine semantic segmentation information based on bird's-eye view feature information; and the sparse target object detection head is used to determine the target object detection result based on the query feature information.

[0060] In one implementation of the second aspect above, the planning module includes a trajectory prediction decoder and a diffusion optimizer. The trajectory prediction decoder is used to determine predefined anchor point information and obtain the first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and query feature information. The diffusion optimizer is used to obtain the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information and the first driving trajectory information.

[0061] In one implementation of the second aspect described above, the perceptual losses of the decoder, target object detection head, and dense segmentation head are determined, and the perception module is optimized based on these perceptual losses. The prediction loss of the trajectory prediction encoder and the optimization loss of the trajectory optimizer are determined, and the total loss is obtained based on the perceptual loss, prediction loss, and optimization loss. The perception module and planning module are then optimized based on the total loss. Finally, the driving trajectory prediction loss is determined, and the encoder, perception module, and planning module are optimized based on the driving trajectory prediction loss and the total loss.

[0062] Thirdly, this application also discloses an electronic device for implementing the vehicle trajectory prediction method provided by any of the implementations of the first aspect.

[0063] The relevant beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0064] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0065] Figure 1 A schematic diagram of a vehicle trajectory prediction system provided in an embodiment of this application;

[0066] Figure 2 A schematic diagram illustrating another structure and principle of the vehicle trajectory prediction system provided in this application embodiment;

[0067] Figure 3 A schematic diagram illustrating the structure and principle of a trajectory optimizer provided in an embodiment of this application;

[0068] Figure 4A schematic diagram illustrating the structure and principle of a fine-grained semantic interaction module provided in an embodiment of this application;

[0069] Figure 5 A schematic flowchart of a vehicle trajectory prediction method provided in an embodiment of this application;

[0070] Figure 6 This is a schematic flowchart illustrating the processing of vehicle driving information provided in an embodiment of this application.

[0071] Figure 7 A flowchart illustrating the determination of first driving trajectory information provided in an embodiment of this application;

[0072] Figure 8 A flowchart illustrating the determination of a second driving trajectory information provided in an embodiment of this application;

[0073] Figure 9 A schematic diagram illustrating another process and principle of the vehicle trajectory prediction method provided in this application embodiment;

[0074] Figure 10 (a) in the figure is a schematic diagram showing the relationship between the driving trajectory predicted based on the prior art and the driving trajectory predicted based on the method provided in this application and the surrounding driving vehicles, respectively.

[0075] Figure 10 (b) is a schematic diagram showing the relationship between the driving trajectory predicted based on the prior art and the driving trajectory predicted based on the method provided in this application, respectively, and the roadside parked vehicles.

[0076] Figure 10 (c) in this application is a schematic diagram showing the relationship between the driving trajectory predicted based on the prior art and the driving trajectory predicted based on the method provided in this application and the current driving lane, respectively.

[0077] Figure 10 (d) in the figure is a schematic diagram showing the relationship between the driving trajectory predicted based on the prior art and the driving trajectory predicted based on the method provided in this application and the driving road surface, respectively. Detailed Implementation

[0078] In recent years, the application of vehicle-assisted driving technologies in vehicles has become increasingly widespread. For example, end-to-end autonomous driving technology (E2E-AD) has made significant progress. End-to-end autonomous driving technology can directly map raw sensor inputs into driving actions or driving trajectories. Traditional vehicle driving trajectories rely on separate perception, prediction, and planning modules, while the end-to-end approach optimizes the entire driving process in a unified way, improves the robustness between modules, and simplifies the deployment process.

[0079] For example, existing methods for predicting autonomous vehicle driving trajectories typically perform a single regression or classification of a driving trajectory database based on perceived features or camera input to obtain a deterministic trajectory. While these regression-based methods are computationally efficient, they have fundamental limitations in handling the multimodal characteristics of trajectory prediction tasks. Regression prediction averages multimodal characteristics, leading to decreased accuracy, especially in complex intersection scenarios, where it fails to predict the optimal trajectory for the vehicle. Therefore, using these methods to predict vehicle trajectories in real-world driving scenarios cannot yield accurate vehicle trajectories.

[0080] Based on this, this application proposes a vehicle trajectory prediction method. After obtaining bird's-eye view feature information from vehicle driving information, and then obtaining query feature information, semantic segmentation information, and target object detection result information from the vehicle driving information and bird's-eye view feature information, the method first obtains the first driving trajectory information corresponding to the target vehicle based on predefined anchor point information and query feature information. Then, a coarse-grained trajectory prediction result is generated based on the predefined anchor points combined with the query feature information. Further, a second driving trajectory information corresponding to the target vehicle is obtained based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, and the first driving trajectory information. Thus, the first driving trajectory information is used as structured prior information to provide high-level guidance for subsequent fine-grained optimization, resulting in a fine-grained trajectory prediction result. Therefore, based on the coarse-to-fine trajectory prediction strategy, the final predicted driving trajectory is a more accurate vehicle driving trajectory, improving the accuracy of the vehicle driving trajectory.

[0081] The vehicle trajectory prediction method and system proposed in this application will be described in detail below.

[0082] like Figure 1 As shown, the vehicle trajectory prediction system provided by this application includes an acquisition module, an encoder module, a perception module, and a planning module.

[0083] The acquisition module is used to determine vehicle driving information.

[0084] The encoder module is used to determine the feature information of the bird's-eye view (BEV) obtained based on the vehicle's driving information.

[0085] The perception module is used to obtain query feature information, semantic segmentation information, and target object detection results based on vehicle driving information and bird's-eye view feature information.

[0086] The planning module is used to determine predefined anchor point information, obtain the first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and query feature information, and obtain the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information and the first driving trajectory information.

[0087] In one implementation of this application, the acquisition module includes a vehicle camera, a vehicle speed sensor, and a navigation information collector. Vehicle driving information includes vehicle perceived images, vehicle driving status information, and vehicle navigation information; the vehicle driving status information includes vehicle speed information and acceleration information.

[0088] Among them, vehicle cameras are used to collect vehicle perception images (Raw Cameras), such as raw images captured by a vehicle's 360-degree camera.

[0089] Vehicle speed sensors are used to acquire vehicle speed information, such as reading vehicle speed information from the vehicle's dashboard.

[0090] Accelerometers are used to acquire vehicle acceleration information.

[0091] The navigation information collector is used to collect vehicle navigation information, such as reading instructions on the vehicle navigation map, such as instructions for the vehicle to perform, turn left, turn right, etc., as well as road-level navigation routes issued by the navigation map.

[0092] It should be noted that the acquisition module may also include other sensors, and the vehicle driving information may also include other information related to vehicle driving trajectory prediction. The vehicle driving status information may also include information such as vehicle steering angle and heading angle.

[0093] The encoder module includes an image encoder.

[0094] Vehicle perception images captured by devices such as vehicle cameras are input into an image encoder, which then extracts image features from the vehicle perception images to obtain image feature information.

[0095] Furthermore, such as Figure 2 As shown, the encoder module also includes a bird's-eye view encoder (BEV Encoder). Image feature information is input into the bird's-eye view encoder so that the encoder can generate bird's-eye view feature information based on the image feature information. The bird's-eye view feature information is three-dimensional feature information.

[0096] The perception module includes a Transformer decoder, a sparse target object detection head (Det Head), and a dense segmentation head (Seg Head).

[0097] The Transformer decoder is used to obtain query feature information based on bird's-eye view feature information (BEV features) and vehicle driving information, including vehicle driving status information (Status) and vehicle navigation information (Command). The query feature information (Updated Query, i.e., the new Query) includes target object feature information and trajectory feature information.

[0098] For example, the target object feature information is the target object token (Agent Token), and the trajectory feature information is the trajectory token (Planning Token).

[0099] The Transformer decoder is equipped with learnable query features, enabling it to obtain updated query features based on bird's-eye view features, vehicle driving status information, and vehicle navigation information.

[0100] The sparse target object detection head is used to obtain target object detection results based on the target object feature information.

[0101] The target object detection result information includes the target object's location and category. Target objects include obstacles such as vehicles, pedestrians, roadblocks, and buildings.

[0102] The dense segmentation head is used to obtain semantic segmentation information based on the bird's-eye view feature information. The semantic segmentation information is a full-scene semantic understanding of the bird's-eye view feature information, which can realize the environmental segmentation of target objects such as road elements, dynamic obstacles and static obstacles.

[0103] The planning module includes a trajectory prediction decoder and a diffusion optimizer.

[0104] The trajectory prediction decoder is used to determine predefined anchor point information. Based on the predefined anchor point information and the trajectory feature information included in the query feature information, the first driving trajectory information (predicted trajectory) corresponding to the target vehicle is obtained.

[0105] For example, the trajectory prediction decoder obtains the driving trajectory query feature information based on predefined anchor point information and query feature information, determines the anchor point offset, adjusts the driving trajectory query feature information (i.e., adjusts the anchor point) based on the anchor point offset, and obtains the first driving trajectory information (i.e., the predicted trajectory).

[0106] The predefined anchor point information consists of trajectory point information clustered based on offline clustering methods for different directions of vehicle travel. For example, it includes trajectory point information clustered for different travel trajectories such as left turn trajectory, left front travel trajectory, straight travel trajectory, right front travel trajectory, right turn trajectory, and U-turn trajectory.

[0107] The diffusion optimizer is used to obtain the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, and first driving trajectory information.

[0108] For example, the diffusion optimizer performs iterative denoising processing on the first driving trajectory information based on the diffusion model and multi-layer perceptron (MLP) to obtain the first optimized query feature information. It then performs feature extraction based on the bird's-eye view feature information and semantic segmentation information to obtain semantic bird's-eye view feature information (semantic BEV features). Finally, based on the semantic bird's-eye view feature information, query feature information, target object detection result information, the first driving trajectory information, and the first optimized query feature information, the second driving trajectory information is obtained.

[0109] Furthermore, such as Figure 2 As shown, the diffusion and sampling are also performed based on the driving trajectory determined by the trajectory prediction decoder to train the diffusion model.

[0110] like Figure 3 As shown, the diffusion optimizer is an optimizer based on the diffusion model, which is also a fine-grained denoising decoder. By designing a fine-grained denoising decoder, the predicted trajectory can be accurately adapted to the surrounding environment through constraint expansion calculation.

[0111] The fine-grained denoising decoder includes an encoder, a bird's-eye view interaction module (BEV interaction module), a fine-grained semantic interaction module (FGSIM), and a decoder.

[0112] The encoder is a fine-grained denoising encoder based on a diffusion model, which is used to encode the first driving trajectory information (Proposal) based on the diffusion model to obtain the first optimized query feature information (Refiner Query), and then send the first optimized query feature information to the bird's-eye view modulation module.

[0113] The first driving trajectory information is the noisy driving trajectory information obtained based on the predictive decoder, and the diffusion model is the model that can denoise the driving trajectory information after model training based on the noisy driving trajectory information.

[0114] The bird's-eye view interaction module is used to extract spatial feature information (Spatial BEV) based on trajectory from the bird's-eye view feature information, obtain the driving trajectory spatial feature information, and initialize the fine-grained semantic interaction module based on the first optimized query feature information and the driving trajectory spatial feature information.

[0115] The fine-grained semantic interaction module optimizes the first-order query feature information through two consecutive optimization stages.

[0116] The first stage is the road perception optimization stage, such as... Figure 4 As shown, the fine-grained semantic interaction module includes a feature projector (as an example of a feature projection module), which obtains semantic bird's-eye view feature information (Semantic BEVFeature) based on the target object detection result information, bird's-eye view feature information and semantic segmentation information. The semantic bird's-eye view feature information includes drivable area bird's-eye view feature information and target object area bird's-eye view feature information.

[0117] In this process, the feature projector first projects the target object detection result information and semantic segmentation information onto the bird's-eye view features to obtain target semantic feature information. This target semantic feature information includes drivable area semantic feature information and target object semantic feature information. The drivable area semantic feature information is then projected onto the bird's-eye view feature information to obtain drivable area semantic bird's-eye view feature information (e.g., ...). Figure 3 The medium-fine granular semantic interaction module (driving area) projects the semantic feature information of the target object onto the bird's-eye view feature information to obtain the bird's-eye view feature information of the target object area (e.g., Figure 3 Medium-fine granular semantic interaction module (traffic participants)).

[0118] Furthermore, based on the cross-attention mechanism (Cross-Attn), road constraint trajectory feature information is obtained from the semantic bird's-eye view feature information and the first optimized query feature information.

[0119] The second stage is interactive perception optimization, which uses the Deformable Attention (Deform-Attn) mechanism to obtain obstacle avoidance constraint trajectory feature information based on semantic bird's-eye view feature information, first driving trajectory information, and road constraint trajectory feature information.

[0120] Furthermore, based on the gate fusion mechanism, the second optimized query feature information (Updated Refiner Query) is obtained by combining the road constraint trajectory feature information and the obstacle avoidance constraint trajectory feature information.

[0121] In the implementation of this application, since map-based reasoning and interaction-based reasoning are two autonomous planning stages, a fine-grained semantic interaction module is introduced to explicitly process the drivable area of ​​the target vehicle and the interaction between the target vehicle and the target object. By aligning the trajectory optimizer with map semantic information and the interaction with the target object, the performance of the trajectory optimizer is gradually enhanced. At the same time, the entire global scene context and local structural cues are layered during the trajectory prediction process to improve scene understanding capabilities.

[0122] Furthermore, continue as Figure 3 As shown, after obtaining the second optimized query feature information, the third optimized query feature information is obtained based on the target object feature information and the second optimized query feature information using the cross-attention mechanism (Cross-Attn).

[0123] Based on the cross-attention mechanism, the fourth optimized query feature information is obtained from the trajectory feature information and the third optimized query feature information.

[0124] The decoder obtains the second driving trajectory information (Refinement) and the trajectory confidence information of the second driving trajectory information based on the fourth optimized query feature information.

[0125] Thus, based on a two-layer cross-attention layer to capture the relationships between target objects and the motion constraints of target vehicles, further optimized query trajectory information is obtained. Finally, the parallel multilayer perceptron outputs the optimized trajectory information and its confidence score, ensuring the physical feasibility and contextual consistency of the trajectory.

[0126] In the implementation method of this application, such as Figure 2As shown, the vehicle trajectory prediction system provided in this application integrates three key components—a perception module, a trajectory prediction decoder, and a diffusion optimizer based on a diffusion model—within a unified "coarse-to-fine" trajectory planning architecture. This results in a second driving trajectory that more closely resembles the vehicle's actual future driving trajectory compared to the first trajectory, thus better meeting the driver's needs. Assisted driving based on this second trajectory allows for better obstacle avoidance, safer driving, and an improved user experience. Since the vehicle's actual future driving trajectory is the optimal route, predicting the vehicle's future trajectory using the system provided in this application yields predictions close to the optimal result.

[0127] Furthermore, such as Figure 5 As shown, the vehicle trajectory prediction method provided in this application includes the following steps.

[0128] S100, determine the vehicle driving information corresponding to the target vehicle.

[0129] S200 obtains bird's-eye view feature information based on vehicle driving information.

[0130] S300 obtains query feature information, semantic segmentation information, and target object detection results based on vehicle driving information and bird's-eye view feature information.

[0131] S400: Determine predefined anchor point information, and obtain the first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and query feature information.

[0132] S500 obtains the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, and first driving trajectory information.

[0133] The vehicle trajectory prediction method provided in this application is a vehicle trajectory prediction method for end-to-end autonomous driving (i.e., assisted driving). It is applied to a vehicle trajectory prediction model to directly receive raw sensor input and directly predict the vehicle's future trajectory. The vehicle trajectory is denoted as:

[0134]

[0135] Where T is the prediction horizon (i.e., the length of time for prediction). This represents the trajectory state of the target vehicle at time i, including vehicle position and heading angle information.

[0136] The trajectory prediction model includes an encoder, a perception module, and a planning module.

[0137] First, execute step S100 to determine the vehicle driving information corresponding to the target vehicle.

[0138] In one implementation of this application, the vehicle driving information includes vehicle perception images, vehicle driving status information, and vehicle navigation information.

[0139] For example, the acquisition module acquires vehicle perception images, vehicle driving status information, and vehicle navigation information.

[0140] Next, step S200 is executed to obtain bird's-eye view feature information based on vehicle driving information.

[0141] In one implementation of this application, obtaining bird's-eye view feature information based on vehicle driving information includes: obtaining bird's-eye view feature information based on the vehicle perception image using a bird's-eye view encoder.

[0142] For example, the bird's-eye view encoder takes the vehicle perception image as input, and outputs bird's-eye view feature information after image feature extraction, viewpoint transformation processing, and feature fusion processing. The bird's-eye view feature information is, for example, a bird's-eye view feature map (i.e., a BEV feature map). The dimensions of the bird's-eye view feature map are [B, C, H, W], where B is the batch size, C is the number of feature channels, H is the height pixels under the BEV viewpoint, and W is the width pixels under the BEV viewpoint. The BEV feature map includes global spatial information of road elements, dynamic obstacles, and static obstacles.

[0143] Next, step S300 is executed to obtain query feature information, semantic segmentation information, and target object detection result information based on vehicle driving information and bird's-eye view feature information.

[0144] For example, the Transformer decoder obtains query feature information based on vehicle driving information and bird's-eye view feature information. The query feature information includes target object feature information and trajectory feature information. The sparse target object detection head obtains target object detection result information based on target object feature information. The dense segmentation head obtains semantic segmentation information based on bird's-eye view feature information.

[0145] In the implementation method of this application, such as Figure 6 As shown, based on vehicle driving information and bird's-eye view feature information, query feature information, semantic segmentation information, and target object detection result information are obtained, including the following steps.

[0146] S310 encodes and processes vehicle driving status information and vehicle navigation information to obtain representation information.

[0147] For example, in the sparse obstacle calculation process, in order to incorporate vehicle driving state information, the target vehicle's driving state information and the navigation instructions included in the vehicle navigation information are encoded into compact latent representation information.

[0148] S320: Based on the representation information and the bird's-eye view feature information, the query feature information is obtained, which includes the target object feature information and the trajectory feature information.

[0149] For example, representation information is combined with scene context, and a Transformer-based decoder acts on a set of learnable queries Q to obtain query feature information based on representation information and bird's-eye view feature information.

[0150] The query feature information is the query of the target object and the query of the trajectory, which is also the target object token. and trajectory tokens .

[0151] The target object token contains contextual information about the target object, such as target object 1 being a vehicle and target object 2 being a pedestrian.

[0152] The trajectory token contains scene information about the driving trajectory ahead, such as information that a left turn is required at the upcoming intersection.

[0153] S330: Based on the target object feature information, obtain the target object detection result information.

[0154] For example, a sparse obstacle detection head is used to detect a single target object, and the feature information of the target object is processed to detect obstacles around the target vehicle.

[0155] In one implementation of this application, obtaining target object detection result information based on target object feature information includes processing the target object feature information through a target object detection head network to obtain target object detection result information.

[0156] The target object detection results are obtained in the following way:

[0157]

[0158] in, For the target object detection results information, For the target object detection head network, This refers to the characteristic information of the target object.

[0159] S340: Based on the feature information of the bird's-eye view, semantic segmentation information is obtained.

[0160] For example, dense segmentation heads are used to achieve comprehensive scene understanding in order to segment semantic information of target objects such as road elements, dynamic obstacles, and static obstacles.

[0161] It should be noted that steps S330 and S340 can be executed sequentially or simultaneously.

[0162] In one implementation of this application, the semantic segmentation information is a semantic segmentation map. The semantic segmentation information is obtained based on the bird's-eye view feature information, including a dense segmentation head that performs transformation processing on the bird's-eye view feature information based on a semantic segmentation network to obtain the semantic segmentation information.

[0163] The semantic segmentation information is obtained in the following way:

[0164]

[0165] in, For semantic segmentation graphs, For semantic segmentation networks, This refers to the feature information of the bird's-eye view.

[0166] Next, step S400 is executed to determine the predefined anchor point information. Based on the predefined anchor point information and the query feature information, the first driving trajectory information corresponding to the target vehicle is obtained.

[0167] For example, a set of predefined trajectory anchor points is designed, and driving trajectory query feature information is obtained based on the predefined anchor point information and trajectory feature information. The anchor point offset is determined, and the first driving trajectory information is obtained based on the anchor point offset and driving trajectory query feature information.

[0168] In one implementation of this application, such as Figure 7 As shown, the first driving trajectory information corresponding to the target vehicle is obtained based on predefined anchor point information and query feature information, including the following steps.

[0169] S410: Based on predefined anchor point information and trajectory feature information, obtain driving trajectory query feature information.

[0170] For example, based on predefined anchor point information and trajectory feature information, the driving trajectory query feature information is obtained, including: performing sinusoidal position encoding processing on the predefined anchor point information to obtain anchor point position encoding information.

[0171] For example, the predefined anchor point trajectory information is a set of discrete trajectories obtained through offline clustering methods (such as K-means clustering). Each anchor point trajectory is position-encoded to inject temporal and spatial information, resulting in anchor point position-coded information. Offline clustering methods reduce online computation and adapt to the real-time requirements of assisted driving. A sinusoidal position encoding function is used to inject temporal information into the anchor point information, ensuring the capture of the temporal order of the trajectories.

[0172] Based on the anchor point position encoding information, the anchor point projection feature information is obtained using a multilayer perceptron.

[0173] For example, the initial trajectory query information, i.e., the anchor point projection feature information, is formed by the multilayer perceptron (MLP) based on the anchor point position encoding information.

[0174] Based on the cross-attention mechanism, the driving trajectory query feature information is obtained by using anchor point projection feature information and trajectory feature information.

[0175] For example, context-related processing is performed on anchor point projection feature information and trajectory feature information based on cross-attention to obtain context-enhanced driving trajectory query feature information.

[0176] The driving trajectory query feature information is obtained through the following methods:

[0177] )

[0178] in, To query feature information for driving trajectory, This is a cross-attention function that processes anchor point projection features and trajectory features based on a cross-attention mechanism. This is a multilayer perceptron processing function that processes anchor point position encoding information to obtain anchor point projection feature information. To obtain the sinusoidal position encoding function for anchor point position encoding information by performing sinusoidal position encoding processing on predefined anchor point information, For predefined anchor point information, Here, Q represents the trajectory feature information, K represents the query for the cross-attention function, and V represents the key of the cross-attention function.

[0179] S420, determine the anchor point offset, and obtain the first driving trajectory information by querying the feature information and anchor point offset based on the driving trajectory.

[0180] For example, anchor point offsets are predicted based on a lightweight Transformer to correct driving trajectory query feature information. Obtain the first driving trajectory information The first driving trajectory information dimension is [B, T, 3], where T is the prediction time domain and 3 represents the x, y coordinates and heading angle. A lightweight Transformer is used to predict anchor point offsets, avoiding issues of insufficient or redundant fixed anchor point coverage.

[0181] Next, step S500 is executed to obtain the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, and the first driving trajectory information.

[0182] For example, semantic bird's-eye view feature information is obtained based on bird's-eye view feature information and semantic segmentation information; first optimized query feature information is obtained based on first driving trajectory information; drivable area bird's-eye view feature information and target object area bird's-eye view feature information are determined from semantic bird's-eye view feature information based on target object detection result information; road constraint trajectory feature information is obtained based on drivable area bird's-eye view feature information and first optimized query feature information; obstacle avoidance constraint trajectory feature information is obtained based on target object area bird's-eye view feature information, first driving trajectory information, and road constraint trajectory feature information; second optimized query feature information is obtained based on road constraint trajectory feature information and obstacle avoidance constraint trajectory feature information; third optimized query feature information is obtained based on second optimized query feature information and trajectory feature information; fourth optimized query feature information is obtained based on third optimized query feature information and target object feature information; and second driving trajectory information and trajectory confidence information of second driving trajectory information are obtained based on fourth optimized query feature information.

[0183] In one implementation of this application, such as Figure 8 As shown, the second driving trajectory information is obtained based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, and the first driving trajectory information, including the following steps.

[0184] S510, based on the diffusion model and multilayer perceptron, iteratively denoises the first driving trajectory information to obtain the first optimized query feature information.

[0185] For example, regarding the first driving trajectory information Iterative denoising and semantic constraint optimization are performed to generate the final vehicle trajectory that conforms to physical laws and is collision-free.

[0186] Specifically, the first driving trajectory information is iteratively denoised based on the diffusion model and the multilayer perceptron to obtain the first optimized query feature information, including: sinusoidal position encoding processing of the first driving trajectory information based on the diffusion model to obtain trajectory position encoding feature information, and obtaining the first optimized query feature information based on the trajectory position encoding feature information using the multilayer perceptron.

[0187] The first optimized query feature information is obtained in the following way:

[0188]

[0189] in, To optimize query feature information, This is the first driving trajectory information. A sinusoidal position coding function is used to process the first driving trajectory information to obtain trajectory position coding feature information. This is a multilayer perceptron processing function that processes trajectory position encoding feature information based on a multilayer perceptron.

[0190] Furthermore, in the implementation of this application, the diffusion model is obtained in the following way: determining historical driving trajectory information, using the historical driving trajectory information as the noise-adding object, adding Gaussian noise to the historical driving trajectory information based on forward diffusion to obtain noisy driving trajectory information, and training the initial diffusion model based on the noisy driving trajectory information to obtain the diffusion model.

[0191] Among them, the historical driving trajectory information includes the first historical driving trajectory information and / or the second historical driving trajectory information.

[0192] Among these methods, Gaussian noise is added to historical driving trajectory information based on forward diffusion to obtain noisy driving trajectory information, including obtaining noisy driving trajectory information through the following methods:

[0193] ,

[0194] in, For driving trajectory information with noise, This is the cumulative product of the parameters of the Noise Schedule. Historical driving trajectory information, It is Gaussian noise that follows a standard normal distribution.

[0195] For example, during training, the process is gradually advanced through T steps. Add Gaussian noise to generate a noisy driving trajectory. And then train the diffusion model according to Calculate and optimize query feature information.

[0196] Among them, the diffusion model is based on Calculating and optimizing query feature information includes:

[0197]

[0198] in, To optimize the query feature information, For driving trajectory information with noise, To obtain the trajectory position coding feature information by performing sinusoidal position coding processing on noisy driving trajectory information, a sinusoidal position coding function is used. This is a multilayer perceptron processing function that processes trajectory position encoding feature information based on a multilayer perceptron.

[0199] Thus, based on the trained diffusion model and the multilayer perceptron, the noisy first driving trajectory information can be encoded into the first optimized query feature information.

[0200] S520: Based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, first driving trajectory information, and first optimized query feature information, the second driving trajectory information is obtained.

[0201] For example, the bird's-eye view interaction module extracts spatial feature information of the bird's-eye view trajectory (i.e., driving trajectory spatial feature information) from the bird's-eye view feature map, and then initializes the fine-grained semantic interaction module based on the driving trajectory spatial feature information and the first optimized query feature information.

[0202] Furthermore, the second optimized query trajectory information is obtained based on the fine-grained semantic interaction module and the decoder.

[0203] In one implementation of this application, the second driving trajectory information is obtained based on the bird's-eye view feature information, query feature information, semantic segmentation information, target object detection result information, first driving trajectory information, and first optimized query feature information, including the following steps.

[0204] S521. Based on the target object detection results, bird's-eye view feature information, and semantic segmentation information, semantic bird's-eye view feature information is obtained.

[0205] In the implementation of this application, semantic bird's-eye view feature information is obtained based on target object detection result information, bird's-eye view feature information, and semantic segmentation information. This includes: projecting target object detection result information and semantic segmentation information onto bird's-eye view feature information to obtain target semantic feature information, wherein the target semantic feature information includes drivable area semantic feature information and target object semantic feature information; projecting drivable area semantic feature information onto bird's-eye view feature information to obtain drivable area semantic bird's-eye view feature information; and projecting target object semantic feature information onto bird's-eye view feature information to obtain target object area bird's-eye view feature information.

[0206] For example, to achieve targeted interaction, target object detection results and semantic segmentation information are projected onto bird's-eye view feature information. This allows the bird's-eye view feature information to reflect the target object and semantic segmentation information, thereby enabling more accurate extraction of semantically critical regions from the semantic segmentation information based on the projected bird's-eye view feature information. For instance, this involves extracting semantic features of the target object and drivable areas, providing region-level guidance for subsequent reasoning. To identify key areas highly relevant to subsequent planning, target object and semantic segmentation information are utilized... By working together, we can more intuitively extract semantically meaningful regions of interest from semantic segmentation information based on the feature information of the projected bird's-eye view, and obtain the target semantic feature information.

[0207] In one implementation of this application, the semantic segmentation information includes semantic category information and segmentation region information corresponding to the semantic category information.

[0208] Specifically, the target object's category and location information, as well as the semantic category information and corresponding segmentation region information included in the semantic segmentation information, are projected onto the bird's-eye view. Based on the projected bird's-eye view, the target object semantic feature information corresponding to the target object and the drivable area semantic feature information corresponding to the drivable area in the semantic segmentation information are selected.

[0209] The target semantic feature information is obtained in the following way:

[0210]

[0211] in, For target semantic feature information, Let I be the segmentation region information corresponding to the j-th semantic category, where I is the number of segmentation regions and J is the number of categories of the target object.

[0212] Here, the target semantic feature information is the target segmentation region information corresponding to the j-th semantic category. The semantic category is the category information of different semantic categories (e.g., vehicles, pedestrians, traffic lights, buildings, road surfaces) generated by dividing the target object and drivable area in the semantic segmentation information. The segmented region is the region corresponding to different semantic categories.

[0213] Furthermore, based on the aforementioned method, after filtering the semantic segmentation map using a specific semantic mask, the obtained target semantic feature information is projected into the BEV feature space to obtain semantic bird's-eye view feature information.

[0214] In one implementation of this application, the semantic feature information of the drivable area and the semantic feature information of the target object can be projected onto the bird's-eye view feature information to obtain the semantic bird's-eye view feature information of the drivable area and the bird's-eye view feature information of the target object area.

[0215] First, the semantic feature information of the drivable area can be projected onto the bird's-eye view feature information to obtain the semantic bird's-eye view feature information of the drivable area. Then, the semantic feature information of the target object can be projected onto the bird's-eye view feature information to obtain the bird's-eye view feature information of the target object area.

[0216] The semantic bird's-eye view feature information is obtained in the following way:

[0217]

[0218] in, This is semantic bird's-eye view feature information. For bird's-eye view feature information, This is a function that processes the bird's-eye view feature information and the target semantic feature information corresponding to the j-th semantic category.

[0219] The semantic bird's-eye view feature information includes the semantic bird's-eye view feature information of the drivable area and the bird's-eye view feature information of the target object area.

[0220] S523, based on the bird's-eye view feature information of the drivable area and the first optimized query feature information, the road constraint trajectory feature information is obtained.

[0221] For example, semantic bird's-eye view feature information is processed based on a cross-attention mechanism. The cross-attention mechanism aims to jointly capture the global scene context and local geometric details.

[0222] In this implementation, road constraint trajectory feature information is obtained based on semantic bird's-eye view feature information and first optimized query feature information, including obtaining road constraint trajectory feature information through the following methods:

[0223] )

[0224] in, For road-constrained trajectory feature information, For cross-attention function, To optimize query feature information, Q represents the semantic bird's-eye view feature information, K represents the query of the cross-attention function, V represents the key of the cross-attention function, and V represents the value of the cross-attention function.

[0225] For example, the first optimized query feature information is used as the query for the cross-attention mechanism, and the semantic bird's-eye view feature information is used as the key and value of the cross-attention mechanism to obtain road constraint trajectory feature information. The road constraint trajectory feature information mainly reflects the static road structure, which is the aforementioned map-based reasoning. Thus, based on the cross-attention layer, a dense association is established between the trajectory features and the semantic region of the bird's-eye view to obtain the global overall context and scene-level dependencies.

[0226] S524. Obtain obstacle avoidance constraint trajectory feature information based on semantic bird's-eye view feature information, first driving trajectory information, and road constraint trajectory feature information.

[0227] For example, semantic bird's-eye view feature information is processed based on a deformable attention mechanism. Deformable attention mechanisms are designed to focus on regions that are spatially relevant to the trajectory.

[0228] In this application, obstacle avoidance constraint trajectory feature information is obtained based on the target object's bird's-eye view area feature information, the first driving trajectory information, and the road constraint trajectory feature information. This includes obtaining the obstacle avoidance constraint trajectory feature information through the following methods:

[0229] )

[0230] in, To constrain trajectory feature information for obstacle avoidance, It is a deformable attention function. This is semantic bird's-eye view feature information. This is the spatial reference information corresponding to the first driving trajectory information. For road-constrained trajectory feature information, Q is the query for the deformable attention function, K is the key of the deformable attention function, and V is the value of the deformable attention function.

[0231] For example, based on a deformable attention mechanism, by fully considering the road constraint trajectory feature information, the semantic bird's-eye view feature information including the target object bird's-eye view region feature information, and the adaptive spatial reference information of the first driving trajectory information, obstacle avoidance constraint trajectory feature information is obtained. This obstacle avoidance constraint trajectory feature information can reflect the behavior of dynamic traffic participants, which is the aforementioned interaction-based reasoning. Thus, based on the deformable attention layer, trajectory endpoints are selectively aligned with key region semantics, extracting fine-grained local structures and interaction cues.

[0232] S525, based on the road constraint trajectory feature information and the obstacle avoidance constraint trajectory feature information, the fused second optimized query feature information is obtained.

[0233] For example, the road constraint trajectory feature information and obstacle avoidance constraint trajectory feature information are fused based on the gating fusion mechanism to obtain the fused second optimized query feature information.

[0234] In this application's implementation, the second optimized query feature information is obtained based on the road constraint trajectory feature information and the obstacle avoidance constraint trajectory feature information, including obtaining the second optimized query feature information through the following methods:

[0235]

[0236]

[0237] in, It is the sigmoid activation function. For the learnable linear function used to calculate the gating coefficients, To optimize query feature information, To constrain trajectory feature information for obstacle avoidance, This provides road-constrained trajectory feature information.

[0238] when When the value approaches 1, road constraint trajectory feature information is prioritized. When the value is close to 0, obstacle avoidance constraint trajectory feature information is given priority.

[0239] For example, by dynamically balancing the weights of road constraint trajectory feature information and obstacle avoidance constraint trajectory feature information through an adaptive gating network, optimized query feature information that can better represent vehicle trajectory prediction information is obtained. Thus, based on the adaptive gating network, global scene representation and local semantic information are dynamically fused, balancing coarse-grained contextual understanding and precise spatial adaptation.

[0240] S530, based on the second optimized query feature information and the query feature information, obtain the second driving trajectory information.

[0241] For example, the second driving trajectory information is obtained based on the second optimized query feature information, target object feature information and trajectory feature information using a two-layer attention mechanism.

[0242] In one implementation of this application, the second driving trajectory information is obtained based on the second optimized query feature information and the query feature information, including the following steps.

[0243] S531, based on the second optimized query feature information and trajectory feature information, the third optimized query feature information is obtained.

[0244] For example, the second optimized query feature information and trajectory feature information are fused based on the cross-attention mechanism, so that the resulting third optimized query feature information can be information that can perceive navigation intent, thus avoiding conflicts between the generated driving trajectory and navigation instructions.

[0245] S532, based on the third optimized query feature information and the target object feature information, the fourth optimized query feature information is obtained.

[0246] For example, the third optimized query feature information and the target object feature information are fused based on the cross-attention mechanism, so that the resulting fourth optimized query feature information can capture the movement trend of dynamic obstacles and avoid the generated trajectory from colliding with the dynamic obstacles.

[0247] S533, based on the fourth optimized query feature information, obtain the second driving trajectory information and the trajectory confidence information of the second driving trajectory information.

[0248] For example, based on the multilayer perceptron and the fourth optimized query feature information, the second driving trajectory information that conforms to road rules and collision avoidance requirements is output, and a probability value for evaluating the trajectory safety of the second driving trajectory information, such as collision avoidance success rate, road compliance and other information, is output.

[0249] In this implementation, the perceptual loss of the decoder, the target object detection head, and the dense segmentation head is also calculated, using a Transfuser-style perceptual loss ( The decoder, target object detection head, and dense segmentation head are pre-trained until the number of training iterations reaches the limit or the loss function converges, at which point the training ends, in order to improve the accuracy of semantic segmentation and target object detection.

[0250] In the implementation of this application, the prediction loss of the trajectory prediction encoder is also calculated. And the optimization loss of the trajectory optimizer And obtain the total loss:

[0251]

[0252] in, For the total loss, To predict losses, To optimize losses, To perceive loss.

[0253] In this implementation, end-to-end joint optimization of perception and planning is performed, and a "winner-takes-all" strategy is adopted to ensure that the final vehicle trajectory is the closest to the actual trajectory. The trajectory prediction loss of the target vehicle's trajectory is also calculated.

[0254]

[0255] in, For L1 regression loss, For binary cross-entropy classification loss, and These are the corresponding weighting coefficients.

[0256] When all losses converge, it indicates that the obtained second driving trajectory information is the optimal driving trajectory.

[0257] The vehicle trajectory prediction method proposed in this application uses a Transformer decoder trained with bird's-eye view, vehicle driving status information, and vehicle navigation information as input to enhance scene understanding. A coarse-grained trajectory prediction decoder then employs a lightweight Transformer to adjust anchor point information, generating an initial trajectory prediction result (i.e., the first trajectory information). A fine-grained trajectory optimizer based on a diffusion model iteratively denoises and refines the initial trajectory prediction result, generating an optimized trajectory (i.e., the second trajectory information) that better captures the complexity of real-world driving. Thus, by optimizing the adaptability of the trajectory to the current real-world scene, the resulting second trajectory information is a more accurate and superior trajectory prediction result.

[0258] The vehicle trajectory prediction method provided in this application is actually a coarse-to-fine trajectory planning method based on semantic interaction and diffusion optimization for end-to-end autonomous driving scenarios. To implement this method, a two-stage trajectory prediction framework (DiffRefiner) with a coarse-to-fine architecture is proposed. In the first stage, a Transformer-based trajectory prediction decoder regresses from a set of predefined anchor points to generate coarse-grained trajectory prediction results. These results serve as structured prior information, providing high-level guidance for subsequent fine-grained optimization. In the second stage, a conditional diffusion optimizer iteratively optimizes the initial prediction results through a generative diffusion process, thereby capturing detailed trajectory information and obtaining fine-grained trajectory prediction results. This trajectory generation method, which integrates discriminative trajectory prediction and generative trajectory optimization, significantly improves the performance of trajectory planning based on the diffusion model.

[0259] Furthermore, such as Figure 9As shown, the vehicle trajectory prediction method provided in this application includes obtaining bird's-eye view feature information from a vehicle perception image, generating semantic bird's-eye view feature information based on the bird's-eye view feature information using a perception task, generating a trajectory prediction distribution based on the bird's-eye view feature information and predefined anchor points using a trajectory prediction head, and obtaining the final trajectory (Refinement, i.e., the second driving trajectory) based on the semantic bird's-eye view feature information and the trajectory prediction distribution using a trajectory denoising head. As shown in the figure, the second driving trajectory is closer to the vehicle's true future driving trajectory than the first driving trajectory, and is closer to the optimal driving route.

[0260] The effectiveness of the vehicle trajectory prediction method based on the diffusion model provided in this application will be explained below.

[0261] Specifically, DiffRefiner was evaluated using two widely accepted benchmark sets: NAVSIM v2 for open-loop evaluation and Bench2Drive for closed-loop simulation.

[0262] In the NAVSIM benchmark, a subset of the standard navtrain dataset is used for training. Following the NAVSIM v2 dataset specification, the trajectory prediction model processes simultaneous multi-view inputs from the forward-looking, left forward-looking, and right forward-looking cameras to obtain trajectory prediction results.

[0263] In the Bench2Drive evaluation, we followed the dataset configuration and preprocessing workflow established in TF++. Both benchmark sets were initialized with 20 clustered trajectory anchors, and all trajectories were then fed into the optimization module to obtain the final trajectory prediction results.

[0264] All experiments were conducted on a cluster containing eight NVIDIA RTX 4090 GPUs.

[0265] The results show that DiffRefiner achieves state-of-the-art performance: on the NAVSIM v2 dataset, the Extended Predictive Driver Model Score (EPDMS) reaches 87.4; on the Bench2Drive benchmark set, the Driving Score (DS) reaches 87.1 and the Success Rate (SR) reaches 71.4, setting new performance records on both public benchmark sets.

[0266] Furthermore, as shown in Table 1, through comprehensive open-loop evaluation on the NAVSIM v2 benchmark set, the proposed DiffRefiner achieves state-of-the-art performance, significantly outperforming other trajectory prediction methods based on other models. Specifically, it achieves a 3.7% improvement with a ResNet34 backbone and a 1.6% improvement with a V2-99 backbone. The results demonstrate particularly significant improvements in safety-critical metrics, validating the effectiveness of the DiffRefiner architecture design.

[0267] Table 1: NAVSIM v2 Benchmark Results

[0268]

[0269] In the table, Method represents the method, Backbone represents the backbone network, Modality represents the modality, EPDMS represents the extended prediction model score (comprehensive index), NC represents the navigation completion rate, DAC represents the dynamic collision avoidance capability, DDC represents the hazard detection capability, TL represents the traffic compliance rate, EP represents the emergency response capability, TTC represents the collision time prediction, LK represents the lane keeping capability, HC represents the human-vehicle interaction capability, and EC represents the environmental adaptation capability.

[0270] Specifically, the table shows the overall score (EPDMS) for each method and a set of detailed driving metrics. Results are grouped by backbone network type (ResNet34 and V2-99). The DiffRefiner proposed in this application consistently outperforms other methods on most metrics and achieves the best overall performance.

[0271] Furthermore, as shown in Table 2, by further evaluating the proposed method in closed-loop scenarios on the Bench2Drive benchmark set, the proposed method outperforms all existing learning-based baseline methods, achieving state-of-the-art results. Without model ensemble, compared to the previous state-of-the-art method HiPAD (Tang et al., 2025), our method improves DS by 0.3 and SR by 2.3. Unlike previous discriminative or purely generative methods, our framework adopts a hybrid generative paradigm, combining coarse-anchor-based proposals with diffusion-based optimization. This design achieves stable improvements on most multi-capability metrics, demonstrating its robustness and effectiveness in various interactive driving scenarios.

[0272] Table 2: Bench2Drive Benchmark Test Results

[0273]

[0274] Among them, Paradigm is the paradigm type, Overall is the driving average, including DS and SR, Multi-Ability is the multi-functional rating, including lane merging (Merge), overtaking (Overtake), emergency braking (EBrake), giving way (GiveWay), traffic signs (Tsign), and mean (Mean). Rule based is the trajectory-based paradigm type, Discriminative is the discriminative paradigm type, Generative is the generative paradigm type, and Hybrid is a paradigm type based on both discriminative and generative methods, i.e., a hybrid type.

[0275] The DiffRefiner proposed in this application achieves the best overall performance among existing methods, with significant improvements in DS, SR and most multi-capability tasks.

[0276] Furthermore, the performance of the DiffRefiner proposed in this application was evaluated again through ablation experiments.

[0277] As shown in Table 3, the impact of the DiffRefiner planning framework at each stage was evaluated. Comparing rows 1 and 6, it can be seen that adding the optimizer improved EPDMS by 1.2, indicating that the optimizer significantly improves the quality of trajectory prediction. Rows 2 and 6 further show that higher quality first-stage trajectory prediction results raise the optimization ceiling; row 2 uses two denoising iterations, while row 6 uses only one iteration, both achieving better performance. Meanwhile, rows 1 and 5 show that appropriate supervision of the optimizer helps with proposal learning. Finally, the comparison between rows 4 and 6 highlights the advantages of our generative optimizer over the discriminative optimizer in fine-grained trajectory adjustment.

[0278] Table 3: Ablation Experiments Using the DiffRefiner Planning Framework

[0279]

[0280] Where Ref is the optimizer; Pro is the predictor decoder; RT is the optimizer type; Src is the downstream control input source; Gen is the generative class; Dis is the discriminative class; Param is the total number of model parameters; and Latency is the inference latency of the planning module.

[0281] Furthermore, the performance of the trajectory optimizer provided in this application is further evaluated through ablation experiments on the optimizer.

[0282] As shown in Table 4, we evaluated the contribution of each module in the optimizer through systematic ablation experiments. The results show that all components bring stable performance improvements, verifying the effectiveness of the overall design. Comparing rows 4-6, it can be seen that the semantic interaction mechanism gradually improves scene understanding and reduces collision-related errors by utilizing fine-grained semantic cues.

[0283] Table 4: Ablation Experiments of the Optimizer Component

[0284]

[0285] Where P is the planning token; A is the target object token; M is the BEV modulation; DA is the drivable area in FGSIM; and TP is the traffic participant in FGSIM.

[0286] Furthermore, as shown in Table 5, further evaluation of the components included in the fine-grained semantic interaction module reveals that both global context and local target cues can independently improve performance, demonstrating their complementary effects. However, due to information conflicts, simply adding and fusing the two together leads to performance degradation. In contrast, the adaptive gating mechanism proposed in this application can adaptively balance the weights of both, achieving optimal performance and verifying the advantages of adaptive feature fusion.

[0287] Table 5: Ablation experiments of FGSIM components

[0288]

[0289] In this context, Global refers to global cross-attention, Local refers to locally deformable attention, Fusion refers to the fusion method, Addition refers to additive fusion, and Gating refers to gated fusion.

[0290] Furthermore, as shown in Table 6, ablation experiments on the denoising step reveal that near-optimal performance can be achieved with only one denoising step. This finding highlights the effectiveness of high-quality proposals as strong prior information, making the diffusion-based optimization process more efficient, and also demonstrates that our framework is suitable for real-time end-to-end autonomous driving scenarios.

[0291] Table 6: Ablation experiments with the number of denoising steps

[0292]

[0293] Furthermore, such as Figure 10As shown, in complex interactive scenarios, the driving trajectory prediction method proposed in this application is superior to existing methods (such as the DiffusionDrive method). The driving trajectory planning in this application can better capture fine-grained scene details, reduce the probability of collision with surrounding obstacles, and more strictly comply with map constraints, thereby generating higher quality trajectories.

[0294] from Figure 10 As can be seen in (a) above, under the same front view, the driving trajectory determined by existing methods (such as the DiffusionDrive method) carries the risk of collision with surrounding obstacles, while the driving trajectory determined by the method proposed in this application can effectively avoid collisions with other surrounding vehicles. Figure 10 As can be seen from (b) in the paper, when a vehicle leaves the roadside parking area, the driving trajectory determined by the existing method carries the risk of colliding with other parked vehicles on the roadside. However, the driving trajectory determined by the method proposed in this application is closer to the reference trajectory (i.e., the optimal trajectory) and can avoid collisions with other vehicles.

[0295] Furthermore, from Figure 10 As can be seen from (c) in the text, at complex traffic intersections, the driving trajectory determined by existing methods has the problem of deviating from the current driving lane and entering other lanes. However, the driving trajectory determined by this application can avoid the problem of entering other lanes. Figure 10 As can be seen from (d) in the paper, the driving trajectory determined by the existing method has the problem of deviating from the driving road surface, while the driving trajectory determined by this application can avoid the problem of driving off the driving road surface.

[0296] Therefore, the vehicle trajectory prediction method provided by this application determines a more accurate vehicle trajectory that better meets actual driving needs, effectively improving the safety of vehicle assisted driving.

[0297] The vehicle trajectory prediction method provided in this application can be applied to vehicles, where the vehicles drive according to the aforementioned second trajectory information, or it can be applied to a cloud server, where the cloud server sends the determined second trajectory information to the vehicle, and the vehicle drives according to the second trajectory information. Of course, it can also be applied to other electronic devices such as computers. Therefore, this application also provides an electronic device for implementing the vehicle trajectory prediction method.

[0298] It should be noted that, in addition to the specific embodiments described above, those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of describing the invention in conjunction with the implementation is to cover other options or modifications that may be derived from this application. To provide a thorough understanding of this application, many specific details are included in the above description, and this application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0299] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0300] It should be noted that the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0301] It should be noted that some structural or methodological features may be shown in the accompanying drawings in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0302] Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art should understand that the above description is a further detailed explanation of the application in conjunction with specific implementations, and should not be construed as limiting the specific implementation of the application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. A method for predicting vehicle trajectory, characterized in that, The method includes: Determine the vehicle driving information corresponding to the target vehicle; Based on the vehicle driving information, bird's-eye view feature information is obtained; Based on the vehicle driving information and the bird's-eye view feature information, query feature information, semantic segmentation information, and target object detection result information are obtained. The query feature information includes target object feature information and trajectory feature information. The target object feature information is a target object token, which is the scene context information of the target object, where the target object is an obstacle. The trajectory feature information is a trajectory token, which is the scene information of the target vehicle's forward driving trajectory. The semantic segmentation information is full-scene semantic understanding information based on the bird's-eye view feature information. The target object detection result information is obstacle detection result information. Determine the predefined anchor point information of the target vehicle, and obtain the first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and the query feature information; Based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information, and the first driving trajectory information, the second driving trajectory information corresponding to the target vehicle is obtained.

2. The vehicle trajectory prediction method according to claim 1, characterized in that, The vehicle driving information includes vehicle driving status information and vehicle navigation information. Based on the vehicle driving information and the bird's-eye view feature information, query feature information, semantic segmentation information, and target object detection result information are obtained, including: The vehicle driving status information and the vehicle navigation information are encoded to obtain representation information; The query feature information is obtained based on the characterization information and the bird's-eye view feature information; Based on the target object feature information, the target object detection result information is obtained; The semantic segmentation information is obtained based on the bird's-eye view feature information.

3. The vehicle trajectory prediction method according to claim 2, characterized in that, Based on the predefined anchor point information and the query feature information, the first driving trajectory information corresponding to the target vehicle is obtained, including: Based on the predefined anchor point information and the trajectory feature information, the driving trajectory query feature information is obtained; Determine the anchor point offset, and obtain the first driving trajectory information based on the driving trajectory query feature information and the anchor point offset.

4. The vehicle trajectory prediction method according to claim 3, characterized in that, Based on the predefined anchor point information and the trajectory feature information, driving trajectory query feature information is obtained, including: The predefined anchor point information is subjected to sinusoidal position encoding processing to obtain anchor point position encoding information; Based on the anchor point position encoding information, the anchor point projection feature information is obtained using a multilayer perceptron. Based on the cross-attention mechanism, the driving trajectory query feature information is obtained according to the anchor point projection feature information and the trajectory feature information.

5. The vehicle trajectory prediction method according to any one of claims 1-4, characterized in that, Based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information, and the first driving trajectory information, the second driving trajectory information is obtained, including: Based on the diffusion model and multilayer perceptron, the first driving trajectory information is iteratively denoised to obtain the first optimized query feature information; The second driving trajectory information is obtained based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information, the first driving trajectory information, and the first optimized query feature information.

6. The vehicle trajectory prediction method according to claim 5, characterized in that, Based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information, the first driving trajectory information, and the first optimized query feature information, the second driving trajectory information is obtained, including: Based on the target object detection result information, the bird's-eye view feature information, and the semantic segmentation information, semantic bird's-eye view feature information is obtained, which includes drivable area bird's-eye view feature information and target object area bird's-eye view feature information. Based on the semantic bird's-eye view feature information and the first optimized query feature information, the road constraint trajectory feature information is obtained; Obstacle avoidance constraint trajectory feature information is obtained based on the semantic bird's-eye view feature information, the first driving trajectory information, and the road constraint trajectory feature information; Based on the road constraint trajectory feature information and the obstacle avoidance constraint trajectory feature information, the fused second optimized query feature information is obtained; The second driving trajectory information is obtained based on the second optimized query feature information and the query feature information.

7. The vehicle trajectory prediction method according to claim 6, characterized in that, Based on the second optimized query feature information and the query feature information, the second driving trajectory information is obtained, including: Based on the second optimized query feature information and the trajectory feature information, the third optimized query feature information is obtained; Based on the third optimized query feature information and the target object feature information, the fourth optimized query feature information is obtained; Based on the fourth optimized query feature information, the second driving trajectory information and the trajectory confidence information of the second driving trajectory information are obtained.

8. The vehicle trajectory prediction method according to claim 1, characterized in that, The vehicle driving information includes vehicle perception images. Based on the vehicle driving information, bird's-eye view feature information is obtained, including: The bird's-eye view feature information is obtained based on the perceived vehicle image using the bird's-eye view encoder.

9. A vehicle trajectory prediction system, characterized in that, The vehicle trajectory prediction system includes an acquisition module, an encoder module, a perception module, and a planning module, wherein... The acquisition module is used to determine the vehicle driving information corresponding to the target vehicle; The encoder module is used to obtain bird's-eye view feature information based on the vehicle driving information; The perception module is used to obtain query feature information, semantic segmentation information, and target object detection result information based on the vehicle driving information and the bird's-eye view feature information. The query feature information includes target object feature information and trajectory feature information. The target object feature information is a target object token, which is the scene context information of the target object. The target object is an obstacle. The trajectory feature information is a trajectory token, which is the scene information of the target vehicle's forward driving trajectory. The semantic segmentation information is full-scene semantic understanding information based on the bird's-eye view feature information. The target object detection result information is obstacle detection result information. The planning module is used to determine the predefined anchor point information of the target vehicle, obtain the first driving trajectory information corresponding to the target vehicle based on the predefined anchor point information and the query feature information, and obtain the second driving trajectory information corresponding to the target vehicle based on the bird's-eye view feature information, the query feature information, the semantic segmentation information, the target object detection result information and the first driving trajectory information.

10. An electronic device, characterized in that, The electronic device is used to perform the vehicle trajectory prediction method as described in any one of claims 1-8.

Images