Occupancy grid and world model based autonomous driving trajectory prediction method and system

Abstract

The application provides an autonomous driving trajectory prediction method and system based on an occupancy grid and a world model. The method comprises collecting scene data, generating a three-dimensional occupancy grid based on the scene data; constructing a geometric aggregation tokenizer, training the geometric aggregation tokenizer using the three-dimensional occupancy grid to obtain a trained geometric aggregation tokenizer; encoding and quantizing the three-dimensional occupancy grid corresponding to each frame based on the trained geometric aggregation tokenizer to generate discrete latent features corresponding to each frame; constructing a dynamic world model, training the dynamic world model using the latent features to obtain a trained dynamic world model; inputting scene data of a trajectory to be predicted into the trained geometric aggregation tokenizer and the dynamic world model, and outputting an autonomous driving trajectory prediction result of a vehicle. The application can maintain the spatial coherence and geometric consistency of the scene in a long-time prediction process, and effectively suppresses the cumulative error in trajectory prediction.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to an autonomous driving trajectory prediction method and system based on occupancy grids and a world model. Background Technology

[0002] Understanding and predicting the 3D structure of the surrounding environment is fundamental to autonomous driving. Specifically, 3D occupancy meshes discretize continuous driving scenes into voxel meshes, capturing fine-grained geometric details and semantic information of objects of arbitrary shapes. In recent years, occupancy mesh world models have proven to provide a powerful intermediate representation for predicting the spatiotemporal evolution of the environment, thus serving downstream planning and decision-making. Most such methods formulate this task as an autoregressive next-frame prediction problem, employing GPT-like autoregressive architectures or diffusion-based architectures to deduce the future 3D occupancy mesh from past observations. However, despite their success, current methods face a critical problem: progressive scene degradation occurs as the prediction span increases.

[0003] In world models for autonomous driving, existing methods can be broadly categorized into video-driven approaches and volumetric scene-based approaches. Video-driven models aim to generate videos consistent with driving logic based on past image sequences and current actions. While they achieve impressive visual generation quality, these methods often struggle to maintain consistency across multiple views due to the lack of explicit 3D spatial representation. In contrast, volumetric scene-based methods utilize uniform 3D scene representations (such as LiDAR or 3D occupancy meshes), thus better maintaining geometric consistency across views. Furthermore, since 3D occupancy meshes are represented by dense voxels, directly performing world evolution is very expensive in terms of memory and computation. Most previous methods assume that scene dynamics primarily occur on the road surface and use learnable class embeddings to convert 3D occupancy meshes into bird's-eye view (BEV) features, but this largely eliminates height information, making it difficult to recover the complete 3D occupancy mesh after spatiotemporal modeling.

[0004] A literature search of existing technologies revealed a Chinese patent with publication number CN118850109A, which proposes an end-to-end autonomous driving planning method and system based on a dynamic-static decoupling world model. This method models the 3D environment in BEV space, decouples the environment representation from dynamics and statics, and uses independent branches to model the motion evolution of dynamic and static objects separately, making it more reasonable and efficient. However, this method does not decouple the vehicle's motion state from fine-grained scene dynamics, which can easily lead to error accumulation and scene degradation in long-sequence prediction.

[0005] The root cause of long-sequence scene degradation lies in a key design choice: existing models implicitly couple vehicle motion (the movement of the vehicle itself) and scene dynamics (object motion and structural changes) within a single transformation function. This coupling forces the world model to learn both rigid global transformations and fine-grained dynamic changes simultaneously, which becomes increasingly unstable as the prediction span extends. Because predictions are recursively fed back to the model, even small errors accumulate, leading to drift, spatial inconsistencies, and ultimately, unrealistic scene reconstructions.

[0006] Despite significant progress in world model research based on volumetric scenes, maintaining high fidelity and consistency in long-term time-series predictions still faces two key challenges: 1. Existing methods fail to explicitly decouple the vehicle's motion state from fine-grained scene dynamics. The single scene dynamics learning process is difficult to capture the heterogeneous motion correlations of the surrounding scene, which inevitably leads to severe error accumulation and scene degradation in long sequence prediction. 2. Existing methods for occupying grid feature transformation (such as direct mapping to BEV space) often lose key height information, which poses a huge challenge to the joint optimization of word segmentation efficiency and dynamic fidelity. It is difficult to maintain geometric consistency in the full 3D space while keeping computational efficiency. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this application is to provide an autonomous driving trajectory prediction method and system based on occupancy grids and world models.

[0008] According to the first aspect of this application, an autonomous driving trajectory prediction method based on occupancy grid and world model is provided, comprising: Collect scene data of the vehicle during autonomous driving, and generate a three-dimensional occupancy mesh based on the scene data; Construct a geometric aggregation word segmenter, and train the geometric aggregation word segmenter using the three-dimensional occupied grid to obtain the trained geometric aggregation word segmenter; Based on the trained geometric aggregation word segmenter, the three-dimensional occupied grid corresponding to each frame is encoded and quantized to generate discrete latent features corresponding to each frame; A dynamic world model is constructed that integrates temporal causal cross-attention and spatial self-attention. The dynamic world model is used to generate latent features for the next frame based on the latent features of the current frame. The dynamic world model is trained using the latent features to obtain the trained dynamic world model. The scene data of the trajectory to be predicted is input into the trained geometric aggregation word segmenter and dynamic world model, and the autonomous driving trajectory prediction result of the vehicle is output.

[0009] Optionally, the geometric aggregation word segmenter includes a perception module, an encoder, a multi-scale quantization module, and a decoder; the perception module is used to generate a fused bird's-eye view feature map with height dimension based on the three-dimensional occupancy grid; the encoder is used to encode the fused bird's-eye view feature map to generate a latent feature map; the multi-scale quantization module is used to quantize the latent feature map based on the nearest neighbor allocation strategy to obtain discrete latent features, the latent features being represented by several codebook nodes; the decoder is used to upsample the latent features to reconstruct the three-dimensional occupancy grid.

[0010] Optionally, the perception module is used to generate a fused bird's-eye view feature map with height dimension based on the three-dimensional occupancy grid, including: Map the 3D occupied mesh to a volume-aware embedding; Based on the volume-aware embedding, three pairwise orthogonal view embeddings are extracted, namely bird's-eye view embedding, front view embedding, and side view embedding; the front view embedding and the side view embedding are both used to represent the height dimension of the autonomous driving scenario; Multiply the front view embedding features and the side view embedding features in the height dimension to obtain cross-dimensional interactive features; The cross-dimensional interaction features are convolved, and the convolved cross-dimensional interaction features are superimposed and fused with the bird's-eye view embedding features to obtain an enhanced bird's-eye view representation with height dimension. The enhanced bird's-eye view representation and the embedded bird's-eye view are weighted and fused to generate a fused bird's-eye view feature map.

[0011] Optionally, during the training process, the geometric aggregation word segmenter uses the three-dimensional occupancy grid of the original input as the training ground value, and adopts a multi-loss function including cross-entropy loss, Lovász loss and vector quantization loss. The multi-loss function is minimized until the loss converges or the maximum number of iterations is reached, thus obtaining the trained geometric aggregation word segmenter.

[0012] Optionally, the dynamic world model includes a temporal causal cross-attention layer and a spatial self-attention layer. The input of the temporal causal cross-attention layer is connected to a time warp module, which is used to align the latent features of historical frames to the vehicle coordinate system of the current frame. The temporal causal cross-attention layer takes a predefined dynamic query vector and the aligned latent features of historical frames as common inputs, and outputs dynamic features that characterize changes within the autonomous driving scenario. The dynamic features and the latent features of the current frame are jointly input into the spatial self-attention layer, and after being fused by the spatial self-attention mechanism, the latent features of the next frame are output.

[0013] Optionally, the time warp module is used to align latent features of historical frames to the vehicle coordinate system of the current frame, including: A memory bank is constructed using the latent features of each frame output by the trained geometric aggregation word segmenter; Using the vehicle coordinate system of the current frame as a reference, a rigid transformation is performed on the latent features of historical frames in the memory bank based on the vehicle's pose, so that the latent features of historical frames are aligned to the vehicle coordinate system of the current frame.

[0014] Optionally, during the training process, the dynamic world model uses the potential features of the next frame output by the trained geometric aggregation word segmenter as the training ground value, and adopts a hybrid loss function including mean squared error loss and L2 loss to minimize the hybrid loss function until the loss converges or the maximum number of iterations is reached, thus obtaining the trained dynamic world model.

[0015] Optionally, the step of inputting the scene data of the trajectory to be predicted into the trained geometric aggregation segmenter and dynamic world model, and outputting the autonomous driving trajectory prediction result of the vehicle, includes: The scene data of the trajectory to be predicted is converted into a 3D occupancy grid. The 3D occupancy grid of the current frame is encoded by a trained geometric aggregation segmenter. The latent features of the next frame are generated by a trained dynamic world model. The features are then decoded and reconstructed by the trained geometric aggregation segmenter to obtain the reconstructed 3D occupancy grid of the next frame. The autonomous driving trajectory prediction result of the vehicle is formed based on the reconstructed 3D occupancy grids of each future frame.

[0016] According to a second aspect of this application, an autonomous driving trajectory prediction system based on occupancy grids and a world model is provided, comprising: The acquisition module is used to collect scene data of the vehicle during the autonomous driving process and generate a three-dimensional occupancy mesh based on the scene data; The word segmentation module provides a geometric aggregation word segmenter, which is trained using the three-dimensional occupancy grid to obtain the trained geometric aggregation word segmenter. The feature extraction module is used to encode and quantize the three-dimensional occupied grid corresponding to each frame based on the trained geometric aggregation word segmenter, and generate discrete potential features corresponding to each frame. The dynamic module provides a dynamic world model that integrates temporal causal cross-attention and spatial self-attention. The dynamic world model is used to generate latent features for the next frame based on the latent features of the current frame. The dynamic world model is trained using the latent features to obtain the trained dynamic world model. The prediction module is used to input the scene data of the trajectory to be predicted into the trained geometric aggregation word segmenter and dynamic world model, and output the autonomous driving trajectory prediction results of the vehicle.

[0017] According to a third aspect of this application, an electronic device is provided, characterized in that it comprises: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute, according to the obtained program instructions, the steps of an autonomous driving trajectory prediction method based on an occupancy grid and a world model as provided in the first aspect of this application.

[0018] This application provides an autonomous driving trajectory prediction method based on occupancy grids and world models. It employs a geometric aggregation word segmenter, which can discretize continuous 3D occupancy grids into multi-scale latent feature representations, significantly reducing data dimensionality and facilitating the learning and training of dynamic world models. The temporal causal cross-attention mechanism and spatial self-attention mechanism in the dynamic world model can extract and accurately predict fine-grained residual dynamics caused by moving objects and structural changes. This can effectively suppress the accumulated error in the autoregressive prediction of the dynamic world model. In the continuous multi-step, long-term autoregressive prediction process, it maintains the spatial coherence and geometric consistency of the scene, making the predicted 3D scene structure stable and maintaining strict physical spatial constraints with the vehicle's spatial position, driving posture, and driving space.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of an autonomous driving trajectory prediction method in one embodiment of this application; Figure 2 This is a schematic diagram of a geometric aggregation word segmenter in one embodiment of this application; Figure 3 This is a schematic diagram of a sensing module in one embodiment of this application; Figure 4 This is a schematic diagram of a dynamic world model in one embodiment of this application; Figure 5 This is a schematic diagram of an autonomous driving trajectory prediction system in one embodiment of this application. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0023] Understanding and predicting the 3D structure of the surrounding environment is fundamental to autonomous driving. In recent years, occupancy grid world models have proven to provide a powerful intermediate representation for predicting the spatiotemporal evolution of the environment, thus serving downstream planning and decision-making. However, existing methods fail to explicitly decouple the vehicle's motion state from fine-grained scene dynamics. A single scene dynamics learning process struggles to capture the heterogeneous motion correlations of the surrounding scene, inevitably leading to severe error accumulation and scene degradation during long-sequence predictions. Furthermore, existing occupancy grid feature transformation methods often lose crucial height information, making it difficult to maintain computational efficiency while ensuring geometric consistency across the entire 3D space. Based on these issues, this application provides an autonomous driving trajectory prediction method based on occupancy grids and world models to address the aforementioned problems.

[0024] Reference Figure 1 As shown, this application provides an autonomous driving trajectory prediction method based on occupancy grids and a world model, including: S1. Collect scene data of the vehicle during the autonomous driving process, and generate a three-dimensional occupancy mesh based on the scene data; S2. Construct a geometric aggregation word segmenter. Train the geometric aggregation word segmenter using a three-dimensional occupied grid to obtain the trained geometric aggregation word segmenter. S3. Based on the trained geometric aggregation word segmenter, the three-dimensional occupied grid corresponding to each frame is encoded and quantized to generate discrete latent features corresponding to each frame. S4. Construct a dynamic world model that integrates temporal causal cross-attention and spatial self-attention. The dynamic world model is used to generate latent features for the next frame based on the latent features of the current frame. Train the dynamic world model using the latent features to obtain the trained dynamic world model. S5. Input the scene data of the trajectory to be predicted into the trained geometric aggregation word segmenter and dynamic world model, and output the autonomous driving trajectory prediction result of the vehicle.

[0025] For example, scene data includes road image data and 3D point cloud data of the road's surrounding environment. By performing voxelization processing on the road image data and 3D point cloud data, a 3D occupancy grid for each frame can be obtained.

[0026] The embodiments described above employ a geometric aggregation segmenter, which can discretize continuous 3D occupied grids into multi-scale latent feature representations, significantly reducing data dimensionality and facilitating the learning and training of dynamic world models. The temporal causal cross-attention mechanism and spatial self-attention mechanism in the dynamic world model can extract and accurately predict fine-grained residual dynamics caused by moving objects and structural changes, effectively suppressing the accumulated error in the autoregressive prediction of the dynamic world model. In the continuous multi-step, long-term autoregressive prediction process, the spatial coherence and geometric consistency of the scene are maintained, making the predicted 3D scene structure stable and maintaining strict physical spatial constraints with the vehicle's spatial position, driving posture, and driving space.

[0027] Reference Figure 2 As shown in some specific embodiments of this application, the geometric aggregation word segmenter includes a perception module, an encoder, a multi-scale quantization module, and a decoder; the perception module is used to generate a fused bird's-eye view feature map with height dimension based on the three-dimensional occupancy grid; the encoder is used to encode the fused bird's-eye view feature map to generate a latent feature map; the multi-scale quantization module is used to quantize the latent feature map based on the nearest neighbor allocation strategy to obtain discrete latent features, and the latent features are represented by several codebook nodes; the decoder is used to upsample the latent features to reconstruct the three-dimensional occupancy grid.

[0028] In the embodiments described above, the encoder employs a lightweight two-dimensional encoder. The fused bird's-eye view feature map can be encoded into a latent feature map using the two-dimensional encoder. Subsequently, a multi-scale quantization module is applied, based on a preset codebook, to discretize the latent feature map into codebook features (i.e., latent features) containing multiple codebook nodes through a nearest neighbor allocation strategy, thereby forming a discrete latent space for the dynamic world model. In the reconstruction stage, the decoder upsamples the latent features back to the resolution of the original fused bird's-eye view feature map and reshapes the channel dimensions to restore the vertical axis (i.e., the height dimension). Finally, the final three-dimensional occupancy grid prediction result is generated through a voxel-level normalized exponential function (such as the softmax function).

[0029] Reference Figure 3 As shown, in some specific embodiments of this application, the perception module is used to generate a fused bird's-eye view feature map with height dimension based on a three-dimensional occupancy grid, and may further include: S21. Map the 3D occupied mesh to a volume-aware embedding; S22. Based on volume-aware embedding, three pairwise orthogonal view embeddings are extracted, namely bird's-eye view embedding, front view embedding, and side view embedding; the front view embedding and side view embedding are both used to represent the height dimension of the autonomous driving scenario. S23. Perform a product operation on the height dimension of the front view embedding feature and the side view embedding feature to obtain cross-dimensional interactive features; S24. Perform convolution processing on the cross-dimensional interaction features, and then superimpose and fuse the convolution-processed cross-dimensional interaction features with the bird's-eye view embedding features to obtain an enhanced bird's-eye view representation with height dimension. S25. Weighted fusion of the enhanced bird's-eye view representation and the bird's-eye view embedding is performed to generate a fused bird's-eye view feature map.

[0030] Existing methods assume that the dynamics of autonomous driving scenarios primarily occur on the road surface and use learnable category embeddings to convert 3D occupied meshes into bird's-eye view features; however, this largely eliminates height information. To overcome this problem and capture fine-grained geometric details of objects of arbitrary shapes, the embodiments described above employ a perception module to preserve height information in the features. (Refer to...) Figure 3 As shown, firstly, the 3D occupancy grid is mapped to a volume-aware embedding; then, three orthogonal view embeddings are derived based on the volume-aware embedding: bird's-eye view embedding, front view embedding, and side view embedding. These three views provide complementary geometric priors and reduce the information loss caused by relying on a single view. The front view embedding is used to represent the depth and height dimensions (spatial height direction perpendicular to the ground) of the autonomous driving scene; the side view embedding is used to represent the height and width dimensions of the autonomous driving scene; the bird's-eye view embedding is used to represent the depth dimension (vehicle front-to-back direction) and width dimension (vehicle left-to-right direction) of the autonomous driving scene; importantly, the front view embedding and the side view embedding retain height as a spatial dimension. In order to establish model interaction between views and retain height as a spatial dimension, this application introduces a height-aware interaction mechanism. Specifically, the height-aware interaction mechanism performs a product operation on the height feature dimension of the side view embedding features and the front view embedding features to obtain cross-dimensional interaction features to capture the correlation across height and depth. Subsequently, the cross-dimensional interaction features are convolved and then superimposed with the original bird's-eye view embedding features to finally generate an enhanced bird's-eye view representation with height awareness. After obtaining the enhanced bird's-eye view representation, its spatial adaptive weights are further calculated using a neural network, and the enhanced bird's-eye view representation is weighted and fused with the original spatial bird's-eye view embedding based on the spatial adaptive weights.

[0031] In the embodiments described above, the geometric aggregation segmenter can convert a 3D occupancy grid into a volume-aware embedding. It preserves highly sensitive geometric cues by fusing orthogonal views (such as front and side views) and utilizes an encoder-decoder architecture combined with a multi-scale quantization mechanism to discretize a continuous 3D scene into multi-scale latent features containing rich geometric priors. These embodiments aggregate highly perceptual features from the 3D occupancy grid, enhancing the spatial perception capability of bird's-eye view features.

[0032] In some specific embodiments of this application, during the training process, the geometric aggregation word segmenter uses the three-dimensional occupancy grid of the original input as the training ground value, and adopts a multi-loss function including cross-entropy loss, Lovász loss and vector quantization loss. The multi-loss function is minimized until the loss converges or the maximum number of iterations is reached, thus obtaining the trained geometric aggregation word segmenter.

[0033] Reference Figure 4 As shown in some specific embodiments of this application, the dynamic world model includes a temporal causal cross-attention layer and a spatial self-attention layer. The input end of the temporal causal cross-attention layer is connected to a time warp module, which is used to align the latent features of historical frames to the vehicle coordinate system of the current frame. The temporal causal cross-attention layer takes a predefined dynamic query vector and the aligned latent features of historical frames as common inputs, and outputs dynamic features that characterize changes in the autonomous driving scenario. The dynamic features and the latent features of the current frame are jointly input into the spatial self-attention layer, and after being fused by the spatial self-attention mechanism, the latent features of the next frame are output.

[0034] For example, existing models typically implicitly entangle vehicle motion and scene dynamics within a single transformation function, forcing the model to simultaneously learn rigid global transformations and fine-grained dynamic changes. To mitigate scene degradation in long-term predictions, this application explicitly decomposes scene evolution into a temporal world transformation and a dynamic world model. The temporal world transformation can employ a time-warping module, while the dynamic world model is used to predict the residual dynamics of the output features of the time-warping module to capture local object motion and structural changes. The dynamic world model consists of two key components: temporal causal cross-attention and spatial self-attention. The dynamic world model inputs a predefined dynamic query vector and latent features from historical frames into the temporal causal cross-attention layer to extract dynamic features representing scene changes. Subsequently, spatial awareness updates of latent and dynamic features are achieved through a spatial self-attention mechanism. The entire dynamic world model comprises multi-level hierarchical spatial self-attention layers. Each layer downsamples the feature map to capture multi-granular motion changes. Higher-level (later in the stack) spatial self-attention layers capture overall scene dynamics with their coarse-grained spatiotemporal correlations, while fine-grained modeling refines local changes, enabling precise voxel-level updates for occlusion and minute movements. To fully utilize these features, the dynamic world model introduces a hierarchical latent fusion strategy to fuse multi-level features in a coarse-to-fine manner. This strategy includes layer-by-layer feature concatenation, inter-layer cross-attention, and layer-by-layer geometric aggregation. Specifically, the dynamic features output from the temporal-causal cross-attention layer are input together with the latent features of the current frame into the spatial self-attention layer. Feature fusion is completed during multi-layer spatial modeling, generating the predicted latent features for the next frame.

[0035] In the embodiments described above, the dynamic world model extracts and accurately predicts fine-grained residual dynamics caused by moving objects and structural changes through temporal causal cross-attention and spatial self-attention mechanisms. This explicit decoupling architecture effectively suppresses the cumulative error in autoregressive predictions, maintains long-term spatial coherence, and generates clearer and more consistent future occupancy predictions.

[0036] In some specific embodiments of this application, the time warp module is used to align the latent features of historical frames to the vehicle coordinate system of the current frame, and may further include: S41. Construct a memory bank using the latent features of each frame output by the trained geometric aggregation word segmenter; S42. Using the vehicle coordinate system of the current frame as a reference, perform a rigid transformation on the latent features of historical frames in the memory bank based on the vehicle's posture, so that the latent features of historical frames are aligned to the vehicle coordinate system of the current frame.

[0037] For example, the time warp module uses information from previous frames in the memory bank (i.e., latent features of historical frames) and the known vehicle posture to perform a rigid transformation, mapping the latent features of the previous frames to the coordinate system of the current frame. This deterministic, geometry-aware operation effectively isolates the vehicle's own motion. After isolating the vehicle's own motion, the changes avoid misjudging the static road and building frame offsets caused by vehicle movement as dynamic changes, ensuring the accuracy of subsequent dynamic feature extraction. At the same time, considering that the evolution rate of the 3D occupied mesh varies in different regions, in order to model the heterogeneous dynamic patterns of the surrounding scene, this application introduces a specific set of dynamic query vectors to extract local dynamic changes across multiple historical frames and the next time step. These dynamic features have the same spatial dimension as the latent features.

[0038] In the above embodiments of this application, the time warp module uses the known vehicle posture to perform a rigid transformation on the features of the previous frame in the memory bank and aligns them to the coordinate system of the current vehicle, thereby effectively isolating vehicle motion through deterministic geometric perception operations.

[0039] In the world dynamic prediction stage, this application innovatively introduces a time warp mechanism to isolate the rigid coordinate transformation caused by the vehicle's physical displacement, and specifically constructs a dynamic world model to handle the remaining residual evolution caused by heterogeneous motion and structural changes within the scene. This architecture, which clearly separates rigid transformation and residual dynamics, enforces the physical rationality of spatial evolution and significantly improves the geometric consistency of long-term prediction.

[0040] In some specific embodiments of this application, during the training process, the dynamic world model uses the potential features of the next frame output by the trained geometric aggregation word segmenter as the training ground value, adopts a hybrid loss function including mean squared error loss and L2 loss, minimizes the hybrid loss function until the loss converges or the maximum number of iterations is reached, and obtains the trained dynamic world model.

[0041] This application is applicable to various 3D occupancy grid data. A geometric aggregation segmenter is used to extract compressed 3D latent features, which are then input into a dynamic world model to predict future 3D latent features. Finally, a decoder is used to obtain the future 3D occupancy grid. This application employs the following two-stage training strategy to optimize the model: 1. First stage (Geometric aggregation word segmenter training) In the first stage, only the geometric aggregation word segmenter is trained. The training process uses cross-entropy loss, Lovász loss, and vector quantization loss for comprehensive optimization. Specifically, the total loss of the geometric aggregation word segmenter is the sum of the cross-entropy loss, Lovász loss, and vector quantization loss. These three loss functions work together to ensure accurate reconstruction of the 3D occupied grid geometry, as well as the stability of feature extraction and feature space discretization representation.

[0042] 2. Second Stage (Dynamic World Model Training) In the second stage, this application utilizes the latent features extracted by a pre-trained geometric aggregation segmenter to predict the latent features for the next time step. Specifically, the total loss of the dynamic world model is the sum of the mean squared error loss and the L2 loss. This application applies the mean squared error loss to minimize the difference between the predicted future features and the actual future features. Furthermore, this application also utilizes the L2 loss to directly supervise the transformation process of rigid transformations, ensuring that the dynamic world model can accurately learn the physical laws governing the dynamics of the world and the consistency of spatial geometry.

[0043] This application aims to address the long-term scene degradation and error accumulation problems caused by the implicit coupling of global motion and local changes in existing next-frame prediction paradigms. This application proposes a novel 3D occupancy mesh world model construction method that explicitly decouples the vehicle's motion state from fine-grained scene dynamics, effectively mitigating error accumulation and scene degradation in long-term predictions. A geometric aggregation segmenter preserves highly sensitive geometric cues and discretizes features, combined with a time-warping module (for isolating rigid vehicle motion) and a dynamic world model (for predicting residual dynamics caused by moving objects), thereby generating more physically plausible and spatially coherent future features. The above embodiments of this application are highly efficient and physically meaningful, suitable for 3D environmental perception and long-sequence spatiotemporal evolution prediction tasks in autonomous driving. The core of this application is to explicitly separate the rigid coordinate offset caused by the vehicle's motion from the local residual motion, thereby maintaining geometric consistency and constraining the prediction space within a physically plausible trajectory.

[0044] Furthermore, extensive experiments on the publicly available benchmark dataset Occ3D-nuScenes validated the effectiveness of the proposed method. Table 1 shows a comparison of the trajectory planning results using this method with existing world models (OccWorld, OccLLaMA, DOME).

[0045] Table 1. Comparison of Experimental Results As shown in Table 1, this application outperforms existing world models in short-term frame rate results (1 second / 2 seconds / 3 seconds), mean (Avg), and instantaneous frame rate (FPS). Therefore, the method in this application not only significantly alleviates scene degradation and surpasses existing methods in the generation of long-term 3D occupancy sequences, but also maintains extremely high computational efficiency, demonstrating the effectiveness of this method in improving prediction fidelity, maintaining long-term spatial consistency, and generating coherent long sequences.

[0046] In some specific embodiments of this application, the scene data of the trajectory to be predicted is input into the trained geometric aggregation word segmenter and dynamic world model, and the autonomous driving trajectory prediction result of the vehicle is output, which may further include: The scene data of the trajectory to be predicted is converted into a 3D occupancy grid. The 3D occupancy grid of the current frame is encoded by a trained geometric aggregation segmenter. The latent features of the next frame are generated by a trained dynamic world model. The features are then decoded and reconstructed by the trained geometric aggregation segmenter to obtain the reconstructed 3D occupancy grid of the next frame. The autonomous driving trajectory prediction result of the vehicle is formed based on the reconstructed 3D occupancy grids of each future frame.

[0047] The embodiments described above in this application are capable of predicting future driving scenarios based on existing scene data. Specifically, the existing scene data is converted into a three-dimensional occupancy grid, which serves as the occupancy grid for existing frames (such as frame T, frame T-1, and frame T-2). The encoder of the geometric aggregation segmenter can generate latent features of existing frames based on the existing frame occupancy grid. The dynamic world model can predict latent features of future frames (such as frame T+1) based on the latent features of existing frames. Then, the decoder of the geometric aggregation segmenter decodes the latent features of future frames to obtain the three-dimensional occupancy grid of future frames. Finally, the future driving scenario is output based on the three-dimensional occupancy grids of several future frames.

[0048] Reference Figure 5 As shown, based on the same inventive concept, another embodiment of this application provides an autonomous driving trajectory prediction system based on an occupancy grid and a world model. The autonomous driving trajectory prediction system 100 includes: The acquisition module 110 is used to acquire scene data of the vehicle during the autonomous driving process and generate a three-dimensional occupancy mesh based on the scene data. The word segmentation module 120 provides a geometric aggregation word segmenter, which is trained using a three-dimensional occupancy grid to obtain the trained geometric aggregation word segmenter. The feature extraction module 130 is used to encode and quantize the three-dimensional occupied grid corresponding to each frame based on the trained geometric aggregation word segmenter, and generate discrete latent features corresponding to each frame. Dynamic Module 140 provides a dynamic world model that integrates temporal causal cross-attention and spatial self-attention. The dynamic world model is used to generate latent features for the next frame based on the latent features of the current frame. The dynamic world model is trained using the latent features to obtain the trained dynamic world model. The prediction module 150 is used to input the scene data of the trajectory to be predicted into the trained geometric aggregation word segmenter and dynamic world model, and output the autonomous driving trajectory prediction result of the vehicle.

[0049] It should be noted that the modules in the autonomous driving trajectory prediction system based on occupied grid and world model provided in the above embodiments of this application correspond to the steps of the autonomous driving trajectory prediction method based on occupied grid and world model in any of the above embodiments. Those skilled in the art can refer to the step features of the autonomous driving trajectory prediction method based on occupied grid and world model to implement the corresponding modules in the autonomous driving trajectory prediction system based on occupied grid and world model, which will not be repeated here.

[0050] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the above-described autonomous driving trajectory prediction method based on occupancy grid and world model according to the obtained program instructions.

[0051] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0052] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0053] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0054] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0060] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. An autonomous driving trajectory prediction method based on occupancy grids and a world model, characterized in that, include: Collect scene data of the vehicle during autonomous driving, and generate a three-dimensional occupancy mesh based on the scene data; Construct a geometric aggregation word segmenter, and train the geometric aggregation word segmenter using the three-dimensional occupied grid to obtain the trained geometric aggregation word segmenter; Based on the trained geometric aggregation word segmenter, the three-dimensional occupied grid corresponding to each frame is encoded and quantized to generate discrete latent features corresponding to each frame; A dynamic world model is constructed that integrates temporal causal cross-attention and spatial self-attention. The dynamic world model is used to generate latent features for the next frame based on the latent features of the current frame. The dynamic world model is trained using the latent features to obtain the trained dynamic world model. The scene data of the trajectory to be predicted is input into the trained geometric aggregation word segmenter and dynamic world model, and the autonomous driving trajectory prediction result of the vehicle is output.

2. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 1, characterized in that, The geometric aggregation word segmenter includes a perception module, an encoder, a multi-scale quantization module, and a decoder; the perception module is used to generate a fused bird's-eye view feature map with height dimension based on the three-dimensional occupancy grid; the encoder is used to encode the fused bird's-eye view feature map to generate a latent feature map; The multi-scale quantization module is used to quantize the latent feature map based on the nearest neighbor allocation strategy to obtain discrete latent features, wherein the latent features are represented by several codebook nodes. The decoder is used to upsample the latent features and reconstruct a three-dimensional occupancy grid.

3. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 1, characterized in that, The perception module is used to generate a fused bird's-eye view feature map with height dimension based on the three-dimensional occupancy grid, including: Map the 3D occupied mesh to a volume-aware embedding; Based on the volume-aware embedding, three pairwise orthogonal view embeddings are extracted, namely bird's-eye view embedding, front view embedding, and side view embedding; the front view embedding and the side view embedding are both used to represent the height dimension of the autonomous driving scenario; Multiply the front view embedding features and the side view embedding features in the height dimension to obtain cross-dimensional interactive features; The cross-dimensional interaction features are convolved, and the convolved cross-dimensional interaction features are superimposed and fused with the bird's-eye view embedding features to obtain an enhanced bird's-eye view representation with height dimension. The enhanced bird's-eye view representation and the embedded bird's-eye view are weighted and fused to generate a fused bird's-eye view feature map.

4. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 1, characterized in that, During the training process, the geometric aggregation word segmenter uses the original input's three-dimensional occupancy grid as the training ground value and employs a multi-loss function that includes cross-entropy loss, Lovász loss, and vector quantization loss. It minimizes the multi-loss function until the loss converges or the maximum number of iterations is reached, thus obtaining the trained geometric aggregation word segmenter.

5. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 1, characterized in that, The dynamic world model includes a temporal causal cross-attention layer and a spatial self-attention layer. The input of the temporal causal cross-attention layer is connected to a time warp module, which is used to align the latent features of historical frames to the vehicle coordinate system of the current frame. The temporal causal cross-attention layer takes a predefined dynamic query vector and the aligned latent features of historical frames as common inputs, and outputs dynamic features that characterize changes within the autonomous driving scenario. The dynamic features and the latent features of the current frame are input into the spatial self-attention layer, and after being fused by the spatial self-attention mechanism, the latent features of the next frame are output.

6. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 5, characterized in that, The time warp module is used to align latent features of historical frames to the vehicle coordinate system of the current frame, including: A memory bank is constructed using the latent features of each frame output by the trained geometric aggregation word segmenter; Using the vehicle coordinate system of the current frame as a reference, a rigid transformation is performed on the latent features of historical frames in the memory bank based on the vehicle's pose, so that the latent features of historical frames are aligned to the vehicle coordinate system of the current frame.

7. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 5, characterized in that, During the training process, the dynamic world model uses the potential features of the next frame output by the trained geometric aggregation word segmenter as the training ground value. It adopts a hybrid loss function that includes mean squared error loss and L2 loss, and minimizes the hybrid loss function until the loss converges or the maximum number of iterations is reached, thus obtaining the trained dynamic world model.

8. The autonomous driving trajectory prediction method based on occupancy grid and world model according to claim 1, characterized in that, The process of inputting scene data of the trajectory to be predicted into the trained geometric aggregation word segmenter and dynamic world model, and outputting the autonomous driving trajectory prediction result of the vehicle, includes: The scene data of the trajectory to be predicted is converted into a 3D occupancy grid. The 3D occupancy grid of the current frame is encoded by a trained geometric aggregation segmenter. The latent features of the next frame are generated by a trained dynamic world model. The features are then decoded and reconstructed by the trained geometric aggregation segmenter to obtain the reconstructed 3D occupancy grid of the next frame. The autonomous driving trajectory prediction result of the vehicle is formed based on the reconstructed 3D occupancy grids of each future frame.

9. An autonomous driving trajectory prediction system based on occupancy grids and a world model, characterized in that, include: The acquisition module is used to collect scene data of the vehicle during the autonomous driving process and generate a three-dimensional occupancy mesh based on the scene data; The word segmentation module provides a geometric aggregation word segmenter, which is trained using the three-dimensional occupancy grid to obtain the trained geometric aggregation word segmenter. The feature extraction module is used to encode and quantize the three-dimensional occupied grid corresponding to each frame based on the trained geometric aggregation word segmenter, and generate discrete potential features corresponding to each frame. The dynamic module provides a dynamic world model that integrates temporal causal cross-attention and spatial self-attention. The dynamic world model is used to generate latent features for the next frame based on the latent features of the current frame. The dynamic world model is trained using the latent features to obtain the trained dynamic world model. The prediction module is used to input the scene data of the trajectory to be predicted into the trained geometric aggregation word segmenter and dynamic world model, and output the autonomous driving trajectory prediction results of the vehicle.

10. An electronic device, characterized in that, include: At least one memory for storing program instructions; At least one processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of claims 1-8 according to the obtained program instructions.

Images