An automatic driving scene generation method based on a space-time decoupling world model

Abstract

This application discloses an autonomous driving scene generation method based on a spatiotemporal decoupled world model, belonging to the field of autonomous driving technology. The method includes: firstly, acquiring a multimodal autonomous driving dataset; after preprocessing to construct a state sequence; then, discretizing the state sequence into a pose word sequence and an image word sequence using an equidistant binning strategy and an improved time-aware vector quantization encoder; subsequently, inputting the word sequence into a spatiotemporal multimodal fusion module; and decoupling spatiotemporal information and predicting the potential state features of the next moment by alternately stacking temporal Transformer layers and spatial multimodal Transformer layers; during the training phase, employing a random masking strategy to prevent long-term generation drift, and calculating the cross-entropy loss between predicted words and ground truth words generated by the internal state autoregression module to update the model parameters; finally, generating autonomous driving scene data through binning inverse operation and a time-aware decoder; the method of this invention has the ability to generate long-term, high-fidelity videos.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, specifically relating to an autonomous driving scene generation method based on a spatiotemporal decoupled world model. Background Technology

[0002] The development of generative artificial intelligence has provided new ideas for building world models. Currently, the mainstream autoregressive models are mainly divided into two categories, but both have significant shortcomings and are difficult to meet the needs of generating high-resolution long videos for autonomous driving. First, there are methods based on Recurrent State-Space Models (RSSM). These methods treat the environment as a potential dynamic system, combining deterministic recurrent neural networks with stochastic variational autoencoders to map high-dimensional observations to low-dimensional latent states. They then predict future states and reconstruct image observations based on historical hidden states and actions. While RSSM performs well in low-dimensional control tasks, it has significant shortcomings in autonomous driving scenarios: First, the serial computation of recurrent neural networks (RNNs) limits the parallel capabilities of GPUs, resulting in low training efficiency. Second, the fixed capacity of hidden states makes it difficult to handle the large amount of visual detail in complex scenes, leading to blurry generated results. Third, as the sequence length increases, the vanishing or exploding gradient problem intensifies, making it difficult for the model to maintain long-term temporal consistency and easily resulting in drift phenomena that do not conform to physical laws, such as vehicles passing through walls or traffic flowing in the wrong direction.

[0003] Second, traditional autoregressive generation methods based on generative pre-trained Transformers (GPT): Transformers achieve parallel modeling of sequences through a global self-attention mechanism, significantly improving training efficiency, and tokens at any position can directly model long-distance dependencies, better capturing spatiotemporal causal relationships in autonomous driving scenarios. However, this type of method still has shortcomings: First, it forcibly flattens a video with spatial structure into a one-dimensional sequence, destroying intra-frame spatial relationships and affecting visual quality and physical consistency; second, errors accumulate continuously during the generation of long videos, easily leading to scene collapse problems such as the disappearance of road boundaries; third, the image tokenizer based on single-frame training lacks continuity in the temporal dimension, easily producing flickering and jitter after decoding, reducing the temporal consistency of the video. Therefore, it is necessary to design a method for generating autonomous driving scenarios that can improve the above-mentioned problems.

[0004] Invention / Invention Content To address the problems of existing technologies, this invention provides a method for generating autonomous driving scenarios based on a spatiotemporally decoupled world model, comprising the following steps: A multimodal autonomous driving dataset is acquired and preprocessed to obtain a state sequence, the dataset including video frames and pose data; the state sequence is then converted into a discrete token sequence, including a pose token sequence and an image token sequence. The obtained pose token sequence and image token sequence are input into the spatiotemporal multimodal fusion module of the world model. By alternately stacking the temporal Transformer layer and the spatial multimodal Transformer layer, the temporal and spatial information are decoupled to predict the potential state features of the next moment. The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random masking strategy is used to simulate prediction errors during inference and prevent long-term generation drift. The target pose token for the next time step is injected, along with the generated latent state features, as conditions. The predicted token sequence is then generated through the internal state autoregressive model module. The generated predicted token sequence is mapped back to the physical space to obtain directly usable autonomous driving scenario data.

[0005] Furthermore, a multimodal autonomous driving dataset is acquired and preprocessed to obtain a state sequence. This dataset includes video frames and pose data. The state sequence is then transformed into discrete token sequences, including pose token sequences and image token sequences, using different discretization methods. Specifically, this includes: The vehicle pose increment is calculated using a relative coordinate system, and a state sequence containing a preset number of time steps is constructed. Each time step contains the relative pose and the corresponding video frame, and the pose data used as control commands has no corresponding video frame. The vehicle pose increment is discretized into pose tokens using an equal-spacing binning strategy. A pre-trained temporally aware vector quantization encoder, which is based on an improved Vector Quantization Adversarial Generative Network (VQ-GAN) architecture, is used. The temporally aware vector quantization encoder embeds a temporal self-attention mechanism. The quantization encoder downsamples and compresses video frames into feature maps. After processing by the temporal self-attention layer, it assigns discrete image tokens to each spatial location of the feature map based on a preset codebook through minimum distance matching. The output pose token sequence and image token sequence are used as subsequent training data.

[0006] Furthermore, the obtained pose token sequence and image token sequence are input into the spatiotemporal multimodal fusion module of the world model. Through alternating stacked temporal Transformer layers and spatial multimodal Transformer layers, temporal and spatial information are decoupled to predict the potential state features of the next time step, including: The pose token sequence at each time step is concatenated with the image token sequence to form a multimodal token sequence, which is then mapped into a latent feature vector through an embedding layer. Employing a temporal causal self-attention mechanism, processed through a temporal Transformer layer, each token focuses only on tokens at the same position in historical frames, capturing the patterns of temporal evolution. A spatial multimodal self-attention mechanism is adopted, which is processed through a spatial multimodal Transformer layer to enable multimodal token interaction at the same time step and integrate spatial and semantic information; By alternately stacking N layers of temporal Transformer layers and spatial multimodal Transformer layers, the output is the next state feature containing spatiotemporal information of all preceding time steps.

[0007] Furthermore, the input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random masking strategy is employed to simulate prediction errors during inference and prevent long-term generation drift. The target pose token for the next time step is injected, along with the already generated latent state features, as conditions. Through the internal state autoregressive model module, a predicted token sequence is generated, specifically including: The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random mask is triggered with a preset probability to replace the image tokens with noise proportionally. The cross-entropy loss between the predicted tokens and the ground truth tokens is calculated to update the model parameters. The target pose token of the next time step is injected as the preceding token. A causal mask self-attention mechanism is adopted. Based on the features of the next state and the preceding token, the current token is predicted sequentially through the internal state autoregression module. Generation is triggered by the sequence start token, and the first two tokens are forced to be the target pose token.

[0008] Furthermore, the generated predicted token sequence is mapped back to physical space to obtain directly usable autonomous driving scenario data, including: The target pose token is restored by binning inverse operation as a relative increment, and the absolute pose at the current moment is calculated by combining the absolute pose at the previous moment. The image token is input into the time-aware decoder, the codebook feature vector is retrieved and processed by time self-attention, and then upsampled to reconstruct the video frame. If the preset number of frames is not reached, the newly generated video frames and the target pose are added to the historical sequence, the earliest frame is removed, and the process returns to the multimodal feature fusion stage to continue generation; if the preset number of frames is reached, the autonomous driving scene video data is synthesized.

[0009] An autonomous driving scene generation device based on a spatiotemporally decoupled world model includes: Encoding module: Acquires a multimodal autonomous driving dataset and preprocesses it to obtain a state sequence, the dataset including video frames and pose data; converts the state sequence into discrete token sequences, including pose token sequences and image token sequences, through different discretization methods; Fusion Module: The spatiotemporal multimodal fusion module inputs the obtained pose token sequence and image token sequence into the world model. Through alternating stacked temporal Transformer layers and spatial multimodal Transformer layers, it decouples temporal and spatial information and predicts the potential state features of the next moment. Autoregressive module: Input the true token sequence corresponding to the dataset, including true pose token and true image token; adopt a random masking strategy to simulate the prediction error during inference and prevent long-term generation drift, inject the target pose token of the next time step, and use the generated latent state features as conditions to generate the predicted token sequence through the internal state autoregressive model module. Decoding module: Maps the generated predicted token sequence back to the physical space to obtain directly usable autonomous driving scenario data.

[0010] Furthermore, the encoding module is used for: The vehicle pose increment is calculated using a relative coordinate system, and a state sequence containing a preset number of time steps is constructed. Each time step contains the relative pose and the corresponding video frame, and the pose data used as control commands has no corresponding video frame. The vehicle pose increment is discretized into pose tokens using an equal-spacing binning strategy. A pre-trained temporally aware vector quantization encoder, which is based on an improved VQ-GAN architecture, is used to downsample and compress video frames into feature maps. After processing by a temporally self-attention layer, discrete image tokens are assigned to each spatial location of the feature map based on a preset codebook through minimum distance matching. The output pose token sequence and image token sequence are used as subsequent training data.

[0011] Furthermore, the fusion module is a spatiotemporal multimodal fusion module with a 12-layer stacked structure. Each layer integrates a temporal causal self-attention mechanism and a spatial multimodal self-attention mechanism for: The pose token sequence at each time step is concatenated with the image token sequence to form a multimodal token sequence, which is then mapped into a latent feature vector through an embedding layer. Employing a temporal causal self-attention mechanism, processed through a temporal Transformer layer, each token focuses only on tokens at the same position in historical frames, capturing the patterns of temporal evolution. A spatial multimodal self-attention mechanism is adopted, which is processed through a spatial multimodal Transformer layer to enable multimodal token interaction at the same time step and integrate spatial and semantic information; By alternately stacking N layers of temporal Transformer layers and spatial multimodal Transformer layers, the output is the next state feature containing spatiotemporal information of all preceding time steps.

[0012] Furthermore, the autoregressive module is an internal state autoregressive module, used for: The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random mask is triggered with a preset probability to replace the image tokens with noise proportionally. The cross-entropy loss between the predicted tokens and the ground truth tokens is calculated to update the model parameters. The target pose token of the next time step is injected as the preceding token. A causal mask self-attention mechanism is adopted. Based on the features of the next state and the preceding token, the current token is predicted sequentially through the internal state autoregression module. Generation is triggered by the sequence start token, and the first two tokens are forced to be the target pose token.

[0013] Furthermore, the decoding module maps the generated predicted token sequence back to physical space, enabling pose decoding, image reconstruction, and loop generation, and outputting autonomous driving scene video data, including: The target pose token is restored by binning inverse operation as a relative increment, and the absolute pose at the current moment is calculated by combining the absolute pose at the previous moment. The image token is input into the time-aware decoder, the codebook feature vector is retrieved and processed by time self-attention, and then upsampled to reconstruct the video frame. If the preset number of frames is not reached, the newly generated video frames and the target pose are added to the historical sequence, the earliest frame is removed, and the process returns to the multimodal feature fusion stage to continue generation; if the preset number of frames is reached, the autonomous driving scene video data is synthesized.

[0014] The beneficial effects that this application can produce include: 1) This application improves the high fidelity and spatiotemporal consistency of video generation by embedding a temporal self-attention mechanism in the temporal-aware vector quantization encoder, which forces the model to capture the dynamic correlation between frames during the discretization stage; combined with the spatiotemporal multimodal fusion module and internal state autoregression module in the world model, it effectively solves the coupling problem between the internal spatial structure of video frames and the temporal dynamics between frames. The generated video not only has clear and realistic single-frame image quality, but also eliminates non-physical jumps and artifacts in the temporal dimension, ensuring that traffic flow dynamics and road structure strictly conform to the laws of motion in the physical world, and significantly improving visual quality. 2) This application achieves long-duration, controllable video generation through a target pose-guided autoregressive generation mechanism. By using a next-state prediction strategy, it extracts latent state features in parallel using a global perspective, ensuring the complete transmission of contextual information. At the same time, a random masking strategy is introduced during the training phase to actively simulate the uncertainty error in the inference process, greatly enhancing the robustness of the model. This invention can stably generate driving videos with a duration of more than 40 seconds, and maintains a high degree of logical coherence and scene structure stability over a long period of time. It can be applied to autonomous driving algorithm simulation testing, training data augmentation, and virtual driving scene construction, realizing the generation of long-duration, high-fidelity autonomous driving scene videos that conform to physical laws. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the training phase of the autonomous driving scene generation method based on the spatiotemporal decoupling world model in this application. Figure 2 This is a flowchart illustrating the reasoning stage of the autonomous driving scene generation method based on the spatiotemporal decoupled world model in this application. Figure 3 This is a schematic diagram of the architecture of the autonomous driving scene generation method based on the spatiotemporal decoupling world model proposed in this application; Figure 4 This is a schematic diagram of the spatiotemporal multimodal fusion module of this application; Figure 5 This is a schematic diagram of the internal state autoregression module structure of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1-3 This invention provides a method for generating autonomous driving scenarios based on a spatiotemporally decoupled world model, comprising the following steps: A multimodal autonomous driving dataset is acquired and preprocessed to obtain a state sequence, the dataset including video frames and pose data; the state sequence is then converted into a discrete token sequence, including pose token sequences and image token sequences. The obtained pose token sequence and image token sequence are input into the spatiotemporal multimodal fusion module of the world model. Through alternating stacked deep learning model (Transformer) layers with temporal self-attention mechanism and spatial multimodal Transformer layers, temporal and spatial information are decoupled to predict the potential state features of the next moment. The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random masking strategy is used to simulate prediction errors during inference and prevent long-term generation drift. The target pose token for the next time step is injected, along with the generated latent state features, as conditions. The predicted token sequence is generated through the internal state autoregressive model module. The generated predicted token sequence is mapped back to the physical space to obtain directly usable autonomous driving scenario data.

[0018] It should be noted that the training phase collects and integrates a multimodal autonomous driving dataset containing video frames and pose data, while the inference phase collects and integrates a multimodal autonomous driving dataset containing video frames as generation conditions, pose data, and pose data as control commands. Furthermore, a multimodal autonomous driving dataset is acquired and preprocessed to obtain a state sequence. The dataset includes video frames and pose data, specifically including: The vehicle pose increment is calculated using a relative coordinate system, and a state sequence containing a preset number of time steps is constructed. Each time step contains the relative pose and the corresponding video frame, and the pose data used as control commands has no corresponding video frame. Specifically, a multimodal relative coordinate system is used to process vehicle pose data (heading angle θ, coordinates (x, y)) and calculate the current time step. Compared to the previous moment Increment: (1) The construction results in the inclusion The state sequence at each time step The state data at each time step includes the relative pose. Video frames corresponding to the time step It should be noted that the pose used as a control command does not have a corresponding video frame.

[0019] The state sequence is transformed into a discrete token sequence through different discretization methods, including pose token sequences and image token sequences, specifically including: The vehicle pose increment is discretized into pose tokens using an equal-spacing binning strategy. A pre-trained temporally aware vector quantization encoder is used. The temporally aware vector quantization encoder is based on an improved VQ-GAN architecture. The temporally aware vector embeds a temporal self-attention mechanism. The quantization encoder downsamples and compresses video frames into feature maps. After processing by the temporal self-attention layer, it assigns discrete image tokens to each spatial position of the feature map based on a preset codebook and minimum distance matching. It outputs a pose token sequence and an image token sequence as subsequent training data. It should be noted that it will include Frame state sequence Convert them into discrete token sequences so that the Transformer model can process them.

[0020] 1. Vehicle pose discretization: An equally spaced binning strategy is adopted. The relative heading angle is discretized. Mapped to Each category will have a relative position. Mapped to There are several categories. The calculation formula is: (2) (3) in The number of containers indicates the heading angle. , These represent the number of bins along the x-axis and y-axis, respectively. These are the generated heading angle token and position token, respectively. This indicates a floor operation. Preset range for heading angle, Preset the range for the position coordinates.

[0021] 2. Image Temporal Aware Quantization: Utilizing a pre-trained temporal awareness vector quantization encoder, the input image... That is, the video frames are converted into feature maps. Before quantization, the data is processed through a time-dependent attention layer and calculated as follows: (4) (5) in express Video frames at any given moment express Compressed feature map, This indicates the encoder's downsampling compression operation. : Indicates time In feature map Spatial location Discrete image token at the location, Indicates the codebook size. For codebook, Indicates the first in the codebook One vector; This represents a temporal self-attention operation performed along the time dimension. Indicates time In feature map Spatial location The eigenvector at that location; This indicates taking the first element in the time series. One characteristic, Representation of feature map Height and width.

[0022] Final output pose token sequence and image token sequence .

[0023] Furthermore, the obtained pose token sequence and image token sequence are input into the spatiotemporal multimodal fusion module of the world model. Through alternating stacked temporal Transformer layers and spatial multimodal Transformer layers, temporal and spatial information are decoupled to predict the potential state features of the next time step, including: The pose token sequence at each time step is concatenated with the image token sequence to form a multimodal token sequence, which is then mapped into a latent feature vector through an embedding layer. Employing a temporal causal self-attention mechanism, processed through a temporal Transformer layer, each token focuses only on tokens at the same position in historical frames, capturing the patterns of temporal evolution. A spatial multimodal self-attention mechanism is adopted, which is processed through a spatial multimodal Transformer layer to enable multimodal token interaction at the same time step and integrate spatial and semantic information; By alternately stacking N layers of temporal Transformer layers and spatial multimodal Transformer layers, the output is the next state feature containing spatiotemporal information of all preceding time steps.

[0024] It should be noted that the obtained token sequence is input into the spatiotemporal multimodal fusion module of the world model. This module aims to decouple temporal and spatial information and predict the potential state features of the next time step. This module consists of alternating stacked temporal Transformer layers and spatial multimodal Transformer layers, and the specific processing is as follows: 1. Feature Mapping: First, the discrete image tokens are mapped... and position token The feature vectors are mapped to the latent space through the embedding layer. : (6) (7) in represent Discrete token sequences of time-sequence images and poses This represents the mapping operation and time position encoding of the embedding layer.

[0025] 2. Temporal Transformer Layer Processing: Employs a temporal causal self-attention mechanism. For the first... Each token position focuses only on tokens at the same position in historical frames to capture patterns of evolution over time. (8) in Represents the time Transformer layer. This represents the intermediate features after processing by the time-transformer layer.

[0026] 3. Spatial Multimodal Transformer Layer Processing: A spatial multimodal self-attention mechanism is employed. At the same time step... Within this framework, tokens from all modalities (images, poses) are allowed to be visible to each other and interact, thus fusing spatial and semantic information. (9) in Represents a spatial multimodal Transformer layer. This represents the state characteristics after fusion.

[0027] go through After stacking the temporal Transformer layer and the spatial multimodal Transformer layer, the module output includes the previous... Next-state features of complete spatiotemporal information of a frame , Furthermore, the input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random masking strategy is employed to simulate prediction errors during inference and prevent long-term generation drift. The target pose token for the next time step is injected, along with the already generated next-state features, as conditions. Through the internal state autoregressive model module, a predicted token sequence is generated, specifically including: The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random mask is triggered with a preset probability to replace the image tokens with noise proportionally. The cross-entropy loss between the predicted tokens and the ground truth tokens is calculated to update the model parameters. The target pose token of the next time step is injected as the preceding token. A causal mask self-attention mechanism is adopted. Based on the features of the next state and the preceding token, the current token is predicted sequentially through the internal state autoregression module. Generation is triggered by the first token of the sequence, and the first two tokens are forced to be the target pose token. It's important to note the training phase steps: During training, the preceding token input is the ground truth token from the dataset. To simulate prediction errors during inference and prevent long-term drift, a random masking strategy is introduced: a masking operation is triggered with a preset probability, and in the triggered samples, image tokens are randomly replaced with random noise tokens at a specific ratio. The model uses the masked sequence to predict the complete sequence and calculates the cross-entropy loss between the predicted and ground truth values ​​to update the parameters. (10) in This represents the value of the loss function. Indicates the next moment The A true value token, Indicates the included time of the output. And the predicted multimodal token sequence based on previous information, Indicates time Inner The sequence of truth tokens preceding each position, i.e., the preceding token.

[0028] Reasoning stage steps: Inject control signals to set the target pose token for the next moment. The preceding token is input into the autoregressive sequence; a causal mask self-attention mechanism is used to predict the multimodal token sequence generated in the previous stage. Predict the current token based on the generated previous tokens. : (11) in This represents the internal state autoregressive model module. This token represents the start of the sequence and is used to trigger the autoregressive generation process. Control signals that are required to be used at certain times.

[0029] Furthermore, the generated predicted token sequence is mapped back to physical space to obtain directly usable autonomous driving scenario data, including: The target pose token is restored by binning inverse operation as a relative increment, and the absolute pose at the current moment is calculated by combining the absolute pose at the previous moment. The image token is input into the time-aware decoder, the codebook feature vector is retrieved and processed by time self-attention, and then upsampled to reconstruct the video frame. If the preset number of frames is not reached, the newly generated video frames and the target pose are added to the historical sequence, the earliest frame is removed, and the process returns to the multimodal feature fusion stage to continue generation; if the preset number of frames is reached, the autonomous driving scene video data is synthesized. It should be noted that the process of obtaining the absolute pose by decoding the pose token through the binning inverse function and reconstructing the image using a time-aware decoder symmetrical to the improved time-aware vector quantization encoder includes: The generated predicted value token sequence Map back to physical space.

[0030] 1. Pose Decoding: Utilizing the inverse function of the formula mentioned above (symbols have been given above); (12) The target pose token used in this operation is restored to a relative value, and the vehicle's position is calculated. The absolute position at any given moment.

[0031] 2. Image Decoding: The generated image token sequence is input into the time-aware decoder. The decoder first checks the codebook... Retrieve the corresponding feature vector in the middle Then, after processing by the temporal self-attention module in the decoding layer, the inter-frame coherence is enhanced: (13) Finally, a high-fidelity image of the next frame is reconstructed. .

[0032] (14) in This indicates the upsampling reconstruction operation of the decoder.

[0033] 3. Loop Generation: Determine if the current number of generated frames has reached a preset value. If not, generate a new image. The executed target pose is added to the history sequence as a new one. The system retrieves real-time data and, to maintain a fixed sequence length, removes the earliest frame from the sequence. It then returns to the multimodal feature fusion stage to predict and generate the next frame. If this is achieved, the synthesized video frame represents the autonomous driving scene video data.

[0034] A multimodal autonomous driving scene generation device based on autoregressive closed-loop prediction includes: Encoding module: Acquires a multimodal autonomous driving dataset and preprocesses it to obtain a state sequence, the dataset including video frames and pose data; converts the state sequence into discrete token sequences, including pose token sequences and image token sequences, through different discretization methods; Fusion Module: The spatiotemporal multimodal fusion module inputs the obtained pose token sequence and image token sequence into the world model. Through alternating stacked temporal Transformer layers and spatial multimodal Transformer layers, it decouples temporal and spatial information and predicts the potential state features of the next moment. Autoregressive module: Input the true token sequence corresponding to the dataset, including true pose token and true image token; adopt a random masking strategy to simulate the prediction error during inference and prevent long-term generation drift, inject the target pose token of the next time step, and use the generated latent state features as conditions to generate the predicted token sequence through the internal state autoregressive model module. Decoding module: Maps the generated predicted token sequence back to the physical space to obtain directly usable autonomous driving scenario data.

[0035] Furthermore, the encoding module is used for: The vehicle pose increment is calculated using a relative coordinate system, and a state sequence containing a preset number of time steps is constructed. Each time step contains the relative pose and the corresponding video frame, and the pose data used as control commands has no corresponding video frame. The vehicle pose increment is discretized into pose tokens using an equal-spacing binning strategy. A pre-trained temporally aware vector quantization encoder, which is based on an improved VQ-GAN architecture, is used to downsample and compress video frames into feature maps. After processing by a temporally self-attention layer, discrete image tokens are assigned to each spatial location of the feature map based on a preset codebook through minimum distance matching. The output pose token sequence and image token sequence are used as subsequent training data.

[0036] Furthermore, such as Figure 4 As shown, the fusion module is a spatiotemporal multimodal fusion module with a 12-layer stacked structure. Each layer integrates a temporal causal self-attention mechanism and a spatial multimodal self-attention mechanism for: The pose token sequence at each time step is concatenated with the image token sequence to form a multimodal token sequence, which is then mapped into a latent feature vector through an embedding layer. Employing a temporal causal self-attention mechanism, processed through a temporal Transformer layer, each token focuses only on tokens at the same position in historical frames, capturing the patterns of temporal evolution. A spatial multimodal self-attention mechanism is adopted, which is processed through a spatial multimodal Transformer layer to enable multimodal token interaction at the same time step and fuse spatial information; By alternately stacking N layers of temporal Transformer layers and spatial multimodal Transformer layers, the output is the next state feature containing spatiotemporal information of all preceding time steps.

[0037] Furthermore, such as Figure 5 As shown, the autoregressive module is an internal state autoregressive module with a 6-layer decoder-only structure, used for: The sequence of ground truth tokens corresponding to the input dataset is used as the preceding token, including ground truth pose token and ground truth image token. A random mask is triggered with a preset probability, and the image token is replaced with noise proportionally. The cross-entropy loss between the predicted token and the ground truth token is calculated to update the model parameters. The target pose token of the next time step is injected as the preceding token. A causal mask self-attention mechanism is adopted. Based on the features of the next state and the preceding token, the current token is predicted sequentially through the internal state autoregression module. Generation is triggered by the sequence start token, and the first two tokens are forced to be the target pose token.

[0038] Furthermore, the decoding module maps the generated predicted token sequence back to physical space, enabling pose decoding, image reconstruction, and loop generation, and outputting autonomous driving scene video data, including: The target pose token is restored by binning inverse operation as a relative increment, and the absolute pose at the current moment is calculated by combining the absolute pose at the previous moment. The image token is input into the time-aware decoder, the codebook feature vector is retrieved and processed by time self-attention, and then upsampled to reconstruct the video frame. If the preset number of frames is not reached, the newly generated video frames and the target pose are added to the historical sequence, the earliest frame is removed, and the process returns to the multimodal feature fusion stage to continue generation; if the preset number of frames is reached, the autonomous driving scene video data is synthesized.

[0039] An electronic device includes: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method.

[0040] A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described therein. Example 1 This embodiment implements the autonomous driving world model described in this invention on a high-performance computer using the PyTorch deep learning framework. The experimental data used in this embodiment comes from the publicly available large-scale autonomous driving dataset nuPlan, which provides rich, high-quality forward-looking camera video streams and synchronized vehicle kinematic state data.

[0041] In the data preparation and preprocessing stage, we first extracted the raw video data from the forward-facing camera from the nuPlan dataset, setting the sampling frequency to 10Hz. To ensure both visual feature clarity and computational efficiency, we uniformly scaled and cropped the resolution of all video frames to 256×512 pixels. For the vehicle pose data, we extracted the vehicle's global coordinates provided by the dataset. and heading angle This is then converted into a relative coordinate system centered on the vehicle. Specifically, for each time step, the position increment of the current frame relative to the previous frame is calculated. and heading angle increment To achieve Transformer-based discretized modeling, we performed high-precision binning on the continuous pose values: the range of relative heading angle values ​​was divided into 512 discrete categories, and the range of relative position coordinate X-axis and Y-axis values ​​were each divided into 128 discrete categories. Thus, the vehicle control state of each frame is encoded as a high-precision heading angle token and a coordinate position token (including lateral and longitudinal coordinates).

[0042] In the model building phase, a time-aware vector quantization encoder was first built and trained. This encoder is an improvement on the VQ-GAN architecture, with the following network configuration: the number of latent variable channels in the encoder output is set to 256, the quantization codebook size is set to 16384, and the dimension of each embedding vector in the codebook is set to 32. To address the temporal inconsistency problem caused by traditional single-frame quantization, a spatiotemporal Transformer module was inserted after the encoder's convolutional output and before the quantization layer, and before the decoder's input layer. During training, 15 consecutive video frames were input as a sample segment. The model compressed the video into a discrete token sequence through the encoder and reconstructed it through the decoder. By jointly optimizing the reconstruction loss, perceptual loss, and adversarial loss, an encoder capable of compressing 256×512 video frames into a 16×32 discrete token matrix was trained.

[0043] Based on this, the main architecture of the world model is constructed. The model adopts a spatiotemporally decoupled GPT architecture, and the model structure consists of two cascaded parts: the first part is a spatiotemporal multimodal fusion module with 12 layers, each layer integrating a temporal causal self-attention mechanism and a spatial multimodal self-attention mechanism, with a hidden layer dimension of 1536, used to extract the next state features from the input 15-frame historical token sequence; the second part is an internal state autoregressive module with 6 layers, adopting a decoder-only structure, responsible for generating future token sequences based on the extracted features.

[0044] The model training employs a self-supervised learning strategy. We use the AdamW optimizer with a fixed learning rate. To prevent content drift during long sequence generation, we implemented a random masking strategy: during training iterations, we decided whether to mask the current sample with a 50% probability; for selected samples, we randomly replaced the real image token with a noise token at a rate of 30%, forcing the model to learn to repair the scene using the remaining context information.

[0045] Finally, inference and verification were performed. The trained model was deployed in an inference environment, with a 15-frame historical video clip and pose sequence as context, and an external control command specifying the future target trajectory. The model first extracts historical features through a fusion module, then injects the target pose as a prefix into an autoregressive module to generate future image tokens frame by frame, which are then decoded back into a video stream. Experimental results show that the system can stably generate autonomous driving simulation videos exceeding 40 seconds in length at a frame rate of 10Hz, while maintaining a highly coherent scene structure, verifying the effectiveness of this invention in generating long-duration controllable videos.

[0046] It is worth noting that (1) this application adopts a spatiotemporal decoupling architecture design, in which alternating stacked temporal Transformer layers and spatial multimodal Transformer layers are used in the spatiotemporal multimodal fusion module, replacing the traditional serial computation mode of RNN. The temporal Transformer layer uses a temporal causal self-attention mechanism to ensure that each token only focuses on tokens at the same position in historical frames, capturing the temporal evolution pattern while supporting parallel processing; the spatial multimodal Transformer layer realizes efficient interaction of multimodal tokens within the same time step, fully integrating spatial and semantic information. This architecture design fully leverages the parallel computation advantages of the Transformer model, significantly improves training efficiency, and can quickly process large-scale autonomous driving datasets. At the same time, this application achieves efficient carrying of complex features through a refined token discretization strategy. For pose data, an equidistant binning strategy is employed to discretize continuous pose increments into high-precision pose tokens, ensuring no loss of detail in vehicle motion. For image data, a temporally aware vector quantization encoder based on an improved VQ-GAN architecture downsamples and compresses video frames into feature maps, processes them through a temporal self-attention layer, and then assigns discrete image tokens based on a pre-defined codebook. This reduces data dimensionality while preserving the spatial structure and temporal continuity of the images. In this way, the model can effectively carry visual details and spatiotemporal information in complex scenes, laying the foundation for generating high-fidelity scene data.

[0047] (2) This application effectively solves the problems of spatial structure destruction and lack of physical consistency through spatiotemporal decoupling and multimodal fusion technology. In the feature processing stage, the pose token sequence and the image token sequence are concatenated to form a multimodal token sequence, which is then mapped to a latent feature vector through the embedding layer, rather than simply flattening it, thus fully preserving the spatial structure information of the video frame. The spatial multimodal self-attention mechanism enables the pose token and the image token within the same time step to fully interact, ensuring that the spatial relationship between the vehicle position, pose and the surrounding environment in the generated scene remains consistent, avoiding problems such as road boundary disappearance and vehicle shape distortion. In terms of ensuring physical consistency, this application uses a relative coordinate system to calculate the vehicle pose increment and constructs a state sequence containing a preset number of time steps, enabling the model to learn the physical laws of vehicle motion. The random masking strategy introduced in the training stage replaces the image token with noise proportionally to simulate the prediction error during inference, forcing the model to learn robust feature representations and effectively preventing drift phenomena in the long-term generation process. During the inference phase, the target pose token for the next moment is injected as a control signal. Combined with the causal masking mechanism of the internal state autoregressive model module, the generated vehicle trajectory is ensured to conform to the laws of physical motion, avoiding unreasonable situations such as passing through walls or driving in the wrong direction.

[0048] (3) This application fundamentally solves the temporal consistency problem through a time-aware encoding and decoding mechanism and a cyclic generation strategy. In the encoding stage, the time-aware vector quantization encoder embeds a time self-attention mechanism to perform temporal dimension correlation processing on the feature maps of consecutive video frames, ensuring that the image token sequence can reflect the temporal evolution relationship between frames; in the decoding stage, the time-aware decoder retrieves the codebook feature vector and processes it through time self-attention, and then upsamples and reconstructs the video frames, effectively eliminating inter-frame flicker and jitter. To address the problem of error accumulation during long-term generation, this application designs a closed-loop cyclic generation mechanism. During the generation process, when the preset number of frames has not been reached, the newly generated video frames and target poses are added to the historical sequence, while the earliest frame is removed, keeping the sequence length fixed, and returning to the multimodal feature fusion stage to continue generation. This approach enables the model to dynamically adjust subsequent predictions based on the latest generation results, effectively suppressing error accumulation and avoiding scene collapse.

[0049] (4) This application achieves long-term, controllable scene generation through a target pose-guided autoregressive generation mechanism. During the inference phase, the target pose token for the next moment is injected into the autoregressive sequence as a preceding token, which, together with the generated potential state features, serves as the generation condition. The current token is predicted sequentially by the internal state autoregressive module. During the generation process, the sequence's initial token triggers generation, and the first two tokens are forced to be the target pose token, ensuring that the generated scene strictly follows the preset driving trajectory. Combined with the cyclic generation strategy, the model can continuously generate scene data that conforms to the target trajectory, meeting the requirements for long-term scene generation.

[0050] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for generating autonomous driving scenarios based on a spatiotemporally decoupled world model, characterized in that, Includes the following steps: A multimodal autonomous driving dataset is acquired and preprocessed to obtain a state sequence, the dataset including video frames and pose data; the state sequence is then converted into a discrete token sequence, the token sequence including a pose token sequence and an image token sequence. The obtained pose token sequence and image token sequence are input into the spatiotemporal multimodal fusion module of the world model. By alternately stacking the temporal Transformer layer and the spatial multimodal Transformer layer, the temporal and spatial information are decoupled to predict the potential state features of the next moment. The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random masking strategy is used to simulate prediction errors during inference and prevent long-term generation drift. The target pose token for the next time step is injected, along with the generated latent state features, as conditions. The predicted token sequence is then generated through the internal state autoregressive model module. The generated predicted token sequence is mapped back to the physical space to obtain directly usable autonomous driving scenario data.

2. The autonomous driving scene generation method based on a spatiotemporal decoupled world model according to claim 1, characterized in that, A multimodal autonomous driving dataset is acquired and preprocessed to obtain a state sequence. The dataset includes video frames and pose data. The state sequence is transformed into a discrete token sequence, including a pose token sequence and an image token sequence, specifically including: The vehicle pose increment is calculated using a relative coordinate system, and a state sequence containing a preset number of time steps is constructed. Each time step contains the relative pose and the corresponding video frame, and the pose data used as control commands has no corresponding video frame. The vehicle pose increment is discretized into pose tokens using an equal-spacing binning strategy. A pre-trained temporally aware vector quantization encoder, which is based on an improved VQ-GAN architecture and incorporates a temporal self-attention mechanism, is used to downsample and compress video frames into feature maps. After processing by a temporal self-attention layer, discrete image tokens are assigned to each spatial location of the feature map based on a preset codebook through minimum distance matching. The output pose token sequence and image token sequence are then used as subsequent training data.

3. The autonomous driving scene generation method based on a spatiotemporal decoupled world model according to claim 2, characterized in that, The obtained pose token sequence and image token sequence are input into the spatiotemporal multimodal fusion module of the world model. Through alternating stacked temporal and spatial multimodal Transformer layers, temporal and spatial information are decoupled to predict the latent state features for the next time step, including: The pose token sequence at each time step is concatenated with the image token sequence to form a multimodal token sequence, which is then mapped into a latent feature vector through an embedding layer. Employing a temporal causal self-attention mechanism, processed through a temporal Transformer layer, each token focuses only on tokens at the same position in historical frames, capturing the patterns of temporal evolution. A spatial multimodal self-attention mechanism is adopted, which is processed through a spatial multimodal Transformer layer to enable multimodal token interaction at the same time step and integrate spatial and semantic information; By alternately stacking N layers of temporal Transformer layers and spatial multimodal Transformer layers, the output is a latent state feature containing spatiotemporal information from all preceding time steps.

4. The autonomous driving scene generation method based on a spatiotemporal decoupled world model according to claim 3, characterized in that, The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random masking strategy is employed to simulate prediction errors during inference and prevent long-term generation drift. The target pose token for the next time step is injected, along with the generated latent state features, as conditions. An internal state autoregressive model module then generates a predicted token sequence, specifically including: The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random mask is triggered with a preset probability to replace the image tokens with noise proportionally. The cross-entropy loss between the predicted tokens and the ground truth tokens is calculated to update the model parameters. The target pose token of the next time step is injected as the preceding token. A causal mask self-attention mechanism is adopted. Based on the features of the next state and the preceding token, the current token is predicted sequentially through the internal state autoregression module. Generation is triggered by the sequence start token, and the first two tokens are forced to be the target pose token.

5. The autonomous driving scene generation method based on a spatiotemporal decoupled world model according to claim 4, characterized in that, The generated predicted token sequence is mapped back to physical space to obtain directly usable autonomous driving scenario data, including: The target pose token is restored by binning inverse operation as a relative increment, and the absolute pose at the current moment is calculated by combining the absolute pose at the previous moment. The image token is input into the time-aware decoder, the codebook feature vector is retrieved and processed by time self-attention, and then upsampled to reconstruct the video frame. If the preset number of frames is not reached, the newly generated video frames and the target pose are added to the historical sequence, the earliest frame is removed, and the process returns to the multimodal feature fusion stage to continue generation; if the preset number of frames is reached, the autonomous driving scene video data is synthesized.

6. A multimodal autonomous driving scene generation device based on autoregressive closed-loop prediction, characterized in that, include: Encoding module: Acquires a multimodal autonomous driving dataset and preprocesses it to obtain a state sequence. The dataset includes video frames and pose data. The state sequence is transformed into a discrete token sequence, which includes a pose token sequence and an image token sequence. Fusion Module: The spatiotemporal multimodal fusion module inputs the obtained pose token sequence and image token sequence into the world model. Through alternating stacked temporal Transformer layers and spatial multimodal Transformer layers, it decouples temporal and spatial information and predicts the potential state features of the next moment. Autoregressive module: Input the ground truth token sequence corresponding to the dataset, including ground truth pose token and ground truth image token; adopt a random masking strategy to simulate the prediction error during inference and prevent long-term generation drift; inject the target pose token of the next time step, and use the generated latent state features as conditions to generate the predicted token sequence through the internal state autoregressive model module. Decoding module: Maps the generated predicted token sequence back to the physical space to obtain directly usable autonomous driving scenario data.

7. The autonomous driving scene generation device based on a spatiotemporal decoupled world model according to claim 6, characterized in that, Encoding module, used for: The vehicle pose increment is calculated using a relative coordinate system, and a state sequence containing a preset number of time steps is constructed. Each time step contains the relative pose and the corresponding video frame, and the pose data used as control commands has no corresponding video frame. The vehicle pose increment is discretized into pose tokens using an equal-spacing binning strategy. A pre-trained temporally aware vector quantization encoder, which is based on an improved VQ-GAN architecture, is used to downsample and compress video frames into feature maps. After processing by a temporally self-attention layer, discrete image tokens are assigned to each spatial location of the feature map based on a preset codebook through minimum distance matching. The output pose token sequence and image token sequence are used as subsequent training data.

8. The autonomous driving scene generation device based on a spatiotemporal decoupled world model according to claim 7, characterized in that, The fusion module is a spatiotemporal multimodal fusion module with a 12-layer stacked structure. Each layer integrates a temporal causal self-attention mechanism and a spatial multimodal self-attention mechanism for: The pose token sequence at each time step is concatenated with the image token sequence to form a multimodal token sequence, which is then mapped into a latent feature vector through an embedding layer. Employing a temporal causal self-attention mechanism, processed through a temporal Transformer layer, each token focuses only on tokens at the same position in historical frames, capturing the patterns of temporal evolution. A spatial multimodal self-attention mechanism is adopted, which is processed through a spatial multimodal Transformer layer to enable multimodal token interaction at the same time step and integrate spatial and semantic information; Alternately stack 12 layers of temporal Transformer layers and spatial multimodal Transformer layers to output the next state feature containing spatiotemporal information of all preceding time steps.

9. The autonomous driving scene generation device based on a spatiotemporal decoupled world model according to claim 8, characterized in that, The autoregressive module is an internal state autoregressive module with a 6-layer decoder-only structure, used for: The input dataset contains a sequence of ground truth tokens, including ground truth pose tokens and ground truth image tokens. A random mask is triggered with a preset probability to replace the image tokens with noise proportionally. The cross-entropy loss between the predicted tokens and the ground truth tokens is calculated to update the model parameters. The target pose token of the next time step is injected as the preceding token. A causal mask self-attention mechanism is adopted. Based on the features of the next state and the preceding token, the current token is predicted sequentially through the internal state autoregression module. Generation is triggered by the sequence start token, and the first two tokens are forced to be the target pose token.

10. The autonomous driving scene generation device based on a spatiotemporally decoupled world model according to claim 9, characterized in that, The decoding module maps the generated predicted token sequence back to physical space, enabling pose decoding, image reconstruction, and loop generation, and outputting autonomous driving scene video data, including: The target pose token is restored by binning inverse operation as a relative increment, and the absolute pose at the current moment is calculated by combining the absolute pose at the previous moment. The image token is input into the time-aware decoder, the codebook feature vector is retrieved and processed by time self-attention, and then upsampled to reconstruct the video frame. If the preset number of frames is not reached, the newly generated video frames and the target pose are added to the historical sequence, the earliest frame is removed, and the process returns to the multimodal feature fusion stage to continue generation; if the preset number of frames is reached, the autonomous driving scene video data is synthesized.

Images