Methods and devices for generating driving trajectories, storage media, and electronic devices.

By using a spatially perceptive compressed world encoder and a dynamic latent world model, the problems of high computational overhead and insufficient 3D spatial understanding in end-to-end autonomous driving are solved, enabling efficient prediction of future world states and trajectory planning.

CN123104692APending Publication Date: 2026-11-17CHONGQING CHANGAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611314178.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-27
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing end-to-end autonomous driving methods rely on complex perception annotations, which incur high computational costs and lack a deep understanding of 3D space. Current technologies have failed to effectively solve this problem.

Method used

By employing a Spatial Aware Compressed World Encoder (SCWE) and a Dynamic Latent World Model (DLWM), and through a self-attention mechanism and a causal Transformer decoder, we compress visual information and predict future world states to generate optimal trajectories.

Benefits of technology

Significantly reduces computational overhead, enhances 3D spatial perception capabilities and trajectory planning efficiency, and enables efficient prediction of future world states.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123104692A_ABST
    Figure CN123104692A_ABST
Patent Text Reader

Abstract

This invention provides a method, apparatus, storage medium, and electronic device for generating driving trajectories. The method includes: acquiring continuous frames captured in real-time from a target vehicle, each frame including a multi-view image and the vehicle's state; acquiring compressed scene representation features of the multi-view images and vehicle motion state features of the vehicle state; generating a current world state representation using the compressed scene representation features and the vehicle motion state features; predicting a future world state representation based on DLWM (Dynamic World View Model); generating multiple candidate trajectories using the current world state representation and the future world state representation; and selecting the optimal trajectory from the multiple candidate trajectories. This embodiment solves the technical problem of high computational overhead caused by explicit video-based path generation in existing technologies, reducing computational overhead and significantly improving the efficiency of subsequent trajectory planning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and more specifically, to a method and apparatus for generating driving trajectories, a storage medium, and an electronic device. Background Technology

[0002] Among related technologies, end-to-end autonomous driving has attracted much attention from academia and industry in recent years due to its data-driven nature and scalability. Previous methods typically integrate perception and prediction into a unified differentiable network, extracting planning features through dense bird's-eye view (BEV) or vectorized representations. However, these methods heavily rely on complex auxiliary task design and costly perception annotations, greatly limiting the scalability of the models. Recently, world model-based methods have provided dense supervision signals through temporal self-supervised learning, reducing the dependence on perception labels. However, existing technologies still have significant drawbacks: explicit video generation methods incur huge computational overhead, and intermediate features often focus on visual details unrelated to planning; while implicit latent prediction-based methods are more lightweight, the compression of latent representations is insufficient, and they generally lack a deep understanding of 3D space.

[0003] No efficient and accurate solution has yet been found to address the aforementioned issues in the relevant technologies. Summary of the Invention

[0004] This invention provides a method and apparatus for generating driving trajectories, a storage medium, and an electronic device to solve technical problems in related technologies.

[0005] According to an embodiment of the present invention, a method for generating a driving trajectory is provided, comprising: acquiring continuous frames collected in real time from a target vehicle, wherein each frame of the continuous frames includes a multi-view image and a vehicle state; acquiring compressed scene representation features of the multi-view image and acquiring vehicle motion state features of the vehicle state; generating a current world state representation using the compressed scene representation features and the vehicle motion state features; predicting a future world state representation of the current world state representation based on a Dynamic Latent World Model (DLWM); generating multiple candidate trajectories using the current world state representation and the future world state representation, and selecting the optimal trajectory from the multiple candidate trajectories.

[0006] Optionally, obtaining the compressed scene representation features of the multi-view images includes: embedding the multi-view images into image patch tokens, wherein the feature dimensions of the image patch tokens include: time span, number of cameras, and number of image patches; randomly initializing a set of scene query tokens, and concatenating the image patch tokens and the scene query tokens to obtain concatenated features; and using a pre-trained unlabeled distilled visual DINO encoder to extract the compressed scene representation features of the concatenated features, wherein the DINO encoder is used to compress the visual information in the image patch tokens into the tokens through a self-attention mechanism.

[0007] Optionally, extracting compressed scene representation features from the stitched features using a pre-trained DINO encoder includes: calling a lightweight multilayer perceptron (MLP) of the pre-trained DINO encoder, wherein the lightweight MLP is used to project the input features into a latent space; inputting the stitched features into the lightweight MLP, and outputting frame-by-frame, view-by-view scene representations and image tokens respectively; and determining the scene representations and image tokens as compressed scene representation features.

[0008] Optionally, before extracting the compressed scene representation features of the spliced ​​features using a pre-trained DINO encoder, the method further includes: acquiring sample image data, wherein the sample image data is a multi-view image of consecutive sample frames; inputting the sample image data into a geometric basis model to generate sample geometric features, and simultaneously inputting the sample image data into an initial DINO network to output sample image tokens; mapping the sample image tokens to image patch features using a geometric projector; performing layer normalization on the sample geometric features and the image patch features respectively, and then calculating the cosine similarity loss; and training the initial DINO network using the cosine similarity loss distillation to obtain the DINO encoder.

[0009] Optionally, predicting the future world state representation of the current world state representation using DLWM includes: reading the historical world state representation of the target vehicle; generating temporal encoded features using the historical world state representation and the current world state representation; inputting the temporal encoded features into the causal Transformer decoder of the DLWM, and outputting the future world state representation of the target vehicle, wherein the causal Transformer decoder employs a causal attention masking mechanism.

[0010] Optionally, before inputting the temporal encoded features into the causal Transformer decoder of the DLWM, the method further includes: acquiring a sample frame block sequence, wherein the sample frame block sequence includes sample frame blocks of multiple consecutive time steps, each sample frame block including a scene image token and a vehicle state token; for each sample frame block of the sample frame block sequence, associating the scene image token and vehicle state token within each sample frame block through a frame attention mask, and associating only historical sample frame blocks before the current sample frame block across frames through a frame attention mask, to obtain a forced teaching sample sequence; training an initial temporal model using the forced teaching sample sequence to obtain the causal Transformer decoder of the DLWM.

[0011] Optionally, generating temporal coding features using the historical world state representation and the current world state representation includes: determining the 3D coding dimension of the DLWM, wherein the 3D coding dimension includes: time axis, camera spatial position, and state token index; and performing absolute position frequency coding on the historical world state representation and the current world state representation in sequence according to the 3D coding dimension to obtain temporal coding features.

[0012] Optionally, after outputting the future world state representation of the target vehicle, the method further includes: extracting the future vehicle state representation from the future world state representation, wherein the future world state representation includes the future vehicle state representation and the future scene representation; inputting the future vehicle state representation into multiple independent MLP decoders of the DLWM's vehicle state supervision module, and outputting future driving commands, future speeds, and future accelerations respectively, wherein the multiple independent MLP decoders include: a driving command decoder, a speed decoder, and an acceleration decoder.

[0013] Optionally, generating multiple candidate trajectories using the current world state representation and the future world state representation includes: generating a set of randomly initialized learnable trajectory query vectors; inputting the learnable trajectory query vectors, the current world state representation, and the future world state representation into a trajectory decoder, and outputting multiple candidate trajectories.

[0014] Optionally, selecting the optimal trajectory from the multiple candidate trajectories includes: obtaining the control command of the target vehicle at the current time step; and selecting the optimal trajectory that matches the control command from the multiple candidate trajectories.

[0015] According to another embodiment of the present invention, a driving trajectory generation apparatus is provided, comprising: a first acquisition module, configured to acquire continuous frames collected in real time from a target vehicle, wherein each frame of the continuous frames includes a multi-view image and a vehicle state; a second acquisition module, configured to acquire compressed scene representation features of the multi-view image and vehicle motion state features of the vehicle state; a generation module, configured to generate a current world state representation using the compressed scene representation features and the vehicle motion state features; a prediction module, configured to predict a future world state representation of the current world state representation based on a Dynamic Latent World Model (DLWM); and a selection module, configured to generate multiple candidate trajectories using the current world state representation and the future world state representation, and select the optimal trajectory from the multiple candidate trajectories.

[0016] Optionally, the second acquisition module includes: an embedding unit for embedding the multi-view image into an image patch token, wherein the feature dimensions of the image patch token include: time span, number of cameras, and number of image patches; a stitching unit for randomly initializing a set of scene query tokens and stitching the image patch tokens and the scene query tokens to obtain stitched features; and an extraction unit for extracting compressed scene representation features of the stitched features using a pre-trained unlabeled distilled visual DINO encoder, wherein the DINO encoder is used to compress visual information in the image patch tokens into the tokens through a self-attention mechanism.

[0017] Optionally, the stitching unit includes: a calling subunit for calling a pre-trained DINO encoder's lightweight multilayer perceptron (MLP), wherein the lightweight MLP is used to project input features into a latent space; an output subunit for inputting the stitching features into the lightweight MLP and outputting frame-by-frame, view-by-view scene representations and image tokens respectively; and determining the scene representations and image tokens as compressed scene representation features.

[0018] Optionally, the second acquisition module further includes: an acquisition unit, configured to acquire sample image data before the extraction unit extracts the compressed scene representation features of the spliced ​​features using a pre-trained DINO encoder, wherein the sample image data is a multi-view image of consecutive sample frames; a processing unit, configured to input the sample image data into a geometric basis model to generate sample geometric features, and simultaneously input the sample image data into an initial DINO network to output sample image tokens; a mapping unit, configured to map the sample image tokens into image patch features using a geometric projector; a calculation unit, configured to perform layer normalization on the sample geometric features and the image patch features respectively before calculating the cosine similarity loss; and a training unit, configured to train the initial DINO network using the cosine similarity loss distillation to obtain the DINO encoder.

[0019] Optionally, the prediction module includes: a reading unit for reading the historical world state representation of the target vehicle; a generation unit for generating temporal coding features using the historical world state representation and the current world state representation; and a prediction unit for inputting the temporal coding features into the causal Transformer decoder of the DLWM and outputting the future world state representation of the target vehicle, wherein the causal Transformer decoder employs a causal attention masking mechanism.

[0020] Optionally, the prediction module further includes: an acquisition unit, configured to acquire a sample frame block sequence before the prediction unit inputs the temporal encoded features into the causal Transformer decoder of the DLWM, wherein the sample frame block sequence includes sample frame blocks of multiple consecutive time steps, and each sample frame block includes a scene image token and a vehicle state token; an association unit, configured to associate the scene image token and vehicle state token within each sample frame block of the sample frame block sequence with a frame attention mask, and to associate only the historical sample frame blocks before the current sample frame block across frames with a frame attention mask, to obtain a forced teaching sample sequence; and a training unit, configured to train an initial temporal model using the forced teaching sample sequence to obtain the causal Transformer decoder of the DLWM.

[0021] Optionally, the generation unit includes: a determination subunit, used to determine the 3D encoding dimension of the DLWM, wherein the 3D encoding dimension includes: a time axis, camera spatial position, and state token index; and an encoding subunit, used to sequentially encode the historical world state representation and the current world state representation in the 3D encoding dimension using absolute position frequency encoding to obtain temporal encoding features.

[0022] Optionally, the prediction module further includes: an extraction unit, configured to extract the future vehicle state representation from the future world state representation after the prediction unit outputs the future world state representation of the target vehicle, wherein the future world state representation includes the future vehicle state representation and the future scene representation; and a supervision unit, configured to input the future vehicle state representation into multiple independent MLP decoders of the DLWM's vehicle state supervision module, and output future driving commands, future speeds, and future accelerations respectively, wherein the multiple independent MLP decoders include a driving command decoder, a speed decoder, and an acceleration decoder.

[0023] Optionally, the selection module includes: a generation unit for generating a set of randomly initialized learnable trajectory query vectors; and an output unit for inputting the learnable trajectory query vectors, the current world state representation, and the future world state representation into the trajectory decoder and outputting multiple candidate trajectories.

[0024] Optionally, the selection module includes: an acquisition unit for acquiring the control command of the target vehicle at the current time step; and a selection unit for selecting the optimal trajectory that matches the control command from the multiple candidate trajectories.

[0025] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0026] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0027] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.

[0028] The beneficial effects of this invention are: 1. A spatially perceptive compressed world encoder is proposed. The encoder extracts the spatial semantics of the pre-trained geometric foundation model into the visual backbone network through a very small number of learnable queries and geometric knowledge distillation. This greatly reduces the feature dimension while endowing the model with a high degree of 3D spatial perception capability. Relying on the self-attention mechanism, it efficiently compresses the extensive and rich visual information in a large number of image block tokens into a very small number of compact tokens. This not only preserves the key scene features, but also greatly reduces the computational overhead, thereby significantly improving the efficiency of subsequent long-term world model training and trajectory planning. 2. A dynamic latent world model guided by the vehicle's state is proposed. It adopts a forced teaching attention mask and is based on a frame-by-frame attention mask design. While maintaining the causal consistency of the world model's predictions, it can predict all future world states in parallel and efficiently, significantly improving training speed and efficiency. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1This is a hardware structure block diagram of a vehicle according to an embodiment of the present invention; Figure 2 This is a flowchart of a method for generating a driving trajectory according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the hidden world model and causal attention mask design in an embodiment of the present invention; Figure 4 This is a schematic diagram of the end-to-end autonomous driving principle based on the dynamic hidden world model in this embodiment of the invention; Figure 5 This is a flowchart of the end-to-end autonomous driving training method based on the dynamic hidden world model in this embodiment of the invention; Figure 6 This is a structural block diagram of a driving trajectory generation device according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1 The method embodiment provided in Embodiment 1 of this application can be executed in a vehicle, vehicle controller, processor, computer, or similar processing device. Taking its operation in a vehicle as an example, Figure 1 This is a hardware structure block diagram of a vehicle according to an embodiment of the present invention. For example... Figure 1 As shown, a vehicle may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the vehicle may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the vehicle described above. For example, the vehicle may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] The memory 104 can be used to store vehicle programs, such as application software programs and modules, like the vehicle program corresponding to a method for generating a vehicle's driving trajectory in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the vehicle program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the vehicle's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0035] This embodiment provides a method for generating driving trajectories, which can be applied to end-to-end driving scenarios. Figure 2 This is a flowchart of a method for generating a driving trajectory according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain continuous frames collected in real time from the target vehicle, wherein each frame of the continuous frames includes multi-view images and the vehicle status. Optionally, multi-view images can be acquired through vehicle images and radar sensors, or the vehicle status can be read in real time from the vehicle's infotainment system or controller.

[0036] Step S202: Obtain the compressed scene representation features of the multi-view image and the vehicle motion state features of the vehicle state. This embodiment employs a Spatial-Aware Compressive World Encoder (SCWE) to obtain compressed scene representation features from the multi-view images. SCWE, as a scene representation module, consists of a learnable query and a visual encoder forming a feature extractor, compressing the input image information into a small number of compact scene tokens for scene representation. Simultaneously, this module uses a geometrically based model (such as VGGT) to perform knowledge distillation on the encoder, significantly enhancing the spatial understanding capability of the scene representation module. Alternatively, other types of encoders or network models can also be used to obtain compressed scene representation features from the multi-view images.

[0037] Optionally, the vehicle's motion state characteristics include vehicle speed, acceleration, and other vehicle features.

[0038] Step S203: Generate the current world state representation using the compressed scene representation features and the vehicle motion state features; Step S204: Predict the future world state representation of the current world state representation based on the Dynamic Latent World Model (DLWM). The Dynamic Latent World Model (DLWM) in this embodiment is a lightweight causal Transformer decoder. Guided by the vehicle's state and historical scene representations, it predicts future world states through autoregression, achieving temporal dynamic modeling of driving world state transitions using a self-supervised learning paradigm.

[0039] Step S205: Generate multiple candidate trajectories using the current world state representation and the future world state representation, and select the optimal trajectory from the multiple candidate trajectories.

[0040] A trajectory decoder is used to decode the world state representation (current world state representation and future world state representation) output by the feature extractor to predict the future planned trajectory.

[0041] Through the above steps, continuous frames of the target vehicle are acquired in real time, wherein each frame of the continuous frames includes multi-view images and the vehicle's state; compressed scene representation features of the multi-view images and vehicle motion state features of the vehicle's state are acquired; the compressed scene representation features and the vehicle motion state features are used to generate a current world state representation; the future world state representation of the current world state representation is predicted according to the Dynamic Latent World Model (DLWM); multiple candidate trajectories are generated using the current world state representation and the future world state representation, and the optimal trajectory is selected from the multiple candidate trajectories. This solves the technical problem of high computational overhead caused by explicit video generation for path planning in existing technologies, reduces computational overhead, and thus significantly improves the efficiency of subsequent trajectory planning.

[0042] In one implementation scenario of this embodiment, obtaining the compressed scene representation features of the multi-view image includes: embedding the multi-view image into image patch tokens, wherein the feature dimensions of the image patch tokens include: time span, number of cameras, and number of image patches; randomly initializing a set of scene query tokens, and concatenating the image patch tokens and the scene query tokens to obtain concatenated features; and using a pre-trained unlabeled distilled visual DINO encoder to extract the compressed scene representation features of the concatenated features, wherein the DINO encoder is used to compress the visual information in the image patch tokens into the tokens through a self-attention mechanism.

[0043] Multi-view images of consecutive frames are acquired. The images are input into the backbone network via a Spatial Aware Compressed World Encoder (SCWE). A small number of learnable scene queries interact with image block tokens to compress rich scene information into the latent space, resulting in frame-by-frame, view-by-view compressed scene representations. Randomly initialized scene queries are concatenated with image block tokens and input into the DINO (Distillation with NO labels) backbone network. A self-attention mechanism compresses global visual information into a small number of scene query tokens, thereby reducing the computational overhead of subsequent planning modules.

[0044] This embodiment extracts the spatial semantics of the pre-trained geometric foundation model into the visual backbone network through a very small number of learnable queries and geometric knowledge distillation. This greatly reduces the feature dimension while giving the model a high degree of 3D spatial perception capability.

[0045] In one example, extracting compressed scene representation features from the stitched features using a pre-trained DINO encoder includes: invoking a lightweight multilayer perceptron (MLP) of the pre-trained DINO encoder, wherein the lightweight MLP is used to project the input features into a latent space; inputting the stitched features into the lightweight MLP, and outputting frame-by-frame, view-by-view scene representations and image tokens respectively; and determining the scene representations and image tokens as compressed scene representation features.

[0046] In one example, before extracting the compressed scene representation features of the stitched features using a pre-trained DINO encoder, the method further includes: acquiring sample image data, wherein the sample image data is a multi-view image of consecutive sample frames; inputting the sample image data into a geometric basis model to generate sample geometric features, and simultaneously inputting the sample image data into an initial DINO network to output sample image tokens; mapping the sample image tokens to image patch features using a geometric projector; performing layer normalization on the sample geometric features and the image patch features respectively, and then calculating the cosine similarity loss; and training the initial DINO network using the cosine similarity loss to obtain the DINO encoder.

[0047] To distill the powerful spatial understanding capabilities of the geometric foundation model into the DINO encoder, image patch tokens output from the Spatial Aware Compressed World Encoder (SCWE) are selected as carriers to receive dense spatial semantic information from geometric features. This involves multi-view images from consecutive frames. Simultaneously fed into a geometrical fundamental model The latter generates patch-level geometric features. On the other hand, the image tokens output by the DINO backbone network It will be through a geometric projector Perform mapping to obtain Before aligning, first... and Perform layer normalization separately, and then calculate the cosine similarity loss between the two.

[0048] Optionally, geometric features can be extracted and cached offline using a frozen geometric basis model, maximizing the cosine similarity between the projected image patch features and the geometric features during the training phase.

[0049] In one embodiment of this example, predicting the future world state representation of the current world state representation using DLWM includes: reading the historical world state representation of the target vehicle; generating a temporal coding feature using the historical world state representation and the current world state representation; inputting the temporal coding feature into the causal Transformer decoder of the DLWM, and outputting the future world state representation of the target vehicle, wherein the causal Transformer decoder employs a causal attention masking mechanism.

[0050] This embodiment constructs a unified spatiotemporal representation sequence that includes scene and vehicle state, utilizes causal Transformer to jointly learn visual evolution and motion dynamics, and effectively handles complex spatiotemporal coordinates through multimodal rotational position encoding.

[0051] This embodiment predicts future scene representations causally in the latent space, enabling the backbone network to acquire the ability to model temporal dynamics through a self-supervised training paradigm without annotation. Since scene tokens from a single camera can only capture isolated viewpoint-specific information, they are insufficient for global world modeling. To achieve effective future prediction and trajectory planning, this embodiment aggregates these tokens and fuses them with the vehicle's state to construct a unified world state representation frame by frame. Specifically, the vehicle's state (including driving commands, velocity, and acceleration) in consecutive frames is encoded into a vehicle state embedding by a single-layer MLP encoder. Scene token from the spatially-aware compressed world encoder. Embedded with vehicle status Together, they constitute the frame-level world latent state. Scene tokens from different cameras are aggregated to form a holistic perceptual representation of the surrounding environment, which is then combined with a representation of the vehicle's motion state. By piecing together elements, a unified scene is formed – a representation of the state of the autonomous vehicle world. This representation will serve as the basis for predicting future world states and planning trajectories in subsequent latent world models.

[0052] In one example, before inputting the temporal encoded features into the causal Transformer decoder of the DLWM, the method further includes: obtaining a sample frame block sequence, wherein the sample frame block sequence includes sample frame blocks of multiple consecutive time steps, each sample frame block including a scene image token and a vehicle state token; for each sample frame block of the sample frame block sequence, associating the scene image token and vehicle state token within each sample frame block through a frame attention mask, and associating only historical sample frame blocks before the current sample frame block across frames through a frame attention mask, to obtain a forced teaching sample sequence; and training an initial temporal model using the forced teaching sample sequence to obtain the causal Transformer decoder of the DLWM.

[0053] DLWM employs a causal attention masking mechanism to ensure that during sequence generation, the token at each time step only focuses on the tokens of the current frame and historical frames, and predicts all future features in parallel.

[0054] This example defines the dynamic modeling of world state transitions as an autoregressive prediction problem, where all future frame world state predictions are conditioned on historical world state representations (including the overall scene representation and the vehicle's state). This unified autoregressive framework implements a natural training strategy: using alternating scene-vehicle world state token sequences... It is treated as a block-by-frame process, and the world model is trained using standard next-to-toe prediction.

[0055] This embodiment is based on a forced-teaching attention mask. Given a scene—a sequence of alternating vehicle and tokens—the world model predicts each future token by paying attention to all historical tokens. During training, a forced-teaching mechanism is employed: real tokens are used as context to predict subsequent tokens, avoiding the accumulation of errors across different time steps. Causal prediction is achieved through a frame-by-frame attention mask. Within each frame block, tokens can pay attention to each other; however, across frames, each token can only pay attention to tokens that appeared earlier in the sequence, thus forcing temporal causality. This frame-by-frame attention mask design maintains the causal consistency of the world model's predictions while enabling parallel and efficient prediction of all future world states, significantly improving training speed and efficiency.

[0056] Figure 3 This is a schematic diagram of the hidden world model and causal attention mask design in an embodiment of the present invention. T is the total prediction duration, and the sequence of scene image tokens includes... ~ The sequence of vehicle status tokens includes ~ In the autoregressive training framework, the attention mechanism is implemented through an attention mask. The input is a complete sequence of tokens, but the token sequence has a temporal relationship. For the training objective of each token, the attention can only focus on the information of the current frame plus the information of the previous frames, not the information of the future frames. Therefore, it is naturally a triangular attention mask matrix.

[0057] Optionally, generating temporal coding features using the historical world state representation and the current world state representation includes: determining the 3D coding dimension of the DLWM, wherein the 3D coding dimension includes: time axis, camera spatial position, and state token index; and performing absolute position frequency coding on the historical world state representation and the current world state representation in sequence according to the 3D coding dimension to obtain temporal coding features.

[0058] In this embodiment of DLWM, 3D rotational position encoding is introduced, which splits the head dimension of the attention mechanism into three parts, and performs absolute position frequency encoding on the time axis, camera spatial position and state token index respectively.

[0059] Despite the representation of the state of the world It contains world state representations across multiple timestamps and camera views, but the query vector and context features The tokens do not explicitly carry temporal or spatial location information. To enable the model to distinguish the temporal and spatial relationships between tokens, spatiotemporal location information is injected into the multi-head attention mechanism through 3D rotational position encoding to improve the model's understanding.

[0060] Optionally, after outputting the future world state representation of the target vehicle, the method further includes: extracting the future vehicle state representation from the future world state representation, wherein the future world state representation includes the future vehicle state representation and the future scene representation; inputting the future vehicle state representation into multiple independent MLP decoders of the DLWM's vehicle state supervision module, and outputting future driving commands, future speeds, and future accelerations respectively, wherein the multiple independent MLP decoders include: a driving command decoder, a speed decoder, and an acceleration decoder.

[0061] The DLWM in this embodiment also includes a vehicle state supervision module, which extracts the vehicle state representation from the predicted future world state and predicts the future control commands, speed and acceleration through a multilayer perceptron decoder, providing explicit guidance for the evolution of the world state.

[0062] World state representation in this embodiment It also includes scene tokens and vehicle state embeddings. During world model prediction, the vehicle state guides not only the evolution of the scene representation but also its own evolution. Therefore, accurate prediction of the vehicle state is crucial for modeling world dynamics. To provide more accurate guidance for world state transitions, this embodiment introduces vehicle state supervision on top of self-supervised future state prediction. The corresponding vehicle state features are extracted from the predicted future world state and then input into three independent 2-layer MLP decoders: a driving command decoder, a speed decoder, and an acceleration decoder, to obtain the predicted command, speed, and acceleration.

[0063] In one embodiment of this example, generating multiple candidate trajectories using the current world state representation and the future world state representation includes: generating a set of randomly initialized learnable trajectory query vectors; inputting the learnable trajectory query vectors, the current world state representation, and the future world state representation into a trajectory decoder, and outputting multiple candidate trajectories.

[0064] Optionally, selecting the optimal trajectory from the multiple candidate trajectories includes: obtaining the control command of the target vehicle at the current time step; and selecting the optimal trajectory that matches the control command from the multiple candidate trajectories.

[0065] During inference, since the model has internalized powerful spatial and dynamic understanding capabilities during training, high-precision planning can be achieved with only the backbone network and a lightweight trajectory decoder, resulting in inference latency with zero additional perception modules. (Using a trajectory decoder...) Trajectory generation: Capable of generating trajectories for corresponding modes based on control commands. A set of randomly initialized learnable trajectories can be queried. Will be combined with the world state representation of the current frame The input is fed into the track decoder, and after processing, the track token is decoded into... A number of candidate trajectories are generated. Finally, the driving command is given at the current time step t. Select the candidate trajectory of the corresponding mode as the final trajectory. Each decoded candidate trajectory It covers the entire prediction time domain from the current time step t to the future time t+T (where T is the total prediction duration). Each pose is represented as... Therefore, the complete candidate trajectory is located at Space. Within it. and These represent longitudinal and lateral displacements, respectively. This represents the heading angle. All values ​​are expressed in time steps. The local coordinate system of the vehicle is used as the reference, and it is assumed that the time interval between consecutively predicted poses is uniform.

[0066] This invention provides an end-to-end autonomous driving training method based on a dynamic hidden world model, as well as an end-to-end autonomous driving training and system based on a dynamic hidden world model, aiming to address the shortcomings of existing end-to-end autonomous driving models in terms of representation compression, spatial understanding, and temporal dynamic modeling. Figure 4 This is a schematic diagram of the end-to-end autonomous driving principle based on the dynamic latent world model in this embodiment of the invention. It illustrates the training process of the spatial perception compressed world encoder (SCWE) and the dynamic latent world model (DLWM), as well as the trajectory planning process. The training of the dynamic latent world model (DLWM) is achieved by updating the EMA (Exponential Moving Average) to train the world encoder.

[0067] Figure 5This is a flowchart of an end-to-end autonomous driving training method based on a dynamic hidden world model, as described in this invention. The method includes the following steps: Scene feature extraction and compression: Acquire multi-view images of consecutive frames, input the images into the feature extraction backbone network through the Spatial Aware Compressed World Encoder (SCWE), and use a small number of learnable scene queries and image block tokens to interact to compress rich scene information into the latent space, thereby obtaining frame-by-frame and view-by-view compressed scene representations. Spatial geometric capability distillation: The multi-view images are input into a pre-trained geometric foundation model to extract patch-level geometric representations; at the same time, the image patch tokens output by the backbone network in step 1 are up-dimensional through a geometric projection layer and aligned with the above geometric representations. Knowledge distillation is performed by calculating cosine similarity loss to enhance the spatial understanding capability of the backbone network. Dynamic world state modeling and prediction: The compressed scene representation is spliced ​​with the vehicle motion state features to construct a unified world state representation; the causal Transformer decoder in the Dynamic Latent World Model (DLWM) is used to predict the future world state representation in an autoregressive manner. Multimodal trajectory planning: Based on the world state representation of the current frame and the predicted future state, multiple candidate trajectories are generated using a lightweight trajectory decoder, and the final planned trajectory is output according to the current control command.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0069] Example 2 This embodiment also provides a driving trajectory generation device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0070] Figure 6This is a structural block diagram of a driving trajectory generation device according to an embodiment of the present invention, such as... Figure 5 As shown, the device includes: The first acquisition module 61 is used to acquire continuous frames collected in real time from the target vehicle, wherein each frame of the continuous frame includes a multi-view image and the vehicle status. The second acquisition module 62 is used to acquire the compressed scene representation features of the multi-view image and the vehicle motion state features of the vehicle state. Generation module 63 is used to generate a current world state representation using the compressed scene representation features and the vehicle motion state features; Prediction module 64 is used to predict the future world state representation of the current world state representation based on the Dynamic Latent World Model (DLWM). Selection module 65 is used to generate multiple candidate trajectories using the current world state representation and the future world state representation, and to select the optimal trajectory from the multiple candidate trajectories.

[0071] Optionally, the second acquisition module includes: an embedding unit for embedding the multi-view image into an image patch token, wherein the feature dimensions of the image patch token include: time span, number of cameras, and number of image patches; a stitching unit for randomly initializing a set of scene query tokens and stitching the image patch tokens and the scene query tokens to obtain stitched features; and an extraction unit for extracting compressed scene representation features of the stitched features using a pre-trained unlabeled distilled visual DINO encoder, wherein the DINO encoder is used to compress visual information in the image patch tokens into the tokens through a self-attention mechanism.

[0072] Optionally, the stitching unit includes: a calling subunit for calling a pre-trained DINO encoder's lightweight multilayer perceptron (MLP), wherein the lightweight MLP is used to project input features into a latent space; an output subunit for inputting the stitching features into the lightweight MLP and outputting frame-by-frame, view-by-view scene representations and image tokens respectively; and determining the scene representations and image tokens as compressed scene representation features.

[0073] Optionally, the second acquisition module further includes: an acquisition unit, configured to acquire sample image data before the extraction unit extracts the compressed scene representation features of the spliced ​​features using a pre-trained DINO encoder, wherein the sample image data is a multi-view image of consecutive sample frames; a processing unit, configured to input the sample image data into a geometric basis model to generate sample geometric features, and simultaneously input the sample image data into an initial DINO network to output sample image tokens; a mapping unit, configured to map the sample image tokens into image patch features using a geometric projector; a calculation unit, configured to perform layer normalization on the sample geometric features and the image patch features respectively before calculating the cosine similarity loss; and a training unit, configured to train the initial DINO network using the cosine similarity loss distillation to obtain the DINO encoder.

[0074] Optionally, the prediction module includes: a reading unit for reading the historical world state representation of the target vehicle; a generation unit for generating temporal coding features using the historical world state representation and the current world state representation; and a prediction unit for inputting the temporal coding features into the causal Transformer decoder of the DLWM and outputting the future world state representation of the target vehicle, wherein the causal Transformer decoder employs a causal attention masking mechanism.

[0075] Optionally, the prediction module further includes: an acquisition unit, configured to acquire a sample frame block sequence before the prediction unit inputs the temporal encoded features into the causal Transformer decoder of the DLWM, wherein the sample frame block sequence includes sample frame blocks of multiple consecutive time steps, and each sample frame block includes a scene image token and a vehicle state token; an association unit, configured to associate the scene image token and vehicle state token within each sample frame block of the sample frame block sequence with a frame attention mask, and to associate only the historical sample frame blocks before the current sample frame block across frames with a frame attention mask, to obtain a forced teaching sample sequence; and a training unit, configured to train an initial temporal model using the forced teaching sample sequence to obtain the causal Transformer decoder of the DLWM.

[0076] Optionally, the generation unit includes: a determination subunit, used to determine the 3D encoding dimension of the DLWM, wherein the 3D encoding dimension includes: a time axis, camera spatial position, and state token index; and an encoding subunit, used to sequentially encode the historical world state representation and the current world state representation in the 3D encoding dimension using absolute position frequency encoding to obtain temporal encoding features.

[0077] Optionally, the prediction module further includes: an extraction unit, configured to extract the future vehicle state representation from the future world state representation after the prediction unit outputs the future world state representation of the target vehicle, wherein the future world state representation includes the future vehicle state representation and the future scene representation; and a supervision unit, configured to input the future vehicle state representation into multiple independent MLP decoders of the DLWM's vehicle state supervision module, and output future driving commands, future speeds, and future accelerations respectively, wherein the multiple independent MLP decoders include a driving command decoder, a speed decoder, and an acceleration decoder.

[0078] Optionally, the selection module includes: a generation unit for generating a set of randomly initialized learnable trajectory query vectors; and an output unit for inputting the learnable trajectory query vectors, the current world state representation, and the future world state representation into the trajectory decoder and outputting multiple candidate trajectories.

[0079] Optionally, the selection module includes: an acquisition unit for acquiring the control command of the target vehicle at the current time step; and a selection unit for selecting the optimal trajectory that matches the control command from the multiple candidate trajectories.

[0080] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0081] Example 3 Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0082] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1, acquire continuous frames collected in real time from the target vehicle, wherein each frame of the continuous frame includes multi-view images and the vehicle status; S2, obtain the compressed scene representation features of the multi-view image, and obtain the vehicle motion state features of the vehicle state; S3, using the compressed scene representation features and the vehicle motion state features to generate the current world state representation; S4. Predict the future world state representation of the current world state representation based on the Dynamic Latent World Model (DLWM). S5, using the current world state representation and the future world state representation to generate multiple candidate trajectories, and selecting the optimal trajectory from the multiple candidate trajectories.

[0083] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0084] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0085] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0086] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1, acquire continuous frames collected in real time from the target vehicle, wherein each frame of the continuous frame includes multi-view images and the vehicle status; S2, obtain the compressed scene representation features of the multi-view image, and obtain the vehicle motion state features of the vehicle state; S3, using the compressed scene representation features and the vehicle motion state features to generate the current world state representation; S4. Predict the future world state representation of the current world state representation based on the Dynamic Latent World Model (DLWM). S5, using the current world state representation and the future world state representation to generate multiple candidate trajectories, and selecting the optimal trajectory from the multiple candidate trajectories.

[0087] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0091] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating a driving trajectory, characterized in that, include: Acquire continuous frames of the target vehicle in real time, wherein each frame of the continuous frame includes multi-view images and the vehicle's status; Obtain the compressed scene representation features of the multi-view images, and obtain the vehicle motion state features of the vehicle state; The current world state representation is generated using the compressed scene representation features and the vehicle motion state features. Predict the future world state representation based on the current world state representation using the Dynamic Latent World Model (DLWM). Multiple candidate trajectories are generated using the current world state representation and the future world state representation, and the optimal trajectory is selected from the multiple candidate trajectories.

2. The method according to claim 1, characterized in that, Obtaining the compressed scene representation features of the multi-view images includes: The multi-view images are embedded as image block tokens, wherein the feature dimensions of the image block tokens include: time span, number of cameras, and number of image blocks; A set of scene query tokens is randomly initialized, and the image block tokens and the scene query tokens are concatenated to obtain the concatenated features; A pre-trained unlabeled distilled visual DINO encoder is used to extract compressed scene representation features of the stitched features, wherein the DINO encoder is used to compress visual information in the image block token into the token through a self-attention mechanism.

3. The method according to claim 2, characterized in that, The compressed scene representation features extracted using the pre-trained DINO encoder include: Invoke a pre-trained DINO encoder's lightweight multilayer perceptron (MLP), wherein the lightweight MLP is used to project input features into a latent space; The stitching features are input into the lightweight MLP, which outputs a scene representation and an image token frame by frame and view by view, respectively. The scene representation and the image token are identified as compressed scene representation features.

4. The method according to claim 2, characterized in that, Before extracting the compressed scene representation features of the spliced ​​features using a pre-trained DINO encoder, the method further includes: Acquire sample image data, wherein the sample image data is a multi-view image of consecutive sample frames; The sample image data is input into the geometric basis model to generate sample geometric features. At the same time, the sample image data is input into the initial DINO network to output sample image tokens. The sample image tokens are mapped to image patch features using a geometric projector; The sample geometric features and the image patch features are respectively subjected to layer normalization before the cosine similarity loss is calculated. The initial DINO network is trained using the cosine similarity loss distillation method to obtain the DINO encoder.

5. The method according to claim 1, characterized in that, The future world state representation predicted using DLWM includes: Read the historical world state representation of the target vehicle; Temporal coding features are generated using the historical world state representation and the current world state representation; The temporal coding features are input into the causal Transformer decoder of the DLWM to output the future world state representation of the target vehicle, wherein the causal Transformer decoder employs a causal attention masking mechanism.

6. The method according to claim 5, characterized in that, Before inputting the temporal coding features into the causal Transformer decoder of the DLWM, the method further includes: Obtain a sample frame block sequence, wherein the sample frame block sequence includes multiple sample frame blocks at consecutive time steps, and each sample frame block includes a scene image token and a vehicle status token; For each sample frame block in the sample frame block sequence, the scene image token and vehicle status token within each sample frame block are associated through a frame attention mask, and the historical sample frame blocks before the current sample frame block are associated only across frames through the frame attention mask, thus obtaining a forced teaching sample sequence. The initial temporal model is trained using the forced teaching sample sequence to obtain the causal Transformer decoder of the DLWM.

7. The method according to claim 5, characterized in that, The generation of temporal coding features using the historical world state representation and the current world state representation includes: The 3D encoding dimensions of the DLWM are determined, wherein the 3D encoding dimensions include: time axis, camera spatial position, and state token index; The historical world state representation and the current world state representation are sequentially encoded using absolute position frequency in the 3D encoding dimension to obtain temporal encoding features.

8. The method according to claim 5, characterized in that, After outputting the future world state representation of the target vehicle, the method further includes: Extract the future vehicle state representation from the future world state representation, wherein the future world state representation includes the future vehicle state representation and the future scene representation; The future vehicle state representation is input into multiple independent MLP decoders of the DLWM's vehicle state supervision module, and the future driving command, future speed, and future acceleration are output respectively. The multiple independent MLP decoders include a driving command decoder, a speed decoder, and an acceleration decoder.

9. The method according to claim 1, characterized in that, Generating multiple candidate trajectories using the current world state representation and the future world state representation includes: Generate a set of randomly initialized learnable trajectory query vectors; The learnable trajectory query vector, the current world state representation, and the future world state representation are input into the trajectory decoder to output multiple candidate trajectories.

10. The method according to claim 1, characterized in that, Selecting the optimal trajectory from the multiple candidate trajectories includes: Obtain the control command for the target vehicle at the current time step; Select the optimal trajectory that matches the control command from the multiple candidate trajectories.

11. A device for generating driving trajectories, characterized in that, include: The first acquisition module is used to acquire continuous frames collected in real time from the target vehicle, wherein each frame of the continuous frame includes a multi-view image and the vehicle status. The second acquisition module is used to acquire the compressed scene representation features of the multi-view image and the vehicle motion state features of the vehicle state. The generation module is used to generate a current world state representation using the compressed scene representation features and the vehicle motion state features; The prediction module is used to predict the future world state representation of the current world state representation based on the Dynamic Latent World Model (DLWM). The selection module is used to generate multiple candidate trajectories using the current world state representation and the future world state representation, and to select the optimal trajectory from the multiple candidate trajectories.

12. The apparatus according to claim 11, characterized in that, The second acquisition module includes: An embedding unit is used to embed the multi-view image into an image block token, wherein the feature dimensions of the image block token include: time span, number of cameras, and number of image blocks; The splicing unit is used to randomly initialize a set of scene query tokens and splice the image block tokens and the scene query tokens to obtain splicing features; An extraction unit is used to extract compressed scene representation features of the stitched features using a pre-trained unlabeled distilled visual DINO encoder, wherein the DINO encoder is used to compress visual information in the image block token into the token through a self-attention mechanism.

13. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 10 when it is run.

14. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 10.