Automatic driving based trajectory and scene generation method and device, equipment and medium

Abstract

This invention relates to the field of autonomous driving technology, providing a method, apparatus, device, and medium for generating trajectories and scenes based on autonomous driving. It utilizes visual encoders, text encoders, and physical quantity encoders to provide standardized feature representations; it concatenates image features, text features, physical features, and world prior query vectors to construct a unified feature vector encompassing scene perception, semantic intent, motion constraints, and open-world knowledge; it optimizes the generation process based on a truncation-diffusion strategy to reduce invalid noise iterations; and it improves the temporal consistency and spatial detail quality of the generated scene and trajectory based on a two-stage spatiotemporal fidelity refinement strategy; it inputs multimodal fusion features into the action model and world model respectively to obtain the future planned trajectory and future scene image. Based on unified conditional constraints, the two tasks share the same semantic and physical context, avoiding serial dependencies, eliminating the cascading effects of scene prediction errors on trajectory planning, and reducing latency.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus, device, and medium for generating trajectories and scenes based on autonomous driving. Background Technology

[0002] With the development of large language models (LLMs) and vision-language models (VLMs), end-to-end autonomous driving systems are undergoing a paradigm shift. By leveraging the extensive world knowledge and reasoning capabilities of the underlying models, autonomous driving intelligence is manifested in its ability to interpret complex driving scenarios and follow advanced user commands. Simultaneously, the emergence of generative world models enables systems to predict the evolution of future scenarios.

[0003] Despite the progress made, integrating visual language models and world models into a unified driving system with extremely high safety requirements still faces the following two fundamental limitations: First, although visual language models excel at high-level semantic reasoning, existing general-purpose visual language models lack alignment with the real physical world (Physical Grounding) and perform poorly when dealing with precise 3D spatiotemporal dynamics and kinematic constraints.

[0004] Secondly, existing strategies for integrating world models and action planning models face a serious conflict between interaction depth and inference efficiency. Feedback-based architectures require online simulation and evaluation of pre-generated trajectories, but due to sequence dependencies, serial architectures (where the planner must wait for future video frames to be fully generated before it can work) inevitably introduce severe inference latency, failing to meet the real-time requirements of autonomous driving systems.

[0005] Therefore, a unified architecture is needed that can both guarantee physical constraints and achieve low-latency, efficient reasoning, in order to realize safe end-to-end autonomous driving control and high-quality scene generation. Summary of the Invention

[0006] In view of the above, it is necessary to provide a method, device, equipment and medium for generating trajectories and scenes based on autonomous driving, in order to solve the problem of being unable to generate high-quality future trajectories and scenes for autonomous vehicles.

[0007] A method for generating trajectories and scenes based on autonomous driving, the method comprising: In response to trajectory and scene generation instructions triggered by the target autonomous vehicle, the system uses the onboard camera of the target autonomous vehicle to capture multi-view image sequences, and obtains the natural language instructions input by the user and the current physical state sequence of the target autonomous vehicle. The image features of the multi-view image sequence are extracted using a visual encoder, the text features of the natural language instructions are extracted using a text encoder, and the physical state sequence is mapped to physical features using a physical quantity encoder. Randomly initialize a set of learnable world prior query vectors; The image features, text features, physical features, and world prior query vector are concatenated to obtain the concatenated vector; The concatenated vector is input into the backbone network of the visual language model to obtain multimodal fusion features; Obtain a unified physical language model, including parallel decoupled action models and world models, constructed based on a truncation-diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy. The multimodal fusion features are input into the action model, and the future planned trajectory of the target autonomous vehicle is predicted based on the world prior query vector. The multimodal fusion features are input as conditions into the world model to obtain a future scene image of the target autonomous vehicle.

[0008] An autonomous driving-based trajectory and scene generation device, the autonomous driving-based trajectory and scene generation device comprising: The acquisition unit is used to respond to the trajectory and scene generation instructions triggered by the target autonomous vehicle, capture multi-view image sequences using the on-board camera of the target autonomous vehicle, and acquire the natural language instructions input by the user and the current physical state sequence of the target autonomous vehicle. The extraction and mapping unit is used to extract image features of the multi-view image sequence using a visual encoder, extract text features of the natural language instructions using a text encoder, and map the physical state sequence into physical features using a physical quantity encoder. An initialization unit is used to randomly initialize a set of learnable world prior query vectors. The splicing unit is used to splice the image features, the text features, the physical features and the world prior query vector to obtain a spliced ​​vector; The input unit is used to input the concatenated vector into the backbone network of the visual language model to obtain multimodal fusion features; The acquisition unit is also used to acquire a unified physical language model, including a parallel decoupled action model and a world model, constructed based on a truncation diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy. The input unit is also used to input the multimodal fusion features into the action model and predict the future planned trajectory of the target autonomous vehicle based on the world prior query vector; The input unit is also used to input the multimodal fusion features as conditions into the world model to obtain a future scene image of the target autonomous vehicle.

[0009] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the trajectory and scene generation method based on autonomous driving.

[0010] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the trajectory and scene generation method based on autonomous driving.

[0011] As can be seen from the above technical solutions, this invention can extract image features from multi-view image sequences using a visual encoder, extract text features from natural language instructions using a text encoder, and map physical state sequences to physical features using a physical quantity encoder, providing standardized feature representations. It concatenates image features, text features, physical features, and world prior query vectors to construct a unified feature vector that includes scene perception, semantic intent, motion constraints, and open-world knowledge. Based on a truncation-diffusion strategy, it can optimize the generation process and reduce invalid noise iterations. Based on a two-stage spatiotemporal fidelity refinement strategy, it can improve the temporal consistency and spatial detail quality of the generated scene and trajectory. By inputting the multimodal fusion features into the action model and the world model respectively, it obtains the future planned trajectory and the future scene image. It can use the unified multimodal fusion features as conditional constraints, allowing the two tasks to share the same semantic and physical context, avoiding information inconsistencies between different modules, avoiding serial dependencies, eliminating the chain reaction of scene prediction errors on trajectory planning, and reducing latency. Attached Figure Description

[0012] Figure 1 This is a flowchart of a preferred embodiment of the trajectory and scene generation method based on autonomous driving of the present invention; Figure 2 This is a functional block diagram of a preferred embodiment of the trajectory and scene generation device for autonomous driving of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the trajectory and scene generation method based on autonomous driving according to the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the trajectory and scene generation method for autonomous driving according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0015] The trajectory and scene generation method based on autonomous driving is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0016] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0017] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0018] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0019] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0020] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0021] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0022] S10, in response to the trajectory and scene generation command triggered by the target autonomous vehicle, capture a multi-view image sequence using the on-board camera of the target autonomous vehicle, and obtain the natural language command input by the user and the current physical state sequence of the target autonomous vehicle.

[0023] In this embodiment, the trajectory and scene generation instructions can be automatically triggered when autonomous driving is started.

[0024] In this embodiment, the natural language instruction can be a voice instruction input by the user, such as "Please describe the current scene, generate future frames, and plan future trajectories".

[0025] In this embodiment, the physical state sequence may include the vehicle state, camera intrinsic parameters, historical trajectory, and future target trajectory.

[0026] In the above embodiments, the multi-view images are used to provide scene spatial structure and visual details, the natural language instructions are used to provide driving intentions and semantic constraints (such as "turn left" and "go straight"), and the physical state sequence is used to provide vehicle kinematic information.

[0027] By collecting multidimensional data, we can provide complete and complementary raw data for subsequent multimodal fusion, ensuring that the model has visual, semantic and physical perception capabilities at the same time.

[0028] S11, use a visual encoder to extract image features from the multi-view image sequence, use a text encoder to extract text features from the natural language instructions, and use a physical quantity encoder to map the physical state sequence into physical features.

[0029] In this embodiment, the visual encoder and the text encoder can be encoders with a general structure.

[0030] In this embodiment, mapping the physical state sequence to physical features using a physical quantity encoder includes: Each physical state in the physical state sequence is normalized according to its corresponding mode to obtain a normalized sequence; Learnable position codes are added to the historical trajectory and future target trajectory in the normalized sequence to obtain an intermediate sequence; The intermediate sequence is mapped to a latent dimension consistent with the visual language model using a multi-layer perceptron (MLP) to obtain the physical features.

[0031] Since the numerical scale of each physical state differs greatly, it is necessary to first normalize it according to the corresponding mode.

[0032] The learnable positional encoding is used to capture temporal dynamic information.

[0033] The physical characteristics can be represented as follows: E phy =MLP([S ego C calib , τ hist +E pos ,IWM⋅(τ fut +E pos )]); Among them, E phy Indicates the physical characteristics; S ego Indicates the vehicle's status; C calib Indicates camera intrinsic parameters; τ hist Indicates historical trajectory; τ fu Indicates the future target trajectory; E pos represents the learnable positional encoding; IWM represents the indicator function, which takes the value 1 during world model training and 0 during action inference.

[0034] In the above embodiments, the visual encoder is used to retain scene spatial information and dynamic target features, the text encoder is used to convert driving instructions into computable semantic vectors, and the physical quantity encoder is used to encode the vehicle's motion state into dynamic constraint features, thereby providing a standardized feature representation for subsequent cross-modal fusion and avoiding fusion difficulties caused by modal heterogeneity.

[0035] This embodiment introduces a physical quantity encoder to explicitly project the vehicle's own state and trajectory constraints into the embedding space of the large language model, so that the planning results generated by the visual language model are firmly based on physical reality, thus solving the problem that the general model lacks a physical foundation.

[0036] S12, randomly initialize a set of learnable world prior query vectors.

[0037] In this embodiment, the world prior query vector has a dual function.

[0038] Specifically, during world modeling, the world prior query vector is aggregated as a potential placeholder for spatiotemporal dynamics and serves as prior knowledge for the action model.

[0039] In the above embodiments, the world prior query vector can serve as an anchor point for open-world knowledge, used to assist in learning prior knowledge of general driving scenarios (such as road structure, traffic rules, and vehicle motion patterns).

[0040] S13, concatenate the image features, the text features, the physical features, and the world prior query vector to obtain the concatenated vector.

[0041] In this embodiment, by concatenating the image features, text features, physical features, and world prior query vector, a unified feature vector containing scene awareness, semantic intent, motion constraints, and open-world knowledge is constructed. This can provide global semantic and common-sense constraints for subsequent fusion, solve the problem of insufficient generalization ability of pure data-driven models for rare scenarios, and improve the rationality of subsequent generated results.

[0042] S14, the concatenated vector is input into the backbone network of the visual language model to obtain multimodal fusion features.

[0043] In this embodiment, the Vision-Language Model (VLM) includes a multimodal large model with the parameters of the visual encoder, the text encoder, the large language model layer, and the language modeling head (LM Head) frozen.

[0044] In this embodiment, after the spliced ​​vector is input into the backbone network of the visual language model, images, text, physical features and prior knowledge of the world can interact to form a unified multimodal fusion feature.

[0045] In this embodiment, the visual language model can be InternVL3 (Intern Vision Language3, Shusheng·Wanxiang 3.0, third-generation multimodal large model).

[0046] The decoupled design and frozen language modeling head in this embodiment enable the model to maintain its original language interaction advantages, continue to perform visual question answering (VQA), and output natural language explanations of driving decisions, thereby improving training efficiency while ensuring the interpretability of the end-to-end system.

[0047] S15, obtain the Unified Physical Language Model (UniPLM) constructed based on the truncation diffusion strategy and the two-stage spatiotemporal fidelity refinement strategy, which includes the parallel decoupled Action Model DiT and World Model DiT.

[0048] In this embodiment, the action model is constructed based on the truncated diffusion strategy, and the action model includes multiple cascaded diffusion transformer (DiT) layers.

[0049] In this embodiment, before obtaining the unified physical language model, which includes a parallel decoupled action model and a world model, constructed based on a truncation-diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy, the method further includes: Construct a training set that includes real driving trajectories; Configure a set of learnable anchor points, and obtain the optimal anchor point that is closest to the actual driving trajectory from the learnable anchor points; Construct a focus classification loss to handle class imbalance, and an L1 loss only for optimal anchor trajectory regression; The focal classification loss and the L1 loss are weighted according to the classification loss balance coefficient and the regression loss balance coefficient to obtain the total loss function; The action model is obtained by training based on the total loss function and the training set.

[0050] The distance can be a Euclidean distance.

[0051] The learnable anchor points can be a set of independent learnable parameters that are predefined during the model initialization phase and participate in the entire training process, and belong to the built-in independent parameters of the action model.

[0052] The total loss function can be expressed as follows: L Action =λ cls Focal(s,y k )+λ reg ∣∣τ k -τ gt ||1; Among them, L Action λ represents the total loss function value. cls λ represents the classification loss balance coefficient. reg y represents the regression loss balance coefficient; s represents the confidence score of all anchor point trajectories predicted by the model; kFocal(s,y) represents the one-hot encoded target vector corresponding to the optimal anchor point. k ) represents s, y k Between-focus classification loss; τ k τ represents the regression trajectory predicted by the optimal anchor branch. gt This indicates the actual driving trajectory.

[0053] In the above embodiments, a "winner-takes-all" selective regression strategy is adopted, which allows non-target branches to explore alternative behaviors without being penalized, thereby preventing pattern collapse.

[0054] In this embodiment, the world model is constructed based on the two-stage spatiotemporal fidelity refinement strategy.

[0055] The world model can be a Conditional DiT, used for generating future scenes under physical constraints based on flow matching. Its optimization objective is to minimize the matching error of the velocity prediction network in converting noise into data distribution.

[0056] The world model may include an input layer, a DiT backbone network, an ODE (Ordinary Differential Equation) integral inference layer, and a post-processing output layer. The DiT backbone network includes a velocity prediction network, used to describe the velocity field of latent space features evolving from noise to the real image.

[0057] In this embodiment, before obtaining the unified physical language model, which includes a parallel decoupled action model and a world model, constructed based on a truncation-diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy, the method further includes: During the training of the world model, a normalized value of the difference between the latent representations of adjacent real frames is calculated in the latent space to obtain the motion mask; Based on the motion mask, when calculating the loss between the predicted velocity field and the target velocity field, higher loss weights are assigned to the labels of high-motion regions to force the network to prioritize ensuring the vector field fidelity of moving objects.

[0058] The motion mask is used to reflect the degree of dynamic change in the region.

[0059] The loss function used can be expressed as follows: ; in, This represents the loss value of the world model. Expressing expectations; z represents the latent representation of a clean target image in stream matching.t The latent representation at time step t is given by z0 and Linear interpolation constitutes, , representing Gaussian noise sampled from a standard normal distribution; This indicates that a speed prediction network needs to be trained; This indicates the condition signal output after conversion by the adapter from the VLM.

[0060] Through the above embodiments, the velocity loss can be dynamically perceived based on a two-stage spatiotemporal fidelity refinement strategy, thus solving the problems of high-speed dynamic target ambiguity and long-term static structure degradation.

[0061] This embodiment adopts a decoupled architecture of world model and action model, and introduces a world prior injection mechanism.

[0062] S16, input the multimodal fusion features into the action model, and predict the future planned trajectory of the target autonomous vehicle based on the world prior query vector.

[0063] In this embodiment, the step of inputting the multimodal fusion features into the action model and predicting the future planned trajectory of the target autonomous vehicle based on the world prior query vector includes: Obtain real-world trajectories (Ground Truth, GT) in multiple driving modes; Each real trajectory is clustered using a clustering algorithm to obtain each fixed anchor point; Each fixed anchor point undergoes a noise-adding process with fewer than a preset number of steps, and a noisy trajectory is constructed based on each fixed anchor point after the noise addition. In each diffusion converter layer, each waypoint in the noisy trajectory is projected onto a two-dimensional image plane using the extrinsic and intrinsic parameter matrices of the vehicle-mounted camera to obtain each two-dimensional projected coordinate. At each two-dimensional projection coordinate, a deformable cross-attention mechanism is used to sample and interact with the multimodal fusion features and the noisy trajectory to obtain visual context features; Cross-attention calculation is performed on the visual context features and the world prior query vector to obtain world prior enhanced features that incorporate spatiotemporal dynamic priors. Cross-attention calculation is performed on the world prior enhancement features and the physical features to obtain kinematically aligned planning features; The future planning trajectory is obtained by processing the planning features obtained from each diffusion transformer layer using a multilayer perceptron.

[0064] The clustering algorithm can be a K-means (K-means Clustering) clustering algorithm.

[0065] Among them, 32 anchor points representing different potential driving modes can be used, and after a few steps of noise addition, they can be used as noisy trajectories.

[0066] Since VLM lacks explicit bird's-eye view (BEV) features, in order to bridge the domain gap between the 3D trajectory and the 2D image, each waypoint in the noisy trajectory is projected onto the 2D image plane using camera extrinsic and intrinsic parameters. Specifically, the projection can be performed using the following formula: ; in, This represents the projected coordinates of the 3D waypoint on the v-th camera view plane onto the 2D image plane. The intrinsic parameter matrix represents the view plane of the v-th camera; This represents the extrinsic parameter matrix of the v-th camera view plane; Represents the coordinates of each physical waypoint in a noisy 3D trajectory; This indicates a projection operation.

[0067] Among them, sampling interaction processing is performed based on Vision-Language Model-Guided Deformable Cross-Attention (VLM-Guided DCA), and the specific formula can be expressed as follows: H vis =DCA(MLP(τ noise ), ,F vlm ); Among them, H vis The extracted and constructed visual context features are represented by DCA; DCA represents Deformable Cross-Attention; MLP represents Multilayer Perceptron Mapping Network; τ noise Represents the noisy trajectory; F vlm This represents the multimodal fusion feature.

[0068] Specifically, by performing cross-attention calculation on the visual context features and the world prior query vector, the planner can predict future dangers without waiting for image generation. The world prior enhancement features are refined features that integrate spatiotemporal dynamic priors, and the world prior query vector is used to aggregate the spatiotemporal scene dynamic evolution features learned during the pre-training phase.

[0069] Specifically, by performing cross-attention calculations on the world prior enhancement features and the physical features, kinematic feasibility can be enforced. The physical features are used as inputs to the cross-attention layer for physical alignment, encoding the vehicle's current speed, steering, and other dynamic states, as well as kinematic constraints. This acts as a dynamic gating mechanism, ensuring that the final generated planned trajectory kinematically conforms to the vehicle's actual control limitations.

[0070] In the above embodiments, the noisy anchor points are directly used as input action models for noisy trajectories. Through the cross-attention mechanism, the model continuously queries and absorbs multimodal contextual features (such as visual features, physical alignment features, and world priors) for guidance to perform iterative denoising and fine-tuning, and finally selects and optimizes a safe and optimal planning trajectory that conforms to the current scenario.

[0071] This embodiment aggregates dynamic scene change information through world prior query vectors and injects it as a direct prior into the action model. This allows the planner to efficiently "imagine" future dangers without waiting for pixel-level frame generation, completely solving the severe inference latency problem caused by traditional serial architectures and greatly improving real-time performance. Furthermore, a hierarchical multimodal interaction mechanism (visually guided deformable cross-attention, world prior injection, and physical query alignment) is designed in the action model. Through modular design, catastrophic forgetting of the basic model is avoided. While achieving SOTA (State Of The Art) level closed-loop planning performance, it perfectly preserves the original powerful scene reasoning and question-answering capabilities of the visual language large model.

[0072] S17, the multimodal fusion features are input as conditions into the world model to obtain the future scene image of the target autonomous vehicle.

[0073] In this embodiment, the step of inputting the multimodal fusion features as conditions into the world model to obtain the future scene image of the target autonomous vehicle includes: The multimodal fusion features are mapped to conditional feature vectors using an adapter, and the conditional feature vectors are input into the diffusion transformer of the world model as global semantic constraints. In the ordinary differential equation integration loop at each time step, a clear latent representation of the current frame is analytically estimated under the global semantic constraints; The clear latent representation is filtered using a discrete Laplace kernel, and a mask of high-frequency changing regions, including structural edges, is extracted. Based on the time decay factor and the high-frequency changing region mask, local random noise is injected into the deterministic ordinary differential equation path to dynamically regenerate edge details and maintain the temporal consistency of the background, thereby obtaining the future scene image.

[0074] The potential representation of each time step in the future scene image is as follows: ; Where z′ represents the latent representation of the future scene image after targeted perturbation at each time step; γ t The time decay factor is a coefficient used to control the decay of the disturbance intensity over time; M freq ⊙ represents the high-frequency variation region mask calculated and thresholded by the discrete Laplace kernel, often corresponding to fragile structural edges; ⊙ represents the Hadamard product, i.e., element-wise multiplication.

[0075] Through the above embodiments, it is possible to adapt to potential frequency perturbations, estimate the current clear potential representation in the integral cycle of ordinary differential equations, and use discrete Laplace kernels to separate high-frequency variation region masks, and inject local random noise into the region, forcing the network to dynamically regenerate clear details without destroying the consistency of the background.

[0076] This embodiment proposes a two-stage spatiotemporal fidelity refinement strategy in the world model. During the training phase, a dynamically perceived velocity loss mechanism prioritizes moving objects to prevent them from being obscured by the static background. During the inference phase, an adaptive latent frequency perturbation mechanism is introduced to mitigate oversmoothing caused by accumulated autoregressive errors. This strategy significantly improves the clarity and fidelity of future frame generation while ensuring planning feasibility.

[0077] In this embodiment, S10-S17 can be repeatedly executed to achieve end-to-end closed-loop control of the autonomous vehicle. The planned trajectory generated by this embodiment has extremely high safety and comfort, and reaches a leading state-of-the-art level on the NAVSIM (Data-Driven Non-Reactive Autonomous Vehicle Simulation) closed-loop evaluation benchmark.

[0078] In the Navsim Benchmark benchmark, the evaluation is scene-by-scene, and each time the trajectory can be predicted for the next 4 seconds, and the prediction is repeated until the current scene is evaluated.

[0079] If it is a real vehicle deployment, the decision to end can be made based on whether the target point has been reached, whether a collision has occurred, or whether an emergency has been encountered.

[0080] In this embodiment, the method further includes: The multimodal fusion features are input into the language modeling head to obtain natural language text that describes the current driving scenario and explains planning decisions.

[0081] By freezing the parameters of the visual encoder, the text encoder, the large language model layer, and the language modeling head, catastrophic forgetting can be avoided, allowing the model to fully retain the inherent open-world understanding and powerful scene reasoning capabilities of the basic model while adapting to the needs of autonomous driving through lightweight task-specific modules.

[0082] In the above embodiments, since the specific task module is decoupled from the backbone network through the adapter, the original language capabilities of the VLM are retained, and the language modeling head can be used to output explanatory text for driving scenario description, visual question answering, and planning decisions.

[0083] As can be seen from the above technical solutions, this invention can extract image features from multi-view image sequences using a visual encoder, extract text features from natural language instructions using a text encoder, and map physical state sequences to physical features using a physical quantity encoder, providing standardized feature representations. It concatenates image features, text features, physical features, and world prior query vectors to construct a unified feature vector that includes scene perception, semantic intent, motion constraints, and open-world knowledge. Based on a truncation-diffusion strategy, it can optimize the generation process and reduce invalid noise iterations. Based on a two-stage spatiotemporal fidelity refinement strategy, it can improve the temporal consistency and spatial detail quality of the generated scene and trajectory. By inputting the multimodal fusion features into the action model and the world model respectively, it obtains the future planned trajectory and the future scene image. It can use the unified multimodal fusion features as conditional constraints, allowing the two tasks to share the same semantic and physical context, avoiding information inconsistencies between different modules, avoiding serial dependencies, eliminating the chain reaction of scene prediction errors on trajectory planning, and reducing latency.

[0084] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the trajectory and scene generation device based on autonomous driving according to the present invention. The trajectory and scene generation device 11 based on autonomous driving includes an acquisition unit 110, an extraction and mapping unit 111, an initialization unit 112, a splicing unit 113, and an input unit 114. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0085] The acquisition unit 110 is used to respond to the trajectory and scene generation instructions triggered by the target autonomous vehicle, capture multi-view image sequences using the on-board camera of the target autonomous vehicle, and acquire the natural language instructions input by the user and the current physical state sequence of the target autonomous vehicle. The extraction and mapping unit 111 is used to extract image features of the multi-view image sequence using a visual encoder, extract text features of the natural language instructions using a text encoder, and map the physical state sequence to physical features using a physical quantity encoder. The initialization unit 112 is used to randomly initialize a set of learnable world prior query vectors. The splicing unit 113 is used to splice the image features, the text features, the physical features and the world prior query vector to obtain a spliced ​​vector; The input unit 114 is used to input the concatenated vector into the backbone network of the visual language model to obtain multimodal fusion features; The acquisition unit 110 is also used to acquire a unified physical language model, including a parallel decoupled action model and a world model, constructed based on a truncation diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy. The input unit 114 is also used to input the multimodal fusion features into the action model and predict the future planned trajectory of the target autonomous vehicle based on the world prior query vector. The input unit 114 is further configured to input the multimodal fusion features as conditions into the world model to obtain a future scene image of the target autonomous vehicle.

[0086] As can be seen from the above technical solutions, this invention can extract image features from multi-view image sequences using a visual encoder, extract text features from natural language instructions using a text encoder, and map physical state sequences to physical features using a physical quantity encoder, providing standardized feature representations. It concatenates image features, text features, physical features, and world prior query vectors to construct a unified feature vector that includes scene perception, semantic intent, motion constraints, and open-world knowledge. Based on a truncation-diffusion strategy, it can optimize the generation process and reduce invalid noise iterations. Based on a two-stage spatiotemporal fidelity refinement strategy, it can improve the temporal consistency and spatial detail quality of the generated scene and trajectory. By inputting the multimodal fusion features into the action model and the world model respectively, it obtains the future planned trajectory and the future scene image. It can use the unified multimodal fusion features as conditional constraints, allowing the two tasks to share the same semantic and physical context, avoiding information inconsistencies between different modules, avoiding serial dependencies, eliminating the chain reaction of scene prediction errors on trajectory planning, and reducing latency.

[0087] like Figure 3 The diagram shown is a structural schematic of a computer device that implements a preferred embodiment of the trajectory and scene generation method based on autonomous driving according to the present invention.

[0088] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a trajectory and scene generation program based on autonomous driving.

[0089] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0090] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0091] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as code for autonomous driving trajectory and scene generation programs, but also to temporarily store data that has been output or will be output.

[0092] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing trajectory and scene generation programs based on autonomous driving), and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0093] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various embodiments of the autonomous driving-based trajectory and scene generation method described above, for example... Figure 1 The steps are shown.

[0094] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, an extraction mapping unit 111, an initialization unit 112, a splicing unit 113, and an input unit 114.

[0095] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the trajectory and scene generation method based on autonomous driving described in various embodiments of the present invention.

[0096] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0097] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0098] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0099] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0100] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0101] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0102] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.

[0103] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0104] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0105] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0106] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a trajectory and scene generation method based on autonomous driving, and the processor 13 can execute the multiple instructions to achieve the following: In response to trajectory and scene generation instructions triggered by the target autonomous vehicle, the system uses the onboard camera of the target autonomous vehicle to capture multi-view image sequences, and obtains the natural language instructions input by the user and the current physical state sequence of the target autonomous vehicle. The image features of the multi-view image sequence are extracted using a visual encoder, the text features of the natural language instructions are extracted using a text encoder, and the physical state sequence is mapped to physical features using a physical quantity encoder. Randomly initialize a set of learnable world prior query vectors; The image features, text features, physical features, and world prior query vector are concatenated to obtain the concatenated vector; The concatenated vector is input into the backbone network of the visual language model to obtain multimodal fusion features; Obtain a unified physical language model, including parallel decoupled action models and world models, constructed based on a truncation-diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy. The multimodal fusion features are input into the action model, and the future planned trajectory of the target autonomous vehicle is predicted based on the world prior query vector. The multimodal fusion features are input as conditions into the world model to obtain a future scene image of the target autonomous vehicle.

[0107] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0108] It should be noted that all the data involved in this case was legally obtained.

[0109] If any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention has been obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0110] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0111] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0112] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0114] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0115] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0116] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating trajectory and scene based on autonomous driving, characterized in that, The trajectory and scene generation method based on autonomous driving includes: In response to trajectory and scene generation instructions triggered by the target autonomous vehicle, the system uses the onboard camera of the target autonomous vehicle to capture multi-view image sequences, and obtains the natural language instructions input by the user and the current physical state sequence of the target autonomous vehicle. The image features of the multi-view image sequence are extracted using a visual encoder, the text features of the natural language instructions are extracted using a text encoder, and the physical state sequence is mapped to physical features using a physical quantity encoder. Randomly initialize a set of learnable world prior query vectors; The image features, text features, physical features, and world prior query vector are concatenated to obtain the concatenated vector; The concatenated vector is input into the backbone network of the visual language model to obtain multimodal fusion features; Obtain a unified physical language model, including parallel decoupled action models and world models, constructed based on a truncation-diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy. The multimodal fusion features are input into the action model, and the future planned trajectory of the target autonomous vehicle is predicted based on the world prior query vector. The multimodal fusion features are used as conditions to input into the world model to obtain a future scene image of the target autonomous vehicle; The action model is constructed based on the truncated diffusion strategy and includes multiple cascaded diffusion transformer layers. Before obtaining the unified physical language model, which includes a parallel decoupled action model and a world model, constructed based on the truncated diffusion strategy and a two-stage spatiotemporal fidelity refinement strategy, the method further includes: constructing a training set including real driving trajectories; configuring a set of learnable anchors and obtaining the optimal anchor closest to the real driving trajectory from the learnable anchors; constructing a focus classification loss to handle class imbalance and an L1 loss for regression only on the optimal anchor trajectory; weighting the focus classification loss and the L1 loss according to the classification loss balance coefficient and the regression loss balance coefficient to obtain a total loss function; and training the action model based on the total loss function and the training set. The world model is constructed based on the two-stage spatiotemporal fidelity refinement strategy. Before obtaining the unified physical language model, which includes the parallel decoupled action model and world model, constructed based on the truncation-diffusion strategy and the two-stage spatiotemporal fidelity refinement strategy, the method further includes: during the training of the world model, calculating the normalized value of the difference between the latent representations of adjacent real frames in the latent space to obtain a motion mask; based on the motion mask, when calculating the loss between the predicted velocity field and the target velocity field, assigning higher loss weights to the labels of high-motion regions to force the network to prioritize ensuring the vector field fidelity of moving objects.

2. The trajectory and scene generation method based on autonomous driving as described in claim 1, characterized in that, The step of mapping the physical state sequence into physical features using a physical quantity encoder includes: Each physical state in the physical state sequence is normalized according to its corresponding mode to obtain a normalized sequence; Learnable position codes are added to the historical trajectory and future target trajectory in the normalized sequence to obtain an intermediate sequence; The intermediate sequence is mapped to a latent dimension consistent with the visual language model using a multilayer perceptron to obtain the physical features.

3. The trajectory and scene generation method based on autonomous driving as described in claim 1, characterized in that, The visual language model includes a multimodal large model with parameters of the visual encoder, the text encoder, the large language model layer, and the language modeling head frozen.

4. The trajectory and scene generation method based on autonomous driving as described in claim 1, characterized in that, The step of inputting the multimodal fusion features into the action model and predicting the future planned trajectory of the target autonomous vehicle based on the world prior query vector includes: Acquire real-world trajectories under multiple driving modes; Each real trajectory is clustered using a clustering algorithm to obtain each fixed anchor point; Each fixed anchor point undergoes a noise-adding process with fewer than a preset number of steps, and a noisy trajectory is constructed based on each fixed anchor point after the noise addition. In each diffusion converter layer, each waypoint in the noisy trajectory is projected onto a two-dimensional image plane using the extrinsic and intrinsic parameter matrices of the vehicle-mounted camera to obtain each two-dimensional projected coordinate. At each two-dimensional projection coordinate, a deformable cross-attention mechanism is used to sample and interact with the multimodal fusion features and the noisy trajectory to obtain visual context features; Cross-attention calculation is performed on the visual context features and the world prior query vector to obtain world prior enhanced features that incorporate spatiotemporal dynamic priors. Cross-attention calculation is performed on the world prior enhancement features and the physical features to obtain kinematically aligned planning features; The future planning trajectory is obtained by processing the planning features obtained from each diffusion transformer layer using a multilayer perceptron.

5. The trajectory and scene generation method based on autonomous driving as described in claim 1, characterized in that, The step of inputting the multimodal fusion features as conditions into the world model to obtain the future scene image of the target autonomous vehicle includes: The multimodal fusion features are mapped to conditional feature vectors using an adapter, and the conditional feature vectors are input into the diffusion transformer of the world model as global semantic constraints. In the ordinary differential equation integration loop at each time step, a clear latent representation of the current frame is analytically estimated under the global semantic constraints; The clear latent representation is filtered using a discrete Laplace kernel, and a mask of high-frequency changing regions, including structural edges, is extracted. Based on the time decay factor and the high-frequency changing region mask, local random noise is injected into the deterministic ordinary differential equation path to dynamically regenerate edge details and maintain the temporal consistency of the background, thereby obtaining the future scene image.

6. The trajectory and scene generation method based on autonomous driving as described in claim 3, characterized in that, The method further includes: The multimodal fusion features are input into the language modeling head to obtain natural language text that describes the current driving scenario and explains planning decisions.

7. A trajectory and scene generation device based on autonomous driving, characterized in that, Used to perform the trajectory and scene generation method based on autonomous driving as described in any one of claims 1 to 6.

8. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the trajectory and scene generation method based on autonomous driving as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the trajectory and scene generation method based on autonomous driving as described in any one of claims 1 to 6.

Images