Trajectory planning method and electronic device

Abstract

The application discloses a trajectory planning method and an electronic device, and relates to the technical field of vehicles. Visual perception data, query text and historical trajectory data of a vehicle are input into a pre-trained visual language model. A perception feature vector is output through the visual language model. The visual language model comprises a visual encoder, a text encoder, a trajectory encoder and a pre-trained language model. The visual perception data is encoded into a visual token sequence, the query text is encoded into a text token sequence, and the historical trajectory data is encoded into a trajectory token sequence. The visual token sequence, the text token sequence and the trajectory token sequence are input into the pre-trained language model, and the pre-trained language model outputs the perception feature vector. The historical trajectory data is encoded into a trajectory feature vector, and vehicle state data is encoded into a state feature vector. A target planning trajectory is generated according to the perception feature vector, the trajectory feature vector and the state feature vector. The technical problem of insufficient understanding of the environment and poor robustness of trajectory planning in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to trajectory planning methods and electronic devices. Background Technology

[0002] With the rapid development of intelligent driving technology, vehicle driving is gradually moving towards the integration of cognition and decision-making. Complex urban road scenarios place higher demands on vehicle environmental perception, semantic understanding, and trajectory planning capabilities. Therefore, multimodal fusion cognitive modeling is a key technology direction for improving the adaptability and planning accuracy of intelligent driving scenarios.

[0003] In related technologies, trajectory planning often adopts a modular architecture, which separates perception, semantic understanding and trajectory planning for independent execution. It processes various types of data such as vision, instructions and trajectory separately. However, the modal information is fragmented, resulting in insufficient perception of the environment and semantic understanding during the trajectory planning process. At the same time, the phased modular processing is prone to error accumulation, resulting in poor robustness of trajectory planning. Summary of the Invention

[0004] This application provides a trajectory planning method and electronic device to at least solve the problems in related technologies such as fragmented multimodal information, insufficient perception of the environment and understanding of semantics during trajectory planning due to phased modular processing, and poor robustness of trajectory planning.

[0005] This application provides a trajectory planning method, comprising the following steps: acquiring vehicle visual perception data, query text, historical trajectory data, and vehicle state data; inputting the visual perception data, query text, and historical trajectory data into a pre-trained visual language model, and outputting a target-dimensional perception feature vector through the visual language model, wherein the visual language model includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model; encoding the visual perception data into a target-dimensional visual word sequence through the visual encoder, encoding the query text into a target-dimensional text word sequence through the text encoder, and encoding the historical trajectory data into a target-dimensional trajectory word sequence through the trajectory encoder; inputting the visual word sequence, text word sequence, and trajectory word sequence into the pre-trained language model, and outputting a target-dimensional perception feature vector through the pre-trained language model; encoding the historical trajectory data into a target-dimensional trajectory feature vector, encoding the vehicle state data into a target-dimensional state feature vector, and generating the vehicle's target planned trajectory based on the perception feature vector, trajectory feature vector, and state feature vector.

[0006] This application also provides a trajectory planning device, comprising: an acquisition module for acquiring visual perception data, query text, historical trajectory data, and vehicle state data of a vehicle; an input module for inputting the visual perception data, query text, and historical trajectory data into a pre-trained visual language model, and outputting a target-dimensional perception feature vector through the visual language model, wherein the visual language model includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model, wherein the visual encoder encodes the visual perception data into a target-dimensional visual word sequence, the text encoder encodes the query text into a target-dimensional text word sequence, the trajectory encoder encodes the historical trajectory data into a target-dimensional trajectory word sequence, and the visual word sequence, text word sequence, and trajectory word sequence are input into the pre-trained language model, and the pre-trained language model outputs a target-dimensional perception feature vector; and a generation module for encoding the historical trajectory data into a target-dimensional trajectory feature vector, encoding the vehicle state data into a target-dimensional state feature vector, and generating a target planned trajectory for the vehicle based on the perception feature vector, trajectory feature vector, and state feature vector.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the trajectory planning methods described above.

[0008] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described trajectory planning methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described trajectory planning methods.

[0010] Since related technologies mainly use modular architectures for trajectory planning, the fragmentation of multimodal information leads to insufficient environmental perception and semantic understanding during the trajectory planning process. Furthermore, the phased modular processing is prone to error accumulation, resulting in poor robustness of trajectory planning. This application's embodiment can acquire vehicle visual perception data, query text, historical trajectory data, and vehicle status data. The visual perception data, query text, and historical trajectory data are input into a pre-trained visual language model. The visual language model performs multimodal joint modeling of vision, text, and trajectory within a unified representation space, outputting a perceptual feature vector of the target dimension. This significantly improves the semantic understanding and environmental perception capabilities of the visual language model, while simultaneously integrating historical trajectory data. The trajectory data is encoded into a trajectory feature vector in the target dimension, and the vehicle state data is encoded into a state feature vector in the target dimension. Then, the target planned trajectory of the vehicle is generated based on the perception feature vector, trajectory feature vector, and state feature vector. This achieves end-to-end integration of perception, semantics, and planning, avoids error accumulation caused by staged calls, and improves the robustness and security of trajectory planning. Therefore, it can solve the technical problems in related technologies such as the fragmentation of multimodal information and the insufficient ability to perceive the environment and understand semantics during trajectory planning due to staged modular processing, as well as the poor robustness of trajectory planning. It achieves the technical effect of improving the semantic understanding and environmental perception capabilities of visual language models and improving the robustness of trajectory planning. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a hardware architecture diagram of the trajectory planning system provided in the embodiments of this application; Figure 2 A flowchart of the trajectory planning method provided in the embodiments of this application; Figure 3 A schematic diagram of the encoding of the visual language model provided in the embodiments of this application; Figure 4 A schematic diagram illustrating the training of a visual language model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the trajectory decoder provided in an embodiment of this application; Figure 6 This is a schematic diagram of network sampling cross-attention calculation provided in an embodiment of this application; Figure 7 A diagram illustrating the implementation process of the trajectory planning method provided in this application embodiment; Figure 8 The execution flowchart of the trajectory planning method provided in the embodiments of this application is shown below; Figure 9 A schematic diagram of the trajectory planning device provided in the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Before describing the trajectory planning method of this application, the hardware architecture on which the trajectory planning method of this application depends will be described. The trajectory planning method of this application embodiment can be deployed on an automotive-grade central computing platform and supports the collaboration between cloud training clusters and road test edge servers. The overall hardware structure is as follows: Figure 1 As shown, it includes: I. Vehicle-mounted hardware platform.

[0017] 1. Sensor group.

[0018] Visual sensors: 6-10 surround-view cameras, 1-3 forward-looking remote cameras, all cameras support hardware-triggered synchronization.

[0019] Millimeter-wave radar: 5 units (including forward, backward, and angle radars), accessed via CANFD (Controller Area Network Flexible Data-Rate).

[0020] Inertial Measurement Unit: 6-axis or 9-axis, accessed via SPI (Serial Peripheral Interface).

[0021] High-precision positioning module: GPS-RTK (Global Positioning System - Real Time Kinematic) / PPP (Precise Point Positioning), accessed via UART (Universal Asynchronous Receiver / Transmitter).

[0022] Wheel speed sensors: one per wheel, read via CAN bus.

[0023] 2. CCU (Central Control Unit) - Heterogeneous multi-chip architecture.

[0024] Main AI processor: integrates GPU (Graphics Processing Unit) / NPU (Neural Processing Unit) tensor cores, responsible for VLM (Vision Language Model) inference, trajectory generation model denoising, trajectory pattern selector scoring, and diffusion world model inference (optional).

[0025] Real-time control processor (MCU): runs a real-time operating system and is responsible for multi-source sensor fusion, hard real-time trajectory dynamics checks, safety degradation logic, and wire control command issuance.

[0026] The system service CPU (Central Processing Unit) runs Linux (an open-source operating system) / QNX (an in-vehicle real-time operating system) and is responsible for sensor data acquisition, time synchronization, text segmentation, OTA (Over-The-Air) management, and cloud communication.

[0027] Memory and storage: Used to store model weights, prior trajectory library, circular buffer, and experience replay cache.

[0028] Communication interfaces: CANFD, automotive Ethernet, MIPI CSI-2 (Mobile Industry Processor Interface Camera Serial Interface-2), GMSL (Gigabit Multimedia Serial Link), SPI, UART, PCIe (Peripheral Component Interconnect Express).

[0029] 3. Drive-by-wire chassis execution layer.

[0030] The electronic power steering, electronic stability control, brake-by-wire, and electric drive units receive control commands via CAN FD, and the closed-loop feedback is returned via CAN.

[0031] II. Cloud-based training cluster.

[0032] GPU server: 8-16 professional-grade computing graphics cards, with high-speed interconnection via InfiniBand (unlimited bandwidth high-speed interconnect technology).

[0033] Storage: Large-capacity solid-state drives are used to store driving datasets, expert trajectories, and training logs.

[0034] Simulation Cluster: Supports CUDA (Compute Unified Device Architecture) accelerated driving simulation environment, running hundreds of environment instances in parallel for reinforcement learning sampling.

[0035] III. Roadside Edge Server.

[0036] AI computing unit: used to share the inference of the vehicle-side world model.

[0037] V2X communication module: C-V2X (Cellular Vehicle-to-Everything), PC5 (vehicle-to-everything communication interface) direct connection or 5G module, low latency exchange of features and prediction results with vehicle.

[0038] Time synchronization module: used to align with vehicle-side data.

[0039] In short, the vehicle-mounted central computing platform of this application adopts a heterogeneous multi-chip design: the main AI processor is responsible for all neural network calculations; the real-time MCU is responsible for state fusion and trajectory safety hard checks; and the system CPU is responsible for data acquisition, communication, and OTA. The perception layer is accessed through multiple interfaces, drive-by-wire commands are issued via CAN FD, large-scale training and simulation are completed in the cloud, and the roadside edge server can optionally provide diffusion world model inference acceleration to achieve real-time, safe, and evolvable end-to-end trajectory planning from perception to control.

[0040] The embodiments of this application provide a trajectory planning method, and the method is described in detail in conjunction with the execution flow of the trajectory planning method.

[0041] Figure 2 This is a flowchart of a trajectory planning method provided according to an embodiment of this application.

[0042] like Figure 2 As shown, the trajectory planning method includes steps S101 to S103.

[0043] In step S101, the vehicle's visual perception data, query text, historical trajectory data, and vehicle status data are acquired.

[0044] Among them, visual perception data can be collected by multiple cameras or webcams on the vehicle and can be images; query text can be text commands output by the driver, including navigation commands, scene descriptions, etc.; historical trajectory data is a sequence of driving path points of the vehicle over a period of time; vehicle status data can include dynamic information such as vehicle position, heading angle, longitudinal speed, longitudinal acceleration, yaw rate, and curvature.

[0045] In step S102, visual perception data, query text, and historical trajectory data are input into a pre-trained visual language model. The visual language model outputs a perceptual feature vector of the target dimension. The visual language model includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model. The visual encoder encodes the visual perception data into a visual word sequence of the target dimension. The text encoder encodes the query text into a text word sequence of the target dimension. The trajectory encoder encodes the historical trajectory data into a trajectory word sequence of the target dimension. The visual word sequence, text word sequence, and trajectory word sequence are input into the pre-trained language model, and the pre-trained language model outputs a perceptual feature vector of the target dimension.

[0046] Among them, the pre-trained VLM is a multimodal large model that can understand multimodal data and use natural language to answer questions, describe and reason. It is pre-trained on a specific domain on an urban road driving dataset. The pre-trained language model can be an LLM (Large Language Model). The target dimension can be set according to the actual situation, and there is no specific limitation. The perceptual feature vector includes language features, visual features and action features.

[0047] The visual language model in this application includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the encoding of a visual language model, where, The visual encoder encodes visual perception data into a sequence of visual lexical units for the target dimension; The text encoder encodes the query text into a sequence of text terms for the target dimension; The trajectory encoder encodes historical trajectories into a sequence of trajectory terms in the target dimension; Visual lexical sequences, textual lexical sequences, and trajectory lexical sequences are input into a pre-trained language model, which then outputs a perceptual feature vector for the target dimension.

[0048] It is understood that, in the embodiments of this application, visual perception data, query text, and historical trajectory data can be input into a pre-trained visual language model, and the visual language model can output a perceptual feature vector of the target dimension. The visual language model includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model. The visual encoder encodes the visual perception data into a visual word sequence of the target dimension, the text encoder encodes the query text into a text word sequence of the target dimension, and the trajectory encoder encodes the historical trajectory data into a trajectory word sequence of the target dimension. The visual word sequence, text word sequence, and trajectory word sequence are input into the pre-trained language model, and the pre-trained language model outputs a perceptual feature vector of the target dimension. By performing multimodal unified modeling of vision, text, and trajectory in a unified representation space, the semantic understanding and environmental perception capabilities of the visual language model are improved.

[0049] In one embodiment of this application, visual perception data is encoded into a visual word sequence of the target dimension by a visual encoder, including: extracting multi-view visual images from the visual perception data; projecting the multi-view visual images onto a target three-dimensional space; back-projecting the pixel positions and depths of the projected images in the target three-dimensional space onto a vehicle coordinate system; projecting the three-dimensional projection points of the vehicle coordinate system onto multiple mutually orthogonal two-dimensional feature planes constructed based on the vehicle coordinate system; and generating a visual word sequence of the target dimension based on the two-dimensional projection points on the multiple two-dimensional feature planes.

[0050] Among them, multi-view visual images can be acquired by multiple cameras such as surround view or forward view on the vehicle; the target three-dimensional space is the three-dimensional space under the vehicle coordinate system; back projection is to convert the two-dimensional coordinates of pixels in the image into three-dimensional spatial coordinates; the two-dimensional feature plane can be three planes, corresponding to the bird's-eye view plane (BEV, XY plane), the longitudinal profile (XZ plane), and the transverse profile (YZ plane), respectively, to perform low-dimensional and highly expressive structured modeling of the three-dimensional space from different perspectives; the visual word sequence is a fixed-length sequence after flattening multiple two-dimensional plane features.

[0051] It is understood that the embodiments of this application can extract multi-view visual images from visual perception data and project the multi-view visual images onto the target three-dimensional space. The pixel positions and depths of the projected images in the target three-dimensional space are back-projected onto the vehicle coordinate system. Then, the three-dimensional projection points of the vehicle coordinate system are projected onto multiple mutually orthogonal two-dimensional feature planes constructed based on the vehicle coordinate system. A visual word sequence of the target dimension is generated based on the two-dimensional projection points on the multiple two-dimensional feature planes. Through coordinate transformation and back-projection, the number of words is decoupled from the number and resolution of cameras. The images from all perspectives are modeled on a unified two-dimensional feature plane, eliminating the differences between multiple perspectives. At the same time, by continuously modeling the space of multi-view visual images, geometric consistency is maintained while effectively avoiding the high computation and storage costs caused by directly constructing dense three-dimensional voxels or unfolding words per camera. Since the resolution of the two-dimensional feature plane is fixed, the number of generated visual words remains constant in different scenarios, thereby significantly reducing the computational complexity and inference latency of multi-camera perception under high-resolution input conditions and meeting the real-time driving needs of vehicles.

[0052] Specifically, the visual encoder in this application uses a visual Transformer to segment the image into blocks, and then encodes them to form a token (i.e., word) sequence. The specific processing procedure is as follows.

[0053] First, for a single input image of size W×H, it is encoded into patch features (i.e., block features). Then, each image is downsampled by 2x bilinear sampling to obtain... Features. While processing single images directly for multi-view images is simple to implement, it generates a number of tokens that are linearly proportional to the image resolution and the number of cameras. To obtain a 360-degree view of the surrounding environment, vehicles typically use 6 to 10 cameras. The image patch-based tokenization of these cameras generates tens of thousands of tokens at each time step, hindering real-time inference performance.

[0054] To address this, embodiments of this application employ a highly efficient multi-camera characterizer that utilizes three planes as 3D inductive biases to efficiently represent multiple camera images simultaneously. Since the dimensions of the three planes are fixed, the number and resolution of the input cameras are decoupled from the number of tokens generated.

[0055] Specifically, this multi-camera characterizer first constructs three mutually orthogonal two-dimensional feature planes in a unified vehicle coordinate system, corresponding to the BEV (Bird's Eye View) plane (XY plane), the longitudinal profile (XZ plane), and the lateral profile (YZ plane), respectively. These are used for low-dimensional, highly expressive structured modeling of the three-dimensional space from different perspectives. Each plane has a fixed spatial resolution and channel dimension, and its size is pre-set during the model design phase, ensuring that the number of generated tokens is independent of the number of input cameras and image resolution. During feature construction, camera intrinsic and extrinsic parameters are used to extract image patch features from different perspectives. Projected onto a unified three-dimensional space. For the first... Each camera, and its corresponding intrinsic parameter matrix With extrinsic matrix Given. Arbitrary pixel position. and its depth Three-dimensional points that can be back-projected into the vehicle coordinate system: ; in, Represents three-dimensional spatial coordinates. Let be the inverse of the rotation matrix of the nth camera relative to the vehicle coordinate system, and z be the depth. Let be the inverse of the intrinsic parameter matrix of the nth camera. Let be the translation vector of the nth camera relative to the vehicle coordinate system.

[0056] Subsequently, through differentiable geometric mapping and a weighted aggregation mechanism, these two-dimensional image features from multiple cameras and viewpoints are accumulated onto their corresponding three planes. For image features from different cameras... Its corresponding three-dimensional coordinates The projections will be applied to three planes respectively: ; in, For the projection operator of the XY plane, For the projection operator of the XZ plane, This is the projection operator for the YZ plane. The projection operator is used to output the two-dimensional coordinates of the corresponding plane.

[0057] Feature writing is performed on the corresponding plane using bilinear interpolation: ; in, Indicates the interpolation weights. Let be the eigenvalue of the two-dimensional feature map at coordinate (i, j) on the c-th projection plane. This represents the preprocessed pixel feature values ​​output by the nth camera in the original visual image. Features from different cameras and viewpoints are accumulated and normalized on a plane. For any 3D position... Its final spatial features are obtained by interpolating and fusing three planes: ; in, For the final spatial characteristics, , , These are the planar feature mapping functions corresponding to the three projection planes.

[0058] The above process enables continuous modeling of spatial structures, which can effectively avoid the high computational and storage costs associated with directly constructing dense 3D voxels or unfolding tokens per camera while maintaining geometric consistency.

[0059] Finally, after feature aggregation, the feature maps on the three planes are further flattened and mapped into a fixed-length visual token sequence. ; in, A visual token sequence, For feature flattening operation, , , These are the corresponding two-dimensional feature maps on the respective projection planes.

[0060] The final result is a fixed-length sequence of visual tokens, which serves as the visual input to the pre-trained language model. Since the resolution of the three-plane representation is fixed, the number of generated visual tokens remains constant across different scenarios, significantly reducing the computational complexity and inference latency of the model under high-resolution input conditions.

[0061] By introducing a three-plane representation, this application embodiment can achieve efficient compression of multi-camera visual input while maintaining geometric consistency and spatial expressiveness, decoupling the number of visual tokens from the number of cameras and image resolution, and improving the inference speed of the visual language model.

[0062] In one embodiment of this application, before inputting the visual lexical sequence, text lexical sequence, and trajectory lexical sequence into the pre-trained language model, the method further includes: identifying visual lexical units at multiple time steps and multiple perspectives in the visual lexical sequence; performing temporal embedding and camera embedding on the visual lexical units at multiple time steps and multiple perspectives; flattening the embedded visual lexical units into a target input sequence; and compressing the target input sequence according to a pre-set query vector to obtain a compressed visual lexical sequence.

[0063] Among them, temporal embedding is used to identify the frame sequence position of the word; camera embedding is used to identify the camera view of the word; query vector is a learnable global aggregation vector; target input sequence is a spatiotemporally concatenated global word sequence.

[0064] It is understood that the embodiments of this application can identify visual words at multiple time steps and from multiple perspectives in a visual word sequence, perform temporal embedding and camera embedding on the visual words at multiple time steps and from multiple perspectives, flatten the embedded visual words into a target input sequence, and then compress the target input sequence according to a pre-set query vector to obtain a compressed visual word sequence. By using the query vector, the visual words from multiple time steps and from multiple perspectives are globally temporally aggregated, temporally compressed, and organized into a spatiotemporal feature set across time and perspective. Global attention aggregation is then performed through a fixed query vector to achieve cross-frame redundant information compression, improve the inference speed of subsequent visual language models, and meet the real-time driving needs of vehicles.

[0065] Although the above-mentioned use of a two-dimensional feature plane as a three-dimensional inductive bias to simultaneously represent multi-view images has significantly reduced the number of tokens required to represent visual observations, there is still redundancy in cross-frame information and structured features. To further improve the efficiency of visual information processing, this application embodiment can employ a multi-camera video tokenizer to directly encode the entire sequence of camera observations from multiple time steps. By using a fully self-attention layer and a fixed set of query vectors, a set of image tokens from multiple cameras and time steps is compressed, thereby controlling the size of the information bottleneck.

[0066] Specifically, image features from T consecutive time steps and N cameras are first organized into a temporal-multi-view feature set. For each time step t and camera n, the patch features obtained after processing the corresponding image by the visual encoder are regarded as the initial visual token, and explicit temporal embedding and camera view embedding are appended to encode its temporal location and view source information, thus forming a joint spatiotemporal feature representation. Let the input contain T consecutive time steps and N camera views. For the t-th time step and the n-th camera, the patch features obtained after visual encoder and spatial compression are represented as follows: ; in, For the nth camera at the t-th time step Visual features of each patch The t-th frame is obtained by three-plane fusion. A spatial visual token This refers to the number of patches per frame per camera. For the feature dimension, to explicitly model temporal and viewpoint information, temporal embedding and camera embedding are added to each visual token: ; in, The feature vector is obtained by overlaying time and viewpoint embeddings. Let t be the time embedding vector, where t is the t-th time step and time is the time attribute. Let be the view embedding vector, cam be the camera attribute, and n be the nth camera.

[0067] Finally, all time steps are flattened with the camera's visual tokens to form a global input sequence: ; in, The input sequence is T, where T is the number of frames, N is the total number of cameras, P is the number of patches per frame per camera, and D is the dimension of the features in a single patch. To sequentially stitch together spatial visual tokens from different time steps and camera sources.

[0068] Based on this, a fixed set of learnable query vectors is introduced. As compact semantic carriers at the video level, these query vectors interact with visual tokens from all cameras and time steps through a fully self-attention mechanism, thereby adaptively aggregating key information across time and viewpoints. This process can be formally represented as: ; in, Represents a fixed number of query vectors. These represent the key and value features from all time steps, cameras, and patches, respectively. Finally, the visual token sequence (i.e., visual lexical sequence) obtained by compression using a multi-camera video tokenizer is fed into a pre-trained language model of the cognitive backbone. It is jointly modeled with text tokens (i.e., text lexical sequences) and trajectory tokens (i.e., trajectory lexical sequences) in a unified sequence space, providing a compact and high-information-density visual representation for efficient and stable cross-modal temporal reasoning and driving decision-making.

[0069] This application's embodiments can automatically select the most critical spatiotemporal information for the current task globally and compress it into a fixed-length output token sequence. Unlike the frame-by-frame and camera-by-camera independent encoding method, this application explicitly captures cross-frame dynamic changes and long-term temporal dependencies, effectively eliminating a large amount of repetitive static background information between adjacent frames. At the same time, it retains sensitivity to moving targets, interaction relationships, and scene evolution trends. Furthermore, since the number of output tokens is fixed in the design phase, the final number of visual tokens generated is completely decoupled from the input time steps, number of cameras, and image resolution. This significantly reduces the computational burden of subsequent visual language models while ensuring information expression capabilities.

[0070] In summary, this application embodiment performs spatial and temporal compression on multi-camera visual images. First, spatial dimensional compression is performed on single-frame, multi-camera images: each camera image undergoes patch segmentation and initial feature extraction via a visual Transformer. Then, using three planes as 3D spatial inductive biases, the patch features from different cameras are projected onto a unified vehicle coordinate system using camera intrinsic and extrinsic parameters for spatial aggregation, ultimately generating a fixed-length compact visual token. This eliminates the problem of the token number linearly increasing with the number of cameras and image resolution. After this step, the temporal compression stage begins, where the spatially compressed visual tokens from T consecutive frames are added to temporal and camera embeddings, organizing them into a spatiotemporal feature set spanning time and viewpoints. Global attention aggregation is then performed using a fixed query vector to compress redundant information across frames.

[0071] It should be noted that the camera viewpoint is not discarded during the spatial compression stage. The camera viewpoint is retained as location information and used to distinguish the information source during temporal fusion, avoiding semantic aliasing when different viewpoints are aggregated across frames. Specifically, when generating tokens in the spatial stage, a camera embedding is attached to each token. In the temporal stage, all frames / camera tokens are flattened and then the temporal embedding + camera embedding are stitched together, and then attention is input for further compression.

[0072] In one embodiment of this application, encoding query text into a target-dimensional text word sequence using a text encoder includes: extracting natural language instructions and scene description text from the query text; encoding the natural language instructions and scene description text into a natural language text sequence; calculating the probability distribution of sub-word sequences corresponding to text segments in the natural language text sequence; mapping the natural language text sequence into a text word sequence based on the probability distribution; and mapping the text word sequence to the target dimension to obtain the target-dimensional text word sequence.

[0073] Among them, natural language instructions can include navigation commands; scene description text can be text describing the road conditions around the vehicle; sub-word sequence is the smallest semantic unit after word segmentation; probability distribution is the probability of sub-word occurrence.

[0074] It is understood that the embodiments of this application can extract natural language instructions and scene description text from the text, encode the natural language instructions and scene description text into a natural language text sequence, calculate the probability distribution of the sub-word sequence corresponding to the text segment in the natural language text sequence, and then map the natural language text sequence into a text word sequence according to the probability distribution. The text word sequence is then mapped to the target dimension to obtain the text word sequence of the target dimension. By encoding long text into a semantically stable and length-controllable text word sequence, the computational load of the model is reduced. At the same time, the dimensions of the text word sequence and the visual word sequence are aligned to enhance the cross-modal alignment capability, so as to ensure that words from different modalities can interact in the same vector space.

[0075] In one embodiment of this application, mapping a text lexical sequence to a target dimension to obtain a text lexical sequence of the target dimension further includes: identifying sub-word sequences in the text lexical sequence; obtaining vector representations of sub-word sequences in the text lexical sequence; mapping the vector representations of sub-word sequences to text lexical sequences of the target dimension, and introducing positional encoding and semantic type encoding into the text lexical sequences of the target dimension; and generating a text lexical sequence of the target dimension based on the text lexical sequences of the target dimension, the positional encoding, and the semantic type encoding.

[0076] Among them, positional encoding is used to identify the text order; semantic type encoding is used to distinguish the type of instructions and descriptions.

[0077] It is understood that the embodiments of this application can identify sub-word sequences in a text lexical sequence and obtain the vector representation of the sub-word sequence. The vector representation of the sub-word sequence is mapped to text lexical units in the target dimension. Position encoding and semantic type encoding are introduced into the text lexical units in the target dimension. A text lexical sequence in the target dimension is generated based on the text lexical units in the target dimension, the position encoding and the semantic type encoding. By introducing position encoding and semantic type encoding into the text lexical units, different semantic roles such as instructions, scene descriptions and system prompts can be distinguished, thereby improving the pre-trained language model's ability to understand complex language structures and multi-segment instructions.

[0078] Specifically, the text encoder processing procedure of this application embodiment is as follows.

[0079] For text encoders, a word-level segmenter can be used to encode natural language instructions and scene descriptions into semantically stable and length-controllable text token sequences. These tokens can then be mapped to feature dimensions consistent with visual tokens through learnable embeddings, thereby enhancing cross-modal alignment and open-vocabulary generalization capabilities.

[0080] First, the input natural language instructions and scene description text are standardized and preprocessed, including case normalization, punctuation standardization, and the insertion of special control characters, to adapt to the needs of multimodal instruction following and conversational reasoning.

[0081] Subsequently, a word segmenter is used to perform sub-word-level segmentation of the text. Without relying on explicit whitespace delimiters, the original text is mapped to a set of sub-word tokens with controllable length and good statistical coverage, thus effectively alleviating the out-of-vocabulary (OV) word problem caused by open vocabularies and long-tail concepts. Given the input natural language text sequence: ; in, A sequence of natural language text. This represents the i-th character or word fragment in the original text, where L is the total length of the natural language text sequence, and it is mapped to a sub-word-level token sequence: ; in, For sub-word level token sequences, For the i-th text sub-word token, This represents the total length of the text sub-word token sequence.

[0082] This process is achieved by maximizing the probability of the subword sequence: ; in, The length of the current candidate word sequence T. For the word The probability distribution in the pre-trained vocabulary ensures semantic stability while effectively controlling the length of text tokens and supporting open-vocabulary representation.

[0083] After obtaining the text token sequence, each sub-word token is mapped to its corresponding embedding vector using a lookup table, and then a linear projection layer is used to uniformly map them to the same feature dimension as the visual token and trajectory token. This ensures that tokens of different modalities can directly interact with each other within the same vector space. For each text sub-word token... First, the initial word vectors are obtained through table lookup embedding: ; in, Let be the initial word vector for the i-th text sub-word token. For word embedding lookup operations, The dimension of the original word embedding vector. Let be the i-th text sub-word token.

[0084] Subsequently, a learnable linear mapping is used to project the text embedding onto a feature dimension consistent with the visual token. : ; in, This is the projected text feature vector. It is a learnable linear mapping matrix. The table is the first A text sub-word token, This is the bias vector.

[0085] To further enhance sequence modeling capabilities, positional encoding and semantic type encoding are introduced into the embedded text tokens to distinguish different semantic roles such as instructions, scene descriptions, and system prompts, thereby improving the model's ability to understand complex language structures and multi-segment instructions. To enhance sequence structure modeling capabilities, positional encoding and semantic type encoding are introduced into the text token embedding: ; in, This is the feature vector of the embedded text token. For the first The positional encoding of each text sub-word token The semantic types are encoded (such as instructions, scene descriptions, system prompts, etc.), ultimately yielding the output of the text encoder: ; in, A sequence of text tokens. This represents the total length of the text sub-word token sequence.

[0086] The resulting text token sequence is concatenated into the multimodal input sequence and input together with the token sequences from the visual encoder and trajectory encoder into the pre-trained language model of the visual language model backbone. The deep fusion of language semantics and environmental perception information is achieved through a cross-modal self-attention mechanism.

[0087] In one embodiment of this application, encoding historical trajectory data into a trajectory word sequence of the target dimension by a trajectory encoder includes: extracting a path point sequence from the historical trajectory data; calculating the motion increment of adjacent time steps based on the path points in the path point sequence; converting the path point sequence into a discrete trajectory word sequence based on the motion increment; and generating a trajectory word sequence of the target dimension based on the discrete trajectory word sequence.

[0088] Among them, motion increments include changes in displacement, angle, and velocity; discrete trajectory terms are symbols after the motion increments have been quantized.

[0089] It is understood that the embodiments of this application can extract path point sequences from historical trajectory data, calculate the motion increment of adjacent time steps based on the path points in the path point sequence, convert the path point sequence into a discrete trajectory word sequence based on the motion increment, and then generate a trajectory word sequence of the target dimension based on the discrete trajectory word sequence. By converting continuous trajectories into discrete trajectory words, it can adapt to the input of the pre-trained language model, and at the same time, it can filter high-frequency noise in the trajectory data, making the trajectory word sequence smoother.

[0090] In one embodiment of this application, generating a target-dimensional trajectory lexical sequence based on a discrete lexical sequence includes: obtaining vector representations of trajectory lexical units in the discrete trajectory lexical sequence; mapping the vector representations of the trajectory lexical units to target-dimensional trajectory lexical units, and introducing time position encoding into the target-dimensional trajectory lexical units; and generating a target-dimensional trajectory lexical sequence based on the target-dimensional trajectory lexical units and the time position encoding.

[0091] Among them, time location coding is used to identify the trajectory timing.

[0092] It is understood that the embodiments of this application can obtain the vector identifiers of trajectory words in a discrete trajectory word sequence, map the vector representation of the trajectory words to the trajectory words in the target dimension, and introduce time position encoding in the trajectory words in the target dimension to preserve the trajectory time order, so that the subsequent pre-trained language model can distinguish the motion state changes corresponding to different time steps. Based on the trajectory words in the target dimension and the time position encoding, a trajectory word sequence in the target dimension is generated, and the trajectory word sequence is kept synchronized with the visual word sequence and the text word sequence to enhance cross-modal alignment capability, so as to ensure that words in different modalities can interact in the same vector space.

[0093] Specifically, the trajectory encoder in this application converts continuous path point trajectories into discrete token sequences through lexical analysis, and also maps them to feature dimensions consistent with visual tokens through learnable embeddings. The specific processing is as follows.

[0094] First, the original continuous vehicle trajectory is represented as a sequence of path points ordered by time: ; in, This is the original trajectory sequence of the vehicle. Let t be the vehicle state at the t-th time step.

[0095] The state of a single time step is defined as: ; These represent the vehicle's position in the local or global coordinate system, respectively. ), heading angle and speed For this continuous trajectory signal, a trajectory discretization strategy is introduced. The continuous numerical signal is divided into a set of discrete symbol units according to predefined bucketing rules and structured syntax, thereby transforming the trajectory representation into a token sequence that can be efficiently processed by large language models. First, the motion increment between adjacent time steps is calculated: ; in, Let be the vehicle motion increment from time step t to time step (t+1). ) represents the change in position. This represents the change in heading angle. The velocity is the rate of change.

[0096] Subsequently, the motion increments in each dimension are quantized and bucketed: ; in, This represents predefined quantization and classification rules. These are discrete symbols from the trajectory vocabulary. Through the above process, continuous trajectories are converted into discrete trajectory token sequences: ; in, For the complete trajectory token sequence, Let be the i-th trajectory token, i.e., the discrete motion symbol.

[0097] This process explicitly eliminates high-frequency numerical noise and redundant sampling points while maintaining the continuity of motion semantics. For each trajectory token... First, its initial vector representation is obtained through an embedding lookup table: ; in, This is the initial embedding vector corresponding to the trajectory token. This represents the original embedding dimension of the trajectory.

[0098] Subsequently, it is projected onto a feature dimension consistent with both the visual token and the text token using a linear mapping. : ; in, This is the trajectory feature vector after dimension alignment. and These are learnable parameters.

[0099] To preserve the temporal sequence information of the trajectory, time position encoding is added to the trajectory token embedding: ; in, This refers to the trajectory token features output after superimposing time series data. Encodes the time location.

[0100] This enables the model to distinguish the changes in motion state at different time steps, ultimately yielding the trajectory encoder output: ; in, It is a sequence of trajectory tokens.

[0101] Finally, the trajectory token sequence is concatenated with the text token sequence and the visual token sequence in a unified sequence space and input into the backbone network of the large language model. Through a cross-modal self-attention mechanism, the model jointly models historical motion patterns, current environmental perception, and language commands, providing structured motion prior information for subsequent driving decisions and trajectory generation.

[0102] In summary, the pre-trained visual language model in this application includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model. The visual encoder, text encoder, and trajectory encoder respectively convert the input visual perception data, query text, and trajectory into a token stream. The trajectory token is then concatenated with the text token and visual token in the sequence dimension. ; in, This is the concatenated multimodal token sequence. A sequence of text tokens. A visual token sequence, It is a sequence of trajectory tokens.

[0103] The concatenated multimodal token sequence is input into a pre-trained language model, where joint modeling is achieved through a cross-modal self-attention mechanism. The model outputs the hidden states of the final layer, which are then decomposed into language features according to their respective modalities. Visual features and action characteristics .

[0104] In one embodiment of this application, before inputting the visual perception data, query text, and historical trajectory data into the pre-trained visual language model, the method further includes: acquiring historical vehicle operation data; extracting the visual perception data, query text, and historical trajectory data from the historical operation data; extracting at least one multi-view visual image at a target time and a reference visual image after the target time from the visual perception data; extracting at least one historical trajectory at a target time and a reference trajectory after the target time from the historical trajectory data; generating a first training dataset based on the multi-view visual image, query text, historical trajectory, reference visual image, and reference trajectory; and jointly training the visual language model and the pre-established diffuse world model using the first training dataset to obtain the pre-trained visual language model.

[0105] Among them, historical operation data can be labeled driving logs; reference visual images are future real images and serve as training and supervision signals; reference operation trajectories are future real paths and serve as training and supervision signals; and the diffuse world model is a diffusion model that generates future images.

[0106] It is understood that the embodiments of this application can acquire historical operating data of vehicles, extract visual perception data, query text, and historical trajectory data from the historical operating data, and extract at least one multi-view visual image of a target time and a reference visual image after the target time from the visual perception data, and extract at least one historical operating trajectory of a target time and a reference operating trajectory after the target time from the historical trajectory data. Then, a first training dataset is generated based on the multi-view visual image, query text, historical operating trajectory, reference visual image, and reference operating trajectory. The visual language model and the pre-established diffusion world model are jointly trained using the first training dataset to obtain a pre-trained visual language model. By using future reference visual images as training supervision, the supervision deficit problem caused by simply training and supervising the model to predict future trajectories or actions can be alleviated, enabling the visual language model to learn the dynamic evolution law of the real environment and improve the generalization ability of the visual language model.

[0107] In one embodiment of this application, a visual language model and a pre-established diffused world model are jointly trained using a first training dataset to obtain a pre-trained visual language model. This includes: inputting multi-view visual images, query text, and historical running trajectories from the first training dataset into the visual language model; outputting a perceptual feature vector from the visual language model; extracting visual and action features from the perceptual feature vector; generating a target planning trajectory for the vehicle after a target time based on the perceptual feature vector; calculating a first loss of the visual language model based on the target planning trajectory and a reference running trajectory; generating a latent representation based on the reference visual image; generating noisy features based on the latent representation; inputting the noisy features, visual features, and action features into the diffused world model; outputting a reconstructed visual image from the diffused world model; calculating a second loss of the diffused world model based on the reference visual image and the reconstructed visual image; calculating a joint loss based on the first and second losses; and updating the model parameters of the pre-trained language model based on the joint loss until the pre-training of the visual language model is complete.

[0108] Among them, visual features Visual semantic features and action features output by the visual language model. The driving action features output by the visual language model; the latent representation is a low-dimensional encoding of the future reference visual image.

[0109] It is understood that the embodiments of this application can extract visual features and action features from the perceptual feature vectors output by the visual language model, generate the target planning trajectory of the vehicle after the target time based on the perceptual feature vectors, calculate the first loss of the visual language model based on the target planning trajectory and the reference running trajectory, generate a latent representation based on the reference visual image, and add noise to obtain noisy features. The noisy features, visual features and action features are input into the diffusion world model, and the diffusion world model outputs a reconstructed visual image. The second loss of the diffusion world model is calculated based on the reference visual image and the reconstructed visual image. Then, the joint loss is calculated based on the first loss and the second loss. The model parameters of the pre-trained language model are updated based on the joint loss until the pre-training of the visual language model is completed. Through action training supervision of the visual language model and image training supervision of the diffusion world model, the training accuracy of the visual language model is improved, enabling the visual language model to learn environmental changes and improve the generalization ability of the visual language model.

[0110] The joint loss function in this embodiment can be: ; in, For joint losses, The first loss is the loss from motion prediction and monitoring. For balance coefficient, The second loss is the diffuse world model loss, and the vector field matching loss is the vector field matching loss.

[0111] In the visual language model training phase of this application, a supervised loss optimization model can be used to predict real action sequences. : ; in, The visual language model predicts action features, where K is the total length of the action sequence. This is the label for the k-th real action.

[0112] In one embodiment of this application, the loss function for the second loss is: ; in, This is the second loss. Let be the mathematical expectation, representing the expectation at... , , Take the average on the joint distribution. It is a noisy feature. For noise intensity, It is Gaussian noise. For the visual features at time step t, The motion characteristics at time step t, for The latent representation at time step, where t is the prediction step size. For parameters Conditional vector field function, The estimated vector field is predicted based on noisy features, noise intensity, visual features, and motion features. It is a real vector field.

[0113] This application embodiment alleviates the supervision deficit problem of visual language models by adding a diffuse world model during the training process of the visual language model.

[0114] Because the supervision target used in related technologies only supervises the future trajectory actions predicted by the model, this results in a sparse supervision signal, compressing the high-dimensional perceptual input into a few low-dimensional control signals, leading to a supervision deficit. To address this issue, this application introduces a diffuse world model, which generates future images based on the output features of a visual language model by training a latent diffuse model, thus providing dense supervision. Since the model is simultaneously constrained by all current features, predicting future images requires learning to predict dynamics, rather than simply performing a reconstruction task. This world model learns to predict future visual scenes conditioned on the current visual and action features of the VLM. Its training task is to denoise the noisy latent representation of future images and optimize it using the MSE (Mean Squared Error) objective. The entire framework is trained end-to-end by optimizing a joint objective that combines action prediction loss and diffuse world model loss. .

[0115] Specifically, embodiments of this application employ a pre-trained visual autoencoder to process real images at future time steps. Mapping to a low-dimensional latent space yields the corresponding latent representation: ; in, Represents a visual encoder. The latent space dimension, which serves as the modeling object for the diffuse world model, effectively reduces the computational complexity associated with direct modeling in pixel space. During the training phase, the latent representation... By applying a continuous-time Gaussian noise perturbation, a forward diffusion process is constructed: ; in, Let represent noise intensity, and represent the diffusion timescale. The diffusion world model takes the current multimodal state as conditional input, including visual features output from the visual language model. With action characteristics And learn a conditional vector field function: ; The conditional vector field function is used to predict the evolution direction of the latent representation along the diffusion time dimension. The model training objective is to directly regress the velocity field corresponding to the true probability flow, thereby avoiding the explicit construction of the back-diffusion process. The specific loss function is defined as: ; This objective encourages models to learn to push noisy future potential representations back to the true future state along the correct dynamic direction, given the current multimodal conditions, thereby explicitly modeling the temporal evolution and interaction patterns of the environment.

[0116] During the inference phase, the diffused world model starts with a latent representation initialized by random noise, and then, under certain conditions... Under the constraints, future potential states are gradually generated along the learned vector field. And reconstruct the corresponding future image using a decoder: ; in, For the decoder output Future images of time steps Represents a visual decoder. for Potential representation of time steps.

[0117] By introducing this diffusion-based world model, the model receives not only sparse trajectory-action supervision during training but also dense temporal supervision signals from future visual predictions. This supervision forces the model to learn the true dynamic evolution of the environment, rather than merely fitting low-dimensional control outputs, fundamentally alleviating the supervision deficit problem and providing more structured training signals for subsequent policy learning and decision reasoning.

[0118] In summary, the training of the visual language model in this application embodiment is as follows: Figure 4 As shown, the steps are as follows: Step 1: At the time step The model input is constructed from current observation images from multiple cameras, historical trajectory sequences, and language instructions or task descriptions. These are then mapped into a multimodal token sequence of a unified dimension via a visual encoder, a text encoder, and a trajectory encoder, respectively. .

[0119] Step 2: Input the multimodal token sequence into the pre-trained language model, perform joint modeling through cross-modal self-attention, and output a high-dimensional semantic state representation: ; in, This is a high-dimensional semantic state representation, i.e., a perceptual feature vector. It is a multimodal token sequence.

[0120] Classify language features according to modality Visual features With action characteristics .

[0121] Step 3: Use a trajectory encoder to map the real trajectory sequence to a real action token sequence, and combine it with real action features. Calculate the loss for action prediction.

[0122] Step 4: Image the future reality Encoding into the latent space: Encoding future real images Encode the data into the latent space and add noise. The diffuse world model, conditioned on the noisy latent features, diffusion time, and VLM output features, predicts the probability flow vector field and calculates the diffuse world model loss.

[0123] Step 5: Combine the action prediction loss and the world model loss in a weighted manner to form an end-to-end training objective, and perform end-to-end backpropagation and parameter update.

[0124] In step S103, the historical trajectory data is encoded into a trajectory feature vector of the target dimension, the vehicle state data is encoded into a state feature vector of the target dimension, and the target planned trajectory of the vehicle is generated based on the perception feature vector, the trajectory feature vector, and the state feature vector.

[0125] The target planned trajectory is the future driving trajectory that the vehicle can execute; the historical trajectory data and vehicle status data can be encoded using independent MLP (Multilayer Perceptron), i.e., fully connected layers.

[0126] It is understood that the embodiments of this application can encode historical trajectory data into trajectory feature vectors of the target dimension and vehicle state data into state feature vectors of the target dimension. Then, based on the perception feature vector, trajectory feature vector and state feature vector, the target planned trajectory of the vehicle is generated, realizing the integration of perception, semantics and planning, avoiding the accumulation of errors caused by staged calls, and improving the robustness of trajectory planning.

[0127] This application embodiment can acquire vehicle visual perception data, query text, historical trajectory data, and vehicle status data. The visual perception data, query text, and historical trajectory data are input into a pre-trained visual language model. The visual language model performs multimodal joint modeling of vision, text, and trajectory in a unified representation space, and outputs a perception feature vector in the target dimension. This significantly improves the semantic understanding and environmental perception capabilities of the visual language model. At the same time, the historical trajectory data is encoded into a trajectory feature vector in the target dimension, and the vehicle status data is encoded into a state feature vector in the target dimension. Then, the target planned trajectory of the vehicle is generated based on the perception feature vector, trajectory feature vector, and state feature vector. This achieves end-to-end integration of perception, semantics, and planning, avoids error accumulation caused by staged calls, and improves the robustness and security of trajectory planning.

[0128] In one embodiment of this application, generating a target planned trajectory for a vehicle based on a perception feature vector, a trajectory feature vector, and a state feature vector includes: inputting the perception feature vector, the trajectory feature vector, and the state feature vector into a pre-trained trajectory generation model, and outputting a set of candidate planned trajectories for the vehicle through the trajectory generation model; calculating a first prediction score for a first candidate planned trajectory in the set of candidate planned trajectories, and selecting multiple first candidate planned trajectories for the vehicle from the set of candidate planned trajectories based on the first prediction score; calculating a second prediction score for the selected multiple first candidate planned trajectories, and determining the target planned trajectory for the vehicle from the selected multiple first candidate planned trajectories based on the second prediction score.

[0129] The trajectory generation model can also be called a diffusion planner or trajectory planner; the candidate trajectory planning set contains multiple drivable paths; the first prediction score is a coarse-grained score; the second prediction score is a fine-grained score.

[0130] It is understood that, in the embodiments of this application, the perception feature vector, trajectory feature vector, and state feature vector can be input into a pre-trained trajectory generation model. The trajectory generation model outputs a set of candidate planned trajectories for the vehicle, calculates a first prediction score for the first candidate planned trajectory in the set, selects multiple first candidate planned trajectories for the vehicle from the set based on the first prediction score, calculates a second prediction score for the selected multiple first candidate planned trajectories, and determines the vehicle's target planned trajectory from the selected multiple first candidate planned trajectories based on the second prediction score. By using a multi-level prediction score method to filter candidate planned trajectories, the filtering efficiency is improved, so as to select the optimal target planned trajectory.

[0131] In this embodiment, a trajectory mode selector can be added to the output of the trajectory generation model to select the optimal, target-aligned trajectory from different candidate planned trajectories. The higher the score, the stronger the consistency with the overall target. The trajectory mode selector includes a trajectory decoder, a scorer, and has built-in trajectory filtering logic.

[0132] Specifically, in this embodiment, the trajectory coordinates of multiple candidate planned trajectories generated by the trajectory generation model can be used as query input to a trajectory decoder. This trajectory decoder first interacts with perceptual encoded features through deformable spatial cross-attention, then refines the representation through a cross-attention layer with historical trajectories and vehicle status queries. Finally, the context-rich trajectory feature representation is passed to a scorer (i.e., a trajectory pattern selector) based on a multi-layer fully connected network to predict the score. The structure of the trajectory decoder is as follows: Figure 5 As shown, the network sampling cross-attention calculation process is as follows: Figure 6 As shown.

[0133] This application embodiment can employ a two-stage scoring system, progressing from coarse to fine. The process begins with a coarse scorer that initially selects the Top-K candidate trajectories, and then passes them to a fine-grained scorer for more detailed selection. The score learning uses the following loss: ; in, The loss function for trajectory scoring tasks. For true scores, To predict scores, To calculate the cross-entropy loss between the true score and the predicted score, m is the margin, a positive hyperparameter used to widen the difference in predicted scores between superior and inferior trajectories, thereby improving the ranking discrimination. N is the total number of trajectory pairs participating in the ranking comparison. The index represents two different candidate trajectories, signifying a set of trajectory comparison samples. The true order sign function is defined as follows: if trajectory a is better than trajectory b in the true scores, sign = +1; otherwise, sign = -1. This defines the true superiority / inferiority relationship. To predict the score difference, reflecting the relative quality of the two trajectories as judged by the model, this loss guides the scorer to compare the relative quality of the trajectories, avoiding the difficulty of directly regressing their absolute continuous values, and enhancing the scorer's ability to distinguish subtle differences. The specific implementation steps are as follows: Step 1: Obtain the set of candidate planning trajectories.

[0134] Step 2: Extract the trajectory points, trajectory features, and time embeddings required by the scorer from the candidate planning trajectory set to provide a unified input for the subsequent scoring network.

[0135] Step 3: Perform coarse-grained modeling on all trajectories using a coarse-grained trajectory decoder to output trajectory features; then use a coarse scorer to calculate coarse-grained loss and reward as signals for global screening and stable training.

[0136] Step 4: Based on the rewards obtained in Step 3, perform Top-K selection on the trajectories, retaining only a small subset of the best trajectories and their corresponding features and sub-rewards for more refined modeling.

[0137] Step 5: Feed the Top-K trajectories into the fine-grained trajectory decoder for more refined trajectory modeling, and use the fine scorer to calculate the fine-grained reward, sub-loss, and optimal trajectory index.

[0138] In one embodiment of this application, before inputting the perception feature vector, trajectory feature vector, and state feature vector into the pre-trained trajectory generation model, the method further includes: obtaining a prior trajectory set and a reference running trajectory of the vehicle; generating a second candidate planning trajectory with random Gaussian noise and exploration noise based on the prior trajectory of the prior trajectory set; generating a second training dataset based on the second candidate planning trajectory, perception feature vector, trajectory feature vector, state feature vector, and reference running trajectory; performing reinforcement learning training on the trajectory generation model using the second training dataset; obtaining the training loss and imitation learning loss during the reinforcement learning training process; calculating the final loss based on the training loss and imitation learning loss; and updating the model parameters of the trajectory generation model based on the final loss, until the pre-training of the trajectory generation model is completed.

[0139] Among them, the prior trajectory set serves as the anchor point for the clustered expert trajectories; reinforcement learning is used for reward optimization; and imitation learning is used for regression supervision.

[0140] It is understood that the embodiments of this application can generate a second candidate planning trajectory with random Gaussian noise and exploration noise based on the prior trajectory in the prior trajectory set of the vehicle. A second training dataset is generated based on the second candidate planning trajectory, perception feature vector, trajectory feature vector, state feature vector and reference running trajectory. The trajectory generation model is then trained using reinforcement learning on the second training dataset. The final loss is calculated based on the training loss and imitation learning loss during the reinforcement learning training process to optimize the trajectory generation model. By introducing additional imitation learning on the basis of reinforcement learning, the overfitting phenomenon that occurs in reinforcement learning is avoided, while taking into account diversity and safety.

[0141] In one embodiment of this application, the loss function for the final loss is: ; ; ; in, For the final loss, To mitigate training losses during the learning and training process, To imitate learning loss, To balance the weighting coefficients, G represents the total number of prior trajectories, and G represents the total number of candidate planned trajectories corresponding to a single prior trajectory. To truncate the total number of diffusion steps, d is the index of the diffusion denoising iteration step. This is a discount factor used to mitigate instability in the early denoising steps. Let Gaussian policy function be used. For the learnable parameters of the policy network, The output trajectory of the i-th candidate planned trajectory in the k-th trajectory group at the (d-1)-th diffusion step. For the k-th trajectory group, the i-th candidate planned trajectory is the noisy trajectory at the d-th diffusion step. The relative advantages of each trajectory group, Let be the reward estimate of the i-th candidate planning trajectory in the k-th candidate planning trajectory group after denoising. The mean of all candidate trajectories within a single candidate planning trajectory group, std is the standard deviation of all candidate planning trajectories within a single candidate planning trajectory group.

[0142] In one embodiment of this application, the method further includes: identifying the state of the candidate planning trajectory; if the state is the target state, then setting the relative advantage of the trajectory group to which the corresponding candidate planning trajectory belongs to the target value.

[0143] The target state is the state in which a collision occurs; the target value can be -1.

[0144] It is understood that, in the embodiments of this application, when the state of the candidate planned trajectory is the target state, the relative advantage of the trajectory group to which the corresponding candidate planned trajectory belongs will be directly set to the target value, so as to avoid the collision planned trajectory from gaining a positive advantage, guide the trajectory generation model to prioritize the generation of safe planned trajectories, and the strong penalty mechanism forces the strategy to stay away from collision behavior, reduce the probability of the trajectory generation model outputting dangerous trajectories, and improve the safety and reliability of trajectory planning.

[0145] In one embodiment of this application, generating a second candidate planning trajectory with random Gaussian noise and exploration noise based on the prior trajectories of the prior trajectory set includes: generating a normalized trajectory based on the prior trajectories of the prior trajectory set; adding random horizontal Gaussian noise and vertical Gaussian noise to the normalized trajectory, wherein during the process of adding random Gaussian noise, the Gaussian noise is truncated and diffused based on a preset number of truncation and diffusion steps; and diffusion-adding noise to the normalized trajectory after adding random Gaussian noise to generate a second candidate planning trajectory with random Gaussian noise and exploration noise.

[0146] It is understood that the embodiments of this application can generate a normalized trajectory based on the prior trajectory of the prior trajectory set, and add random horizontal Gaussian noise and vertical Gaussian noise to the normalized trajectory to ensure that the planned trajectory obtained by exploration remains smooth. In the process of adding random Gaussian noise, the Gaussian noise is truncated and diffused based on a pre-set number of truncation and diffusion steps, which can shorten the noise time and improve training efficiency. The normalized trajectory after adding random Gaussian noise is diffused and noise-added to obtain a second candidate planned trajectory with random Gaussian noise and exploration noise.

[0147] The training of the trajectory generation model in this application includes: first, training the trajectory generation model by combining the trajectory prior distribution; and second, optimizing the trajectory generation model by using reinforcement learning.

[0148] I. Train the trajectory generation model by combining the prior distribution of the trajectory.

[0149] Trajectory generation models generate trajectories by iteratively refining a random Gaussian noise to learn an inverse Markov noise process. However, general trajectory generation models often suffer from mode collapse, failing to generate diverse driving behaviors and making it difficult to handle complex driving scenarios and provide a rich set of optional trajectories.

[0150] To overcome the mode collapse problem in the trajectory generation model mentioned above, this application embodiment performs K-Means clustering on expert driving behavior, representing the discrete set of driving intentions as... A set of prior trajectories The trajectory distribution is modeled as a Gaussian mixture model distribution, where the prior trajectory... It consists of a series of waypoints, where T is the total number of waypoints. Let i be the state path point of the i-th path point on the k-th prior trajectory. Each path point... These represent the vehicle's position, heading angle, and velocity in the local or global coordinate system, respectively. Each prior trajectory corresponds to a specific region of the trajectory space, thus representing a specific driving intention, such as overtaking, turning left, or maintaining a straight line. Prior Trajectory The trajectory distribution can be represented as: ; in, For probability distribution, The candidate predicted trajectory is generated based on the k-th prior trajectory. = , This represents the conditional input obtained by concatenating the latent feature vectors, trajectory feature vectors, and state feature vectors of the large visual language model. Indicates the first A prior trajectory or baseline motion trajectory serves as the anchor point for diffusion generation. Let z be the mean offset correction term for the k-th prior trajectory under condition z. Let be the covariance matrix of the k-th prior trajectory under condition z.

[0151] Considering that training a model in the original location path point space is easily affected by sensor noise, which can lead to reduced model convergence, and that downstream low-level vehicle controllers typically require smooth trajectory output to ensure consistent and stable execution on the vehicle, this embodiment does not directly learn the trajectory in the original location path point space. Instead, it uses a motion representation based on dynamic control to achieve better closed-loop performance. Specifically, for any path point, it can be represented as: ; in, The time interval between adjacent path points. and Indicates the location point on the BEV plane. Indicates the yaw angle. Indicates speed, Let represent curvature, and 'a' represent acceleration. During training, the least squares formula with Tikhonov regularization is used... Derive the true control sequence To attenuate high-frequency noise.

[0152] Unlike general diffusion models that predict trajectories directly from random noise, embodiments of this application train a model to predict trajectories and their corresponding... The truncated diffusion model of the offset between, where, Represents the prior trajectory In the context of the environment The specific scene offset, and the entire trajectory distribution can be represented as: ; in, Let Z be the mixture probability distribution of the overall trajectory given an environment Z. It is a mixed weight, representing a given condition. , selection and prior trajectory The probability of the associated driving intention.

[0153] This application's embodiments utilize a truncated diffusion process to shorten the standard noise time table, diffusing each prior trajectory into a corresponding Gaussian trajectory distribution: ; Where d is the diffusion iteration step index, Let be the noisy trajectory of the k-th prior trajectory at the d-th diffusion step. , To cut off the diffusion steps, It is the cumulative retention coefficient (the cumulative term of the noise schedule) in the diffusion process, used to control the proportion of the signal that retains the original trajectory information in step d. Corresponding signal attenuation intensity, It is a standard Gaussian noise term that follows a multidimensional normal distribution. This is used to inject random perturbations into the trajectory to construct uncertainties in the diffusion process. During training, this application will use noisy trajectories... As input, and predict the denoised trajectory and probability scores ,in, It is a mixed weight The abbreviation for [missing information]. Based on the above settings, this truncated diffusion model will select a prior trajectory as the positive mode for optimization during training, which is closer to the true trajectory. The most recent prior trajectory is assigned as a positive sample. Other prior trajectories were assigned as negative samples. Its training objectives are as follows: ; in, The training loss for the first-stage trajectory generation model is the imitation learning loss. The denoised predicted trajectory is the output of the model. For the true value trajectory, The trajectory regression loss is used to constrain the deviation between the predicted trajectory and the true trajectory. For sample labels, For probability fractions, This is the cross-entropy loss for binary classification.

[0154] 2. Reinforcement learning is used to optimize the trajectory generation model.

[0155] Considering that the trajectory generation model based solely on diffusion-driven methods is still limited by imitation learning, with only a single driving mode being supervised in each scenario, this model, while generating diverse trajectories, also produces a large number of low-quality trajectories that could lead to collisions and pose a significant hazard to the system. To address this issue, embodiments of this application further employ reinforcement learning to optimize the trajectory generation model.

[0156] First, the trajectory denoising process is modeled by arbitrarily selecting a prior trajectory from the prior trajectory distribution. Noise is added to the initial trajectory to obtain a noisy trajectory. As input to the trajectory generation model, from Each conditional denoising step in the initial diffusion chain is considered as a Gaussian policy: ; in, Let Gaussian policy function be used. It is the mean predicted by the trajectory generation model. These are noise scheduling parameters during the diffusion process. The noisy trajectory of the k-th prior trajectory at the d-th diffusion step This is the trajectory obtained in step d-1 after denoising the noisy trajectory in step d based on the k-th prior trajectory. To explore the coefficients, I is the identity matrix, used to represent independent homoscedastic Gaussian noise in each dimension, thereby controlling the randomness and stability of the denoising process. This is the covariance term. The strategy updates the gradient using the following formula: ; in, This represents the gradient of the objective function corresponding to the k-th trajectory pattern with respect to the parameter θ, used to guide the parameter updates of the diffusion planner or policy network. This is a conditional generation strategy for the k-th prior trajectory / driving intention pattern during the diffusion planning process. In strategy The expected value of the trajectory distribution obtained from sampling, i.e., the statistical average of the generated trajectory samples. In the current trajectory state Under the given conditions, generate the previous state. The conditional probability distribution, i.e. the backward transition probability (or the denoised distribution of the policy output) in the diffusion model. This is the logarithmic form of the conditional probability described above, used for policy gradient optimization, making the generation process differentiable and facilitating gradient propagation. The advantage function measures the relative merits of the current denoising decision in the d-th step and k-th trajectory mode. It is calculated from the trajectory reward (safety, efficiency, comfort, etc.) and the baseline value.

[0157] The trajectory generation model applies DDIM (Denoising Diffusion Implicit Models) update rules, significantly reducing the number of denoising steps. By setting... This update rule is typically used as a deterministic sampler. To enable broader exploration and avoid the problem of calculating likelihoods on the Dirac distribution, a rule is set during training. Introduce exploration noise, while maintaining [property] during verification. Make deterministic inferences.

[0158] Due to the inherent scale inconsistency between the near and far ends of the trajectory, simply applying additive Gaussian noise at each point will destroy the structural integrity of the trajectory and reduce the quality of the exploration. To address this issue, related techniques incorporate additive Gaussian noise... Apply to normalized trajectory Above, among which, Indicates the length of the trajectory. Represents the x and y coordinates applied to the nth trajectory point. The noise in the path typically produces a jagged, broken exploration path, losing its original smoothness. To maintain the coherence of the noisy trajectory, embodiments of this application add multiplicative Gaussian noise to the normalized trajectory both laterally and longitudinally, ensuring that the resulting exploration path remains smooth, as shown below: ; in, A unique index for each trajectory point within a single trajectory. This is the length of the trajectory. , For the first The normalized horizontal and vertical coordinates of each trajectory point For the first Lateral multiplicative Gaussian noise at each trajectory point For the first Longitudinal multiplicative Gaussian noise at each trajectory point The output exploration trajectory after adding multiplicative noise.

[0159] To prevent modal collapse during reinforcement learning, this application treats each prior trajectory as a group center and applies diffusion noise to each prior trajectory to generate a set of trajectories with random Gaussian noise and exploration noise as a corresponding candidate trajectory group. Then, policy optimization and updates are performed independently within each candidate trajectory group. By constraining policy optimization to the state space of each specific behavioral intention, the model is guided to generate safer and more goal-oriented trajectories without compromising its multimodal capabilities. The corresponding reinforcement learning loss function is expressed as follows: ; in, To compensate for training loss in learning, G represents the total number of prior trajectories, and G represents the total number of candidate planned trajectories corresponding to a single prior trajectory. To truncate the total number of diffusion steps, d is the index of the diffusion denoising iteration step. This is a discount factor used to mitigate instability in the early denoising steps. Let Gaussian policy function be used. For the learnable parameters of the policy network, The output trajectory of the i-th candidate planned trajectory in the k-th trajectory group at the (d-1)-th diffusion step. For the k-th trajectory group, the i-th candidate planned trajectory is the noisy trajectory at the d-th diffusion step. The relative advantage of each trajectory group is represented by the relative advantage of the candidate planning trajectory groups. This method estimates the advantage function by calculating the relative advantage of each candidate planning trajectory group, replacing the value model in the reinforcement learning process and avoiding the introduction of additional model parameter learning requirements. The relative advantage of each candidate planning trajectory group is specifically expressed as follows: ; in, Let be the reward estimate of the i-th candidate planning trajectory in the k-th candidate planning trajectory group after denoising. The mean of all candidate trajectories within a single candidate planning trajectory group, std Let be the standard deviation of all candidate trajectories within a single candidate trajectory group. Furthermore, to avoid misleading learning signals from local group dominance estimates, all negative dominance is truncated to 0, and any trajectory that leads to a collision is penalized, i.e.: ; Furthermore, to prevent overfitting of the diffusion planner during reinforcement learning, this scheme adds an additional imitation learning loss to the reinforcement learning loss for regularization. The final loss function is expressed as: ; in, For the final loss, To mitigate training losses during the learning and training process, To imitate learning loss, These are the balancing weighting coefficients.

[0160] Specifically, the optimization process of the reinforcement learning-based trajectory generation model in this application includes the following stages.

[0161] Phase 1: Collect reinforcement learning training samples.

[0162] Step 1: Load the known prior trajectory set Normalization and noise addition are performed using any prior trajectory as the center, generating noise for each prior trajectory. The candidate trajectories constitute the candidate planning trajectory group. .

[0163] Step 2: For any candidate trajectory Encode and inject time conditions, perform sinusoidal position encoding on trajectory points, and obtain candidate planned trajectory features through trajectory encoder encoding; at the same time, send the current time step into time step encoder to obtain time embedding, realize temporal condition diffusion modeling.

[0164] Step 3: Set a denoising time step, use the trajectory generation model to iteratively generate denoised trajectories, and feed the candidate trajectory features, perceptual features, historical trajectories, state codes and time embeddings into the diffusion decoder in the trajectory generation model. The diffusion decoder outputs the trajectory regression results and classification results for each step, and takes the last layer as the denoising prediction for the current time step.

[0165] Step 4: Based on the denoised prediction output of the trajectory generation model, sample the diffusion output and corresponding likelihood probability from the current diffusion state, and use them as the strategy. Action sampling; the likelihood probability and diffusion output of each time step are saved to form a complete decision trajectory, which prepares for subsequent advantage-weighted policy gradient updates.

[0166] Step 5: Process the diffusion output from Step 4 to obtain an executable continuous trajectory representation, and then concatenate it with the real trajectory. Input the concatenation with the real trajectory and input it into the rule-based trajectory evaluation module to calculate the reward and fine-grained sub-rewards (collision, drivable area, comfort, etc.) for each trajectory group. This can be used for reinforcement learning signal construction.

[0167] Step 6: Normalize the rewards from Step 5 by mean-variance for the trajectory group to obtain the standardized advantage; retain only the samples whose rewards are no worse than the real trajectory, and set the remaining advantages to zero or negative penalty to form a learning signal that "aligns with the real trajectory but allows for surpassing it".

[0168] Step 7: Filter invalid samples using safety-related sub-rewards, and directly apply negative advantages to trajectories that violate safety constraints based on sub-rewards such as collision and drivable zones, thereby injecting hard safety rules into the reinforcement learning process.

[0169] Phase 2: Update the trajectory generation model using advantage estimation.

[0170] Step 1: Read the complete diffusion trajectory and corresponding advantages generated in Phase 1, divide the trajectory into the current diffusion state and the previous state of the target according to time, and construct diffusion chain pairs.

[0171] Step 2: For each denoising time step, read the corresponding diffusion state as the action state that has occurred, restore it to a noisy trajectory point sequence through back normalization, encode the trajectory points with sinusoidal position, and encode the candidate trajectory features through the trajectory encoder. Send the current time step into the time step encoder to obtain the time embedding, and construct the condition information that is completely consistent with the first stage.

[0172] Step 3: Feed the candidate trajectory features, perceptual features, historical trajectories, state encoding and temporal embedding together into the diffusion decoder. The diffusion decoder outputs the trajectory regression results and classification results for each step and caches all intermediate outputs for subsequent imitation learning supervision.

[0173] Step 4: Given the current state and the known diffusion output, calculate the likelihood probability of the model predicting the action. Combine this with the likelihood probability obtained in Stage 1 to calculate the reinforcement learning loss. .

[0174] Step 5: Calculate the L1 loss by comparing the trajectory regression outputs of all time steps and all trajectory decoding layers with the actual trajectory. This serves as a supervisory signal for stable training.

[0175] Step 6: Dynamically weighted based on the existence of positive dominance If there are no positive odds in a batch of data (i.e., the reinforcement learning signal is unreliable), then improve... Weights; if positive samples exist, the weighting is weakened. It is used only as a regularization term to prevent the reinforcement learning process from diverging. Finally, the two losses are combined to update the gradient of the entire model.

[0176] In summary, the process of the trajectory generation model and trajectory mode selector in this application embodiment is as follows.

[0177] Step 1: Load the known prior trajectory set Normalization and noise addition are performed using any prior trajectory as the center, generating noise for each prior trajectory. Several candidate trajectories constitute a candidate trajectory group. .

[0178] Step 2: For any candidate trajectory Encode and inject time conditions, perform sinusoidal position encoding on trajectory points, and obtain candidate trajectory features through trajectory encoder; at the same time, send the current time step into time step encoder to obtain time embedding, and realize temporal condition diffusion modeling.

[0179] Step 3: Set the denoising time step, use the diffusion planner to iteratively generate the denoised trajectory, and feed the candidate trajectory features, perceptual features, historical trajectories, state codes and time embeddings into the diffusion decoder. The diffusion decoder outputs the trajectory regression result and classification result of each step, and takes the last layer as the denoising prediction of the current time step.

[0180] Step 4: Based on the denoising prediction output by the diffusion planner, update the current diffusion state to the state of the previous moment to obtain a new diffusion output, thus completing one deterministic back diffusion.

[0181] Step 5: Apply additional noise enhancement to the trajectory generated in Step 4 in the diffusion space, then denormalize and transform the coordinates to obtain an executable continuous trajectory representation.

[0182] Step 6: Send the trajectory generated in Step 5 into the trajectory evaluation module to calculate the overall reward and multi-dimensional sub-rewards (such as safety, drivability, etc.) for each trajectory, and return the simulation trajectory for subsequent screening and learning signal construction.

[0183] Step 7: Filter the trajectories based on rewards, select some trajectories from the predefined trajectories to retain, and concatenate them with the trajectories generated in Step 5 to form an expanded candidate set.

[0184] Step 8: Extract the trajectory points, trajectory features, and temporal embeddings required by the scorer from the final candidate trajectory set to provide a unified input for the subsequent scoring network.

[0185] Step 9: Perform coarse-grained modeling on all trajectories using a coarse-grained trajectory decoder to output trajectory features; then use a coarse scorer to calculate coarse-grained loss and reward as signals for global screening and stable training.

[0186] Step 10: Based on the rewards obtained in Step 9, perform Top-K selection on the trajectories, retaining only a small subset of the best trajectories and their corresponding features and sub-rewards for more refined modeling.

[0187] Step 11: Feed the Top-K trajectories into the fine-grained trajectory decoder for more refined trajectory modeling, and use the fine scorer to calculate the fine-grained reward, sub-loss, and optimal trajectory index.

[0188] Step 12: Merge the sub-loss generated in Step 11 into the loss dictionary of Step 9, summarize the reward metrics, and add the losses of Step 11 and Step 9 to form the final training loss for gradient update of the entire model.

[0189] Based on the above description, the overall implementation process of the trajectory planning method in this application is as follows: Figure 7 As shown.

[0190] This application's embodiments employ a large visual-language model pre-trained on a specific domain on an urban road driving dataset as the cognitive backbone. This backbone encodes visual images, text queries, and historical trajectories into latent states. As a cognitive token carrying prior driving knowledge, among which, The length of the token. For each token's dimension, it is further mapped to a dimension with specific dimensions through a multi-layer fully connected network. Perceptual features Furthermore, two multi-layer fully connected networks are used to encode the historical trajectory and the vehicle's state into feature vectors with the same feature dimensions. , ,in, Given the trajectory length, then... , , As a conditional input to the planning module, a trajectory generation model is used to generate the corresponding executable trajectory of the vehicle based on the input conditions at each moment.

[0191] In other words, the trajectory planning method in this application embodiment is as follows: Figure 8 As shown, it mainly includes: 1. Encode perception data using pre-trained large models: Input multimodal environmental information (such as multiple cameras, LiDAR, radar, etc.) into a pre-trained visual language large model to generate high-dimensional, semantically rich and spatiotemporally aligned perception representations, providing a unified feature foundation for subsequent trajectory generation and decision-making.

[0192] 2. Train the trajectory generation model by combining trajectory prior distribution: Based on historical trajectory data and task prior information, train the trajectory generation model so that it can generate candidate trajectories that conform to physical constraints and traffic rules, while preserving trajectory diversity and rationality.

[0193] 3. Optimize the trajectory generation model using reinforcement learning: In an interactive environment, the trajectory generated by the trajectory generation model is optimized in terms of safety, efficiency, and comfort through reinforcement learning methods, thereby improving the closed-loop strategy and making the generated trajectory more in line with actual driving needs.

[0194] 4. Trajectory Pattern Selector: The generated set of candidate planned trajectories is filtered and optimized by the trajectory pattern selector. The low-risk and high-efficiency trajectory is selected as the final control command output, realizing the safe and efficient collaborative optimization of multiple candidate trajectories.

[0195] The trajectory planning method of this application embodiment is described below through a specific example, including: Step 1: Multimodal data acquisition and preprocessing.

[0196] Image Acquisition and ISP Processing: All cameras are exposed simultaneously via hardware triggers. The raw image is transmitted to the CCU's ISP hardware pipeline via MIPI CSI-2 or GMSL (Gigabit Multimedia Serial Link), where it undergoes bad pixel correction, black level compensation, depigmentation, automatic white balance, color correction matrix, and gamma mapping in sequence. The output is an sRGB (standard Red Green Blue) image (resolution uniformly scaled to the model input size, such as 640×480). The processed image is written to a circular buffer in DDR (Double Data Rate) memory via DMA (Direct Memory Access).

[0197] Radar and positioning data acquisition: The millimeter-wave radar target list (relative distance, speed, angle) is received via CANFD, parsed by the CPU core, and converted to the vehicle coordinate system. IMU (Inertial Measurement Unit) and wheel speed are acquired via SPI / CAN, and GPS-RTK is received via UART. The real-time operating system running on the CPU uses the camera exposure time as a reference, interpolates all sensor data to the same timestamp, packages them into data frames, and stores them in a shared memory pool.

[0198] Vehicle state fusion: The real-time MCU obtains raw IMU, wheel speed, GPS and steering wheel angle signals through Mailbox (Mailbox Register) or shared memory, runs an extended Kalman filter, and outputs a high-precision vehicle state vector: position, heading angle, longitudinal velocity, longitudinal acceleration, yaw rate and curvature. This state vector is provided to the main AI processor through shared memory.

[0199] Text command input: Natural language navigation commands (such as "Turn left at the unprotected intersection ahead, be careful to yield") are input through the in-vehicle voice assistant or HMI (Human-Machine Interface) touch screen. The commands are converted into a text token sequence by the word segmenter on the CPU (such as SentencePiece / BPE (Byte Pair Encoding)) and then positional and semantic type encodings are added.

[0200] Step 2: Pre-trained visual language model perceptual encoding and intensive supervised training.

[0201] 1. Lightweight visual coding (CPU execution).

[0202] Multi-camera feature extraction: The GPU performs visual Transformer block embedding on each frame of image to obtain an initial visual token, followed by three-plane geometric aggregation, including: In the vehicle coordinate system, three orthogonal planes with fixed resolution are constructed: the XY plane, the XZ plane, and the YZ plane. For each camera and each patch token, the pixel coordinates (u,v) and the depth hypothesis z (which can be provided by LiDAR or a monocular depth network) are used to backproject the intrinsic and extrinsic parameters of the camera to obtain 3D points. The 3D points are projected onto the three planes, and the patch features are accumulated onto the planar feature maps through bilinear interpolation. After aggregation, the three planes are flattened and then compressed into compact visual tokens through cross-attention.

[0203] Temporal compression: For a continuous T frames, the above compact visual tokens are added with temporal embedding and camera embedding, input to a multi-camera video tokenizer, and output video-level visual tokens (i.e., visual lexical sequences).

[0204] 2. Text and trajectory encoding (executed by CPU+GPU).

[0205] Text encoding: After the CPU completes word segmentation, it maps the subtotals to D-dimensional vectors through a learnable embedding table, and then obtains a text token sequence (i.e., a text word sequence) through linear projection and position / type encoding.

[0206] Trajectory encoding: The point sequence of historical trajectory is normalized and binned to be converted into discrete trajectory tokens (i.e. trajectory word sequence), which are then embedded and concatenated with visual and text tokens in the sequence dimension.

[0207] 3. Pre-trained language model inference.

[0208] The concatenated multimodal token sequence is input into the main intervention training language model of the visual language model. Self-attention computation is performed on the GPU tensor core, and the hidden state is output, i.e., the perceptual feature vector is output, which is split into language features according to modality. Visual features and action characteristics ,in, Perceptual features are obtained through linear projection and used as conditions for the trajectory generation model.

[0209] 4. Dense supervision of the diffuse world model (training phase).

[0210] Data preparation: The vehicle packages the current perception features and the real images of future moments, and uploads them to the cloud via 5G.

[0211] Latent space encoding: Cloud GPUs load pre-trained visual autoencoders (VQ-VAE (Vector Quantized Variational Autoencoder) or KL-VAE (Kullback-Leibler Variational Autoencoder)) to encode real images into low-dimensional latent representations.

[0212] The diffusion world model is trained by applying continuous-time Gaussian noise to the low-dimensional latent representation. The conditional diffusion model, noise intensity, and conditions are used as inputs to predict the probability flow vector field. The loss function can be MSE.

[0213] Joint training: The diffusion loss and the action prediction loss of the visual language model are combined as the total loss. Gradient backpropagation updates the encoders of both the diffusion world model and the visual language model simultaneously, forcing the perceptual feature learning environment to evolve dynamically and alleviating the supervision deficit.

[0214] Step 3: Trajectory generation model training and reinforcement learning optimization.

[0215] 1. Prior trajectory library construction (offline construction, placed on a cloud CPU cluster).

[0216] The real trajectory is extracted from expert driving data, and the smooth control sequence (curvature, acceleration) is inversely derived from the position sequence using Tikhonov regularized least squares inverse dynamics to attenuate high-frequency noise.

[0217] K-Means clustering is performed on the control sequences, and the number of clusters is set to obtain a set of prior trajectories. Each prior trajectory contains a curvature sequence and an acceleration sequence of fixed length. The prior trajectory library is stored on the server and loaded into GPU memory during training and inference.

[0218] 2. Training the truncated trajectory generation model (cloud training).

[0219] For each training sample, the prior trajectory closest to the true trajectory is selected as the positive sample, and the rest are negative samples. Gaussian noise is added to the prior trajectory according to the truncated diffusion process to obtain the noisy trajectory.

[0220] The trajectory generation model takes noisy trajectories and perceptual features as input to predict denoised trajectories and classification scores. The loss is calculated on a GPU, and the model parameters are updated.

[0221] 3. Reinforcement learning optimization (optimization in the cloud).

[0222] (1) Sampling: Hundreds of simulation environments are run in parallel. Each environment is randomly selected from the prior trajectory library and multiplicative Gaussian noise (lateral + longitudinal) is added to generate G candidate exploration trajectories. The trajectory generation model performs multi-step denoising based on the characteristics of the current simulation state to obtain the complete trajectory. The simulation environment calculates the reward based on safety (collision), efficiency (travel time), and comfort, and collects (state, action, reward) and stores it in the experience replay buffer.

[0223] (2) Update: Sample batch data from the buffer, calculate the relative advantage within the group, force the collision trajectory to be set to -1, calculate the reinforcement learning loss and imitation learning loss, update the parameters of the trajectory generation model, and the entire reinforcement learning process continues for multiple rounds until the collision rate and efficiency converge.

[0224] Step 4: Vehicle-side reasoning and closed-loop control.

[0225] Perception encoding: The vehicle-side CCU executes steps 1, 2, and 3 of step two to obtain the current perception features and vehicle status.

[0226] Trajectory generation model inference: The GPU loads the prior trajectory from memory into video memory, performs truncation and denoising in parallel to generate candidate trajectories, and each trajectory is a sequence of curvature and acceleration.

[0227] Trajectory Reconstruction: The complete path point sequence is synchronously calculated on the GPU using a trapezoidal integral kinematics model.

[0228] Trajectory mode selection: The candidate trajectory is input into the trajectory mode selector (coarse-to-fine two-stage scoring network), and the optimal trajectory with the highest score is output.

[0229] Real-time control processor (MCU) hard check: The control sequence for the optimal trajectory is sent to the real-time MCU via shared memory, and the MCU performs the following checks: Is the curvature less than the vehicle's maximum steering curvature (e.g., 0.2 m)? -1 ); Is the rate of change of curvature less than the maximum response rate of the steering system? Is the acceleration within the comfort range [-3m / s², 2m / s²]? (This range can be relaxed in emergencies). Lateral acceleration constraint v²·κ ≤ 0.8g.

[0230] If all passes, the MCU sends the control sequence to the drive-by-wire actuator via CAN FD. If a constraint is violated, the MCU rejects the trajectory and reverts to the optimal trajectory of the previous frame or triggers a minimum-risk maneuver (decelerating to a stop within the lane).

[0231] Step 5: Vehicle-Cloud Collaboration and Continuous Learning.

[0232] The vehicle records the input features of each planning decision, the generated trajectory, the MCU inspection results, and the safety operator's actions. When a near collision is detected or the trajectory is rejected by the MCU, the scene data packet (containing current perception features, future real-world images, and the reason for rejection) is compressed and uploaded to the cloud via 5G.

[0233] The cloud server fine-tunes the trajectory generation model and the diffusion world model using hard samples (replay reinforcement learning or joint training). After the updated model is optimized by quantization and TensorRT, it is distributed to the fleet via OTA. At the same time, the cloud regularly performs incremental clustering updates on the prior trajectory library and distributes it synchronously.

[0234] In summary, this application achieves efficient, multimodal information fusion by uniformly encoding multi-camera visual observations, historical trajectories, vehicle status, and language commands through a domain-pre-trained visual language model. Simultaneously, it introduces a diffuse world model to provide dense temporal supervision and combines trajectory prior distribution with reinforcement learning to optimize the trajectory generation model, enabling the generation of diverse, safe, and smooth executable trajectories. Furthermore, the final trajectory mode selector can finely filter the optimal solution from candidate planned trajectories, effectively improving the ability to perceive and understand complex dynamic environments, trajectory prediction accuracy, and decision robustness during trajectory planning. This achieves real-time, efficient, and target-aligned multimodal end-to-end planning and control, significantly enhancing the safety and generalization ability of trajectory planning.

[0235] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0236] According to the trajectory planning method provided in this application, vehicle visual perception data, query text, historical trajectory data, and vehicle state data can be acquired. The visual perception data, query text, and historical trajectory data are input into a pre-trained visual language model. The visual language model performs multimodal joint modeling of vision, text, and trajectory in a unified representation space, and outputs a perception feature vector of the target dimension, which significantly improves the semantic understanding and environmental perception capabilities of the visual language model. At the same time, the historical trajectory data is encoded into a trajectory feature vector of the target dimension, and the vehicle state data is encoded into a state feature vector of the target dimension. Then, the target planned trajectory of the vehicle is generated based on the perception feature vector, trajectory feature vector, and state feature vector, realizing end-to-end integration of perception, semantics, and planning, avoiding error accumulation caused by staged calls, and improving the robustness and security of trajectory planning.

[0237] Embodiments of this application also provide a trajectory planning device.

[0238] Figure 9 This is a schematic diagram of a trajectory planning device provided according to an embodiment of this application.

[0239] like Figure 9 As shown, the trajectory planning device 10 includes: an acquisition module 100, an input module 200, and a generation module 300.

[0240] The acquisition module 100 is used to acquire the vehicle's visual perception data, query text, historical trajectory data, and vehicle status data. The input module 200 is used to input the visual perception data, query text, and historical trajectory data into a pre-trained visual language model, and output the target dimension perception feature vector through the visual language model. The visual language model includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model. The visual encoder encodes the visual perception data into a target dimension visual word sequence, the text encoder encodes the query text into a target dimension text word sequence, and the trajectory encoder encodes the historical trajectory data into a target dimension trajectory word sequence. The visual word sequence, text word sequence, and trajectory word sequence are input into the pre-trained language model, and the pre-trained language model outputs the target dimension perception feature vector. The generation module 300 is used to encode the historical trajectory data into a target dimension trajectory feature vector, encode the vehicle status data into a target dimension state feature vector, and generate the vehicle's target planned trajectory based on the perception feature vector, trajectory feature vector, and state feature vector.

[0241] In one embodiment of this application, the input module 200 is further configured to: extract multi-view visual images from visual perception data; project the multi-view visual images onto a target three-dimensional space; back-project the pixel positions and depths of the projected images in the target three-dimensional space onto a vehicle coordinate system; project the three-dimensional projection points of the vehicle coordinate system onto multiple mutually orthogonal two-dimensional feature planes constructed based on the vehicle coordinate system; and generate a visual word sequence of the target dimension based on the two-dimensional projection points on the multiple two-dimensional feature planes.

[0242] In one embodiment of this application, the trajectory planning device 10 of this application embodiment further includes a compression module.

[0243] The compression module is used to identify visual words at multiple time steps and from multiple perspectives in the visual word sequence before inputting the visual word sequence, text word sequence, and trajectory word sequence into the pre-trained language model; to perform temporal embedding and camera embedding on the visual words at multiple time steps and from multiple perspectives; to flatten the embedded visual words into the target input sequence; and to compress the target input sequence according to the pre-set query vector to obtain the compressed visual word sequence.

[0244] In one embodiment of this application, the input module 200 is further configured to: extract natural language instructions and scene description text from the query text; encode the natural language instructions and scene description text into a natural language text sequence; calculate the probability distribution of the sub-word sequence corresponding to the text segment in the natural language text sequence; map the natural language text sequence into a text word sequence according to the probability distribution; and map the text word sequence to the target dimension to obtain the text word sequence of the target dimension.

[0245] In one embodiment of this application, the input module 200 is further configured to: identify sub-word sequences in a text lexical sequence; obtain vector representations of sub-word sequences in a text lexical sequence; map the vector representations of sub-word sequences to text lexical units of the target dimension, and introduce positional encoding and semantic type encoding into the text lexical units of the target dimension; and generate a text lexical sequence of the target dimension based on the text lexical units of the target dimension, the positional encoding, and the semantic type encoding.

[0246] In one embodiment of this application, the input module 200 is further configured to: extract a path point sequence from historical trajectory data; calculate the motion increment of adjacent time steps based on the path points in the path point sequence; convert the path point sequence into a discrete trajectory word sequence based on the motion increment; and generate a trajectory word sequence of the target dimension based on the discrete trajectory word sequence.

[0247] In one embodiment of this application, the input module 200 is further configured to: obtain the vector representation of the trajectory words in the discrete trajectory word sequence; map the vector representation of the trajectory words to the trajectory words of the target dimension, and introduce time position encoding into the trajectory words of the target dimension; and generate a trajectory word sequence of the target dimension based on the trajectory words of the target dimension and the time position encoding.

[0248] In one embodiment of this application, the trajectory planning device 10 of this application embodiment further includes: a first training module.

[0249] The first training module is used to acquire historical vehicle operation data and extract visual perception data, query text, and historical trajectory data from the historical operation data before inputting visual perception data, query text, and historical trajectory data into the pre-trained visual language model. It also extracts multi-view visual images of at least one target time and reference visual images after the target time from the visual perception data, and extracts historical trajectories of at least one target time and reference trajectories after the target time from the historical trajectory data. The module generates a first training dataset based on the multi-view visual images, query text, historical trajectories, reference visual images, and reference trajectories. The first training dataset is then used to jointly train the visual language model and the pre-established diffuse world model to obtain the pre-trained visual language model.

[0250] In one embodiment of this application, the first training module is further configured to: input multi-view visual images, query text, and historical running trajectories from the first training dataset into a visual language model; output a perceptual feature vector through the visual language model; extract visual and action features from the perceptual feature vector; generate a target planning trajectory for the vehicle after the target time based on the perceptual feature vector; calculate a first loss of the visual language model based on the target planning trajectory and the reference running trajectory; generate a latent representation based on the reference visual image; generate noisy features based on the latent representation; input the noisy features, visual features, and action features into a diffuse world model; output a reconstructed visual image through the diffuse world model; calculate a second loss of the diffuse world model based on the reference visual image and the reconstructed visual image; calculate a joint loss based on the first and second losses; and update the model parameters of the pre-trained language model based on the joint loss until the pre-training of the visual language model is completed.

[0251] In one embodiment of this application, the loss function for the second loss is: ; in, This is the second loss. Let be the mathematical expectation, representing the expectation at... , , Take the average on the joint distribution. It is a noisy feature. For noise intensity, For the visual features at time step t, The motion characteristics at time step t, for The latent representation at time step, where t is the prediction step size. For parameters Conditional vector field function, The estimated vector field is predicted based on noisy features, noise intensity, visual features, and motion features. It is a real vector field.

[0252] In one embodiment of this application, the generation module 300 is further configured to: input the perception feature vector, trajectory feature vector, and state feature vector into a pre-trained trajectory generation model, and output a set of candidate planned trajectories for the vehicle through the trajectory generation model; calculate a first prediction score for a first candidate planned trajectory in the set of candidate planned trajectories, and select multiple first candidate planned trajectories for the vehicle from the set of candidate planned trajectories based on the first prediction score; calculate a second prediction score for the selected multiple first candidate planned trajectories, and determine the target planned trajectory for the vehicle from the selected multiple first candidate planned trajectories based on the second prediction score.

[0253] In one embodiment of this application, the trajectory planning device 10 of this application embodiment further includes: a second training module.

[0254] The second training module is used to obtain the vehicle's prior trajectory set and reference running trajectory before inputting the perception feature vector, trajectory feature vector, and state feature vector into the pre-trained trajectory generation model. It then generates a second candidate planning trajectory with random Gaussian noise and exploration noise based on the prior trajectory of the prior trajectory set. A second training dataset is generated based on the second candidate planning trajectory, perception feature vector, trajectory feature vector, state feature vector, and reference running trajectory. The trajectory generation model is then trained using reinforcement learning on the second training dataset. The training loss and imitation learning loss during the reinforcement learning process are obtained. The final loss is calculated based on the training loss and imitation learning loss, and the model parameters of the trajectory generation model are updated based on the final loss until the pre-training of the trajectory generation model is completed.

[0255] In one embodiment of this application, the second training module is further configured to: generate a normalized trajectory based on the prior trajectories of the prior trajectory set; add random horizontal Gaussian noise and vertical Gaussian noise to the normalized trajectory, and in the process of adding random Gaussian noise, truncate and diffuse the Gaussian noise based on a preset number of truncation and diffusion steps; and diffuse and add noise to the normalized trajectory after adding random Gaussian noise to generate a second candidate planning trajectory with random Gaussian noise and exploration noise.

[0256] In one embodiment of this application, the loss function for the final loss is: ; ; ; in, For the final loss, To mitigate training losses during the learning and training process, To imitate learning loss, To balance the weighting coefficients, G represents the total number of prior trajectories, and G represents the total number of candidate planned trajectories corresponding to a single prior trajectory. To truncate the total number of diffusion steps, d is the index of the diffusion denoising iteration step. This is a discount factor used to mitigate instability in the early denoising steps. Let Gaussian policy function be used. For the learnable parameters of the policy network, The output trajectory of the i-th candidate planned trajectory in the k-th trajectory group at the (d-1)-th diffusion step. For the k-th trajectory group, the i-th candidate planned trajectory is the noisy trajectory at the d-th diffusion step. The relative advantages of each trajectory group, Let be the reward estimate of the i-th candidate planning trajectory in the k-th candidate planning trajectory group after denoising. The mean of all candidate trajectories within a single candidate planning trajectory group, std is the standard deviation of all candidate planning trajectories within a single candidate planning trajectory group.

[0257] For a description of the features in the embodiment corresponding to the trajectory planning device, please refer to the relevant description of the embodiment corresponding to the trajectory planning method, which will not be repeated here.

[0258] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described trajectory planning method embodiments.

[0259] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described trajectory planning method embodiments at runtime.

[0260] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0261] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described trajectory planning method embodiments.

[0262] Any of the components, modules, units, parts, methods, and operations described in this application can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, central processing units, field-programmable gate arrays, application-specific integrated circuits, application-specific standard products, systems-on-a-chip (SoCs), complex programmable logic devices, microprocessors, etc. The terms "system," "computing device," or "apparatus" described in this application encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, APP, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0263] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0264] The trajectory planning method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A trajectory planning method, characterized in that, Includes the following steps: Acquire vehicle visual perception data, query text, historical trajectory data, and vehicle status data; The visual perception data, the query text, and the historical trajectory data are input into a pre-trained visual language model. The visual language model outputs a perceptual feature vector of the target dimension. The visual language model includes a visual encoder, a text encoder, a trajectory encoder, and a pre-trained language model. The visual encoder encodes the visual perception data into a visual word sequence of the target dimension; the text encoder encodes the query text into a text word sequence of the target dimension; and the trajectory encoder encodes the historical trajectory data into a trajectory word sequence of the target dimension. The visual word sequence, the text word sequence, and the trajectory word sequence are then input into the pre-trained language model. The language model outputs the perceptual feature vector of the target dimension through the pre-trained language model; before inputting the visual perception data, the query text, and the historical trajectory data into the pre-trained visual language model, the method further includes: acquiring the vehicle's historical operation data; extracting the visual perception data, the query text, and the historical trajectory data from the historical operation data; extracting at least one multi-view visual image of a target time and a reference visual image after the target time from the visual perception data; extracting at least one historical trajectory of a target time and a reference trajectory after the target time from the historical trajectory data; and based on the multi-view visual image, the query text, and the historical trajectory data... The trajectory, the reference visual image, and the reference running trajectory generate a first training dataset. The visual language model and a pre-established diffused world model are jointly trained using the first training dataset to obtain a pre-trained visual language model. The joint training of the visual language model and the pre-established diffused world model using the first training dataset to obtain the pre-trained visual language model includes: inputting the multi-view visual image, the query text, and the historical running trajectory from the first training dataset into the visual language model; outputting a perceptual feature vector through the visual language model; and extracting visual features and action features from the perceptual feature vector; based on the perceptual feature vector... The system generates a target planned trajectory for the vehicle after the target time, calculates a first loss for the visual language model based on the target planned trajectory and the reference running trajectory, generates a latent representation based on the reference visual image, generates noisy features based on the latent representation, inputs the noisy features, the visual features, and the action features into the diffusion world model, outputs a reconstructed visual image through the diffusion world model, calculates a second loss for the diffusion world model based on the reference visual image and the reconstructed visual image, calculates a joint loss based on the first loss and the second loss, and updates the model parameters of the pre-trained language model based on the joint loss until the pre-training of the visual language model is completed. The historical trajectory data is encoded into a trajectory feature vector of the target dimension, and the vehicle state data is encoded into a state feature vector of the target dimension. The target planned trajectory of the vehicle is generated based on the perception feature vector, the trajectory feature vector, and the state feature vector.

2. The trajectory planning method according to claim 1, characterized in that, The step of encoding the visual perception data into a visual word sequence of the target dimension using the visual encoder includes: Extract multi-view visual images from the visual perception data; The multi-view visual images are projected onto the target three-dimensional space; The pixel positions and depths of the projected image in the target three-dimensional space are back-projected to the vehicle coordinate system; The three-dimensional projection points of the vehicle coordinate system are projected onto multiple mutually orthogonal two-dimensional feature planes constructed based on the vehicle coordinate system, and a visual word sequence of the target dimension is generated based on the two-dimensional projection points on the multiple two-dimensional feature planes.

3. The trajectory planning method according to claim 1 or 2, characterized in that, Before inputting the visual lexical sequence, the text lexical sequence, and the trajectory lexical sequence into the pre-trained language model, the method further includes: Identify visual lexical units at multiple time steps and from multiple perspectives in the visual lexical sequence; Temporal and camera embeddings are performed on visual lexical units from multiple time steps and multiple viewpoints; The embedded visual lexical units are flattened into a target input sequence, and the target input sequence is compressed according to a pre-set query vector to obtain the compressed visual lexical unit sequence.

4. The trajectory planning method according to claim 1, characterized in that, The step of encoding the query text into a text word sequence of the target dimension using the text encoder includes: Extract natural language instructions and scene description text from the query text; The natural language instructions and scene description text are encoded into a natural language text sequence; Calculate the probability distribution of the sub-word sequences corresponding to text segments in the natural language text sequence; The natural language text sequence is mapped to the text lexical sequence according to the probability distribution, and the text lexical sequence is mapped to the target dimension to obtain the text lexical sequence of the target dimension.

5. The trajectory planning method according to claim 4, characterized in that, The step of mapping the text lexical sequence to the target dimension to obtain the text lexical sequence of the target dimension further includes: Identify the sub-word sequence in the text lexical sequence; Obtain the vector representation of the sub-word sequence in the text lexical sequence; The vector representation of the sub-word sequence is mapped to text units of the target dimension, and position encoding and semantic type encoding are introduced into the text units of the target dimension. The target dimension text lexicon sequence is generated based on the target dimension text lexicon, the position encoding, and the semantic type encoding.

6. The trajectory planning method according to claim 1, characterized in that, The step of encoding the historical trajectory data into a trajectory word sequence of the target dimension using the trajectory encoder includes: Extract the path point sequence from the historical trajectory data; Calculate the motion increment of adjacent time steps based on the path points in the path point sequence; The path point sequence is converted into a discrete trajectory word sequence based on the motion increment; Generate the target dimension trajectory lexical sequence based on the discrete trajectory lexical sequence.

7. The trajectory planning method according to claim 6, characterized in that, The step of generating the trajectory lexical sequence of the target dimension based on the discrete lexical sequence includes: Obtain the vector representation of the trajectory words in the discrete trajectory word sequence; The vector representation of the trajectory lexicon is mapped to the trajectory lexicon of the target dimension, and time position encoding is introduced into the trajectory lexicon of the target dimension. The trajectory word sequence of the target dimension is generated based on the trajectory words of the target dimension and the time position encoding.

8. The trajectory planning method according to claim 1, characterized in that, The loss function for the second loss is: ； in, This is the second loss. Let be the mathematical expectation, representing in , , Take the average on the joint distribution. It is a noisy feature. For noise intensity, For the visual features at time step t, The motion characteristics at time step t, for The latent representation at time step, where t is the prediction step size. For parameters Conditional vector field function, The estimated vector field is predicted based on noisy features, noise intensity, visual features, and motion features. It is a real vector field.

9. The trajectory planning method according to claim 1, characterized in that, The step of generating the target planned trajectory of the vehicle based on the perception feature vector, the trajectory feature vector, and the state feature vector includes: The perception feature vector, the trajectory feature vector, and the state feature vector are input into a pre-trained trajectory generation model, and the trajectory generation model outputs a set of candidate planned trajectories for the vehicle. Calculate the first prediction score of the first candidate planning trajectory in the candidate planning trajectory set, and select multiple first candidate planning trajectories of the vehicle from the candidate planning trajectory set based on the first prediction score; Calculate the second prediction score of the selected multiple first candidate planning trajectories, and determine the target planning trajectory of the vehicle from the selected multiple first candidate planning trajectories based on the second prediction score.

10. The trajectory planning method according to claim 9, characterized in that, Before inputting the perception feature vector, the trajectory feature vector, and the state feature vector into the pre-trained trajectory generation model, the method further includes: Obtain the vehicle's prior trajectory set and reference running trajectory; A second candidate planning trajectory with random Gaussian noise and exploration noise is generated based on the prior trajectory of the prior trajectory set. A second training dataset is generated based on the second candidate planning trajectory, the perception feature vector, the trajectory feature vector, the state feature vector, and the reference running trajectory. The trajectory generation model is trained using reinforcement learning on the second training dataset. The training loss and imitation learning loss during the reinforcement learning training process are obtained. The final loss is calculated based on the training loss and the imitation learning loss. The model parameters of the trajectory generation model are updated based on the final loss until the pre-training of the trajectory generation model is completed.

11. The trajectory planning method according to claim 10, characterized in that, The step of generating a second candidate planning trajectory with random Gaussian noise and exploration noise based on the prior trajectories of the prior trajectory set includes: A normalized trajectory is generated based on the prior trajectories in the prior trajectory set; Random horizontal Gaussian noise and vertical Gaussian noise are added to the normalized trajectory. During the process of adding random Gaussian noise, the Gaussian noise is truncated and diffused based on a preset number of truncation and diffusion steps. The normalized trajectory with added random Gaussian noise is diffused to generate a second candidate planning trajectory with random Gaussian noise and exploration noise.

12. The trajectory planning method according to claim 10, characterized in that, The loss function for the final loss is: ； ； ； in, For the final loss, To mitigate training losses during the learning and training process, To imitate learning loss, To balance the weighting coefficients, G represents the total number of prior trajectories, and G represents the total number of candidate planned trajectories corresponding to a single prior trajectory. To truncate the total number of diffusion steps, d is the index of the diffusion denoising iteration step. This is a discount factor used to mitigate instability in the early denoising steps. Let Gaussian policy function be used. For the learnable parameters of the policy network, The output trajectory of the i-th candidate planned trajectory in the k-th trajectory group at the (d-1)-th diffusion step. For the k-th trajectory group, the i-th candidate planned trajectory is the noisy trajectory at the d-th diffusion step. The relative advantages of each trajectory group, Let be the reward estimate of the i-th candidate planning trajectory in the k-th candidate planning trajectory group after denoising. The mean of candidate trajectories within a single candidate trajectory group, std is the standard deviation of candidate trajectories within a single candidate trajectory group.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the trajectory planning method as described in any one of claims 1 to 12.

Images