Trajectory planning, model training method and device, autonomous driving system and medium

Abstract

Embodiments of the present application provide a trajectory planning method and device, model training method and device, automatic driving system and medium, and relate to the technical field of automatic driving. The method comprises: obtaining driving perception data of a vehicle in a current scene; using world knowledge built in a large language model, taking lane lines in a future scene associated with the driving perception data as static physical constraints, and taking traffic participants in a drivable area of the vehicle defined by the lane lines and associated with the driving perception data as dynamic physical constraints, reasoning the driving perception data to obtain a future scene image; and using the large language model, taking a space-time thinking chain as an intermediate reasoning carrier, reasoning the driving perception data and the future scene image to obtain a trajectory sequence of the vehicle. The scheme establishes a causal relationship of the physical world, improves the reasoning accuracy of the trajectory sequence, covers long-tail scenes, and improves the safety of automatic driving.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a trajectory planning, model training method, device, autonomous driving system and medium. Background Technology

[0002] In autonomous driving scenarios, the system uses data collected by sensors such as cameras and LiDAR to identify various objects on the road and label their positions, sizes, and relative relationships, mapping the three-dimensional world into a two-dimensional bird's-eye view (BEV). This BEV is then used to predict the vehicle's trajectory sequence, thereby controlling the vehicle's movement along that sequence. However, this approach, while capturing static spatial information, struggles to establish causal relationships in the physical world and is insufficient for covering long-tail scenarios. Summary of the Invention

[0003] The purpose of this application is to provide a trajectory planning, model training method, device, autonomous driving system, and medium to establish causal relationships in the physical world, improve the inference accuracy of trajectory sequences, cover long-tail scenarios, and enhance the safety of autonomous driving. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a trajectory planning method, the method comprising: acquiring driving perception data of a vehicle in the current scene; using the world knowledge built into a large language model, taking lane lines in a future scene associated with the driving perception data as static physical constraints, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, reasoning on the driving perception data to obtain a future scene image; and using the large language model, with a spatiotemporal thought chain as an intermediate reasoning carrier, reasoning on the driving perception data and the future scene image to obtain a trajectory sequence of the vehicle.

[0005] In some embodiments, the step of using the world knowledge built into the large language model, taking lane lines in the future scene associated with the driving perception data as static physical constraints, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, to infer the driving perception data and obtain a future scene image includes: using the world knowledge built into the large language model to infer the driving perception data, sequentially obtaining a first visual marker of the lane lines in the future scene and a second visual marker of traffic participants within the drivable area defined by the first visual marker; using the world knowledge built into the large language model, taking the first visual marker as a static physical constraint and the second visual marker as a dynamic physical constraint, to infer the driving perception data and obtain a third visual marker of the future scene; and using the first visual marker, the second visual marker, and the third visual marker to generate a future scene image.

[0006] In some embodiments, the step of using the world knowledge built into the large language model, taking lane lines in the future scene associated with the driving perception data as static physical constraints, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, to infer the driving perception data and obtain a future scene image includes: obtaining a unified future frame corresponding to the future scene, wherein the unified future frame is generated using a first visual marker and a second visual marker, wherein the first visual marker is a visual marker of lane lines in the future scene obtained by inferring historical driving perception data using the world knowledge built into the large language model, the first visual marker defines the drivable area of ​​the vehicle, and the second visual marker is a visual marker of traffic participants within the drivable area of ​​the vehicle obtained by inferring historical driving perception data using the world knowledge built into the large language model; using the world knowledge built into the large language model, taking lane lines in the unified future frame as static physical constraints, and traffic participants in the unified future frame as dynamic physical constraints, to infer the driving perception data and obtain a third visual marker of the future scene; and generating a future scene image using the first visual marker, the second visual marker, and the third visual marker.

[0007] In some embodiments, the step of generating a future scene image using the first visual marker, the second visual marker, and the third visual marker includes: determining a structural constraint framework for the future scene image using the first visual marker and the second visual marker; and filling the structural constraint framework with fine-grained information indicated by the third visual marker to obtain the future scene image.

[0008] In some embodiments, the text codebook of the large language model includes visual tags generated by encoding image data of a driving scene using a vector quantization variational encoder.

[0009] Secondly, embodiments of this application provide a model training method, the method comprising: acquiring an image dataset for autonomous driving and instruction questions corresponding to each frame of image data in the image dataset; using the world knowledge built into a large language model to reason about each frame of image data and the corresponding instruction questions in the image dataset, thereby obtaining a first visual marker of lane lines in the future scene corresponding to each frame of image data, and a second visual marker of traffic participants within the drivable area defined by the first visual marker; using the world knowledge built into the large language model, with the first visual marker corresponding to each frame of image data as a static physical constraint and the second visual marker corresponding to each frame of image data as a dynamic physical constraint, to reason about each frame of image data and the corresponding instruction questions, thereby obtaining a third visual marker of the future scene corresponding to each frame of image data; using the first visual marker, the second visual marker, and the third visual marker corresponding to each frame of image data to generate a predicted future scene image corresponding to each frame of image data; and using the predicted future scene image corresponding to each frame of image data and the future scene images in the image dataset corresponding to each frame of image data to iteratively train the large language model.

[0010] In some embodiments, the step of generating a predicted future scene image corresponding to each frame of image data using the first visual marker, the second visual marker, and the third visual marker corresponding to each frame of image data includes: determining a structural constraint framework for the future scene image data corresponding to each frame of image data using the first visual marker and the second visual marker corresponding to each frame of image data; and filling the structural constraint framework corresponding to each frame of image data with fine-grained information indicated by the third visual marker corresponding to each frame of image data to obtain the predicted future scene image corresponding to each frame of image data.

[0011] In some embodiments, the text codebook of the large language model includes visual tags generated by encoding image data of a driving scene using a vector quantization variational encoder.

[0012] Thirdly, embodiments of this application provide a trajectory planning device, the device comprising: an acquisition module for acquiring driving perception data of a vehicle in the current scene; a first reasoning module for using the world knowledge built into a large language model, taking lane lines in the future scene associated with the driving perception data as static physical constraints, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, to reason about the driving perception data and obtain a future scene image; and a second reasoning module for using a large language model, with a spatiotemporal thought chain as an intermediate reasoning carrier, to reason about the driving perception data and the future scene image to obtain a trajectory sequence of the vehicle.

[0013] Fourthly, embodiments of this application provide a model training apparatus, the apparatus comprising: an acquisition module, configured to acquire an image dataset for autonomous driving and instruction questions corresponding to each frame of image data in the image dataset; a first inference module, configured to use the world knowledge built into a large language model to infer each frame of image data and the corresponding instruction question in the image dataset, sequentially obtaining a first visual marker of lane lines in the future scene corresponding to each frame of image data, and a second visual marker of traffic participants within the drivable area defined by the first visual marker; a second inference module, configured to use the world knowledge built into the large language model, with the first visual marker corresponding to each frame of image data as a static physical constraint and the second visual marker corresponding to each frame of image data as a dynamic physical constraint, to infer each frame of image data and the corresponding instruction question, obtaining a third visual marker of the future scene corresponding to each frame of image data; a third inference module, configured to use the first visual marker, the second visual marker, and the third visual marker corresponding to each frame of image data to generate a predicted future scene image corresponding to each frame of image data; and a training module, configured to use the predicted future scene image corresponding to each frame of image data and the future scene images in the image dataset corresponding to each frame of image data to iteratively train the large language model.

[0014] Fifthly, embodiments of this application provide an autonomous driving system, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the methods provided in the first aspect or any of the methods provided in the second aspect when executing the program stored in the memory.

[0015] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods provided in the first aspect or any of the methods provided in the second aspect.

[0016] In a seventh aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods provided in the first aspect, or to perform any of the methods provided in the second aspect.

[0017] Beneficial effects of the embodiments in this application:

[0018] In the technical solution provided in this application, the large language module has built-in world knowledge, enabling it to understand dynamic change patterns. The large language model utilizes this built-in world knowledge to infer driving perception data of the vehicle in the current scene, obtaining images of future scenes. Then, using a spatiotemporal thought chain as an intermediate reasoning carrier, the large language model further infers from the driving perception data and future scene images to obtain the vehicle's trajectory sequence. This reasoning process more closely resembles the simulation and deduction of the physical world, enabling it to perceive the dynamic change patterns of the physical world, establish correct causal relationships in the physical world, and improve the accuracy of trajectory sequence reasoning. Furthermore, the large language model, utilizing its built-in world knowledge, can autonomously generate simulated long-tail scenarios, improving the reliability of decision-making and enhancing the safety of autonomous driving.

[0019] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0021] Figure 1 A schematic diagram of the trajectory planning method provided in the embodiments of this application;

[0022] Figure 2 This is a first schematic diagram of the model reasoning process provided in an embodiment of this application;

[0023] Figure 3 This is a second schematic diagram of the model reasoning process provided in the embodiments of this application;

[0024] Figure 4 A schematic diagram of a model training method provided in an embodiment of this application;

[0025] Figure 5 A schematic diagram of a model training scenario provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram comparing different CoT values ​​provided in the embodiments of this application;

[0027] Figure 7 A schematic diagram of a trajectory planning device provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of a model training apparatus provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of an autonomous driving system provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0031] The following is an explanation of the terms used in the embodiments of this application.

[0032] Rule-driven autonomous driving: This first-generation rule-driven architecture relies on engineers manually writing massive amounts of driving rules to define the execution logic for scenarios such as braking and lane changing. This method can still function effectively on structured roads. However, given the complexity and variability of real-world traffic, limited rules cannot cover the infinite number of scenario exceptions, resulting in insufficient flexibility and limited applicability.

[0033] End-to-end autonomous driving: This second-generation end-to-end architecture breaks away from rule dependence. By feeding the model with large-scale real driving data, it enables the model to learn driving strategies autonomously, successfully solving the problem of exhaustive rule enumeration. However, the model in this autonomous driving method is essentially a "data pattern replicator," capable of making decisions only for scenarios that have appeared in the training data. Once it encounters long-tail scenarios that are not covered, it is very prone to decision failure and falls into the dilemma of "data dependence."

[0034] Vision Language Action (VLA)-based autonomous driving: The third-generation VLA architecture, which integrates vision, language, and action, endows the system with cross-task understanding and generalization capabilities, freeing it from being limited to predictions within a single intelligent driving scenario. However, the VLA architecture still has a fatal flaw: its understanding of the world remains at a static level. It can accurately identify elements such as pedestrians and vehicles in the current scene, but it cannot effectively infer the dynamic changes of these elements. For example, the VLA architecture can recognize a person standing on the roadside, but it cannot predict whether they will suddenly cross the road; it can capture the driving status of vehicles ahead, but it cannot accurately infer their next lane change or braking intention.

[0035] World Model for Autonomous Driving: A core model used in autonomous driving systems to model, predict, and infer environmental dynamics. The world model learns from historical perception data such as images and LiDAR to construct representations of the physical world, traffic environment, and its own state. This allows it to predict future environmental changes, assess the consequences of different actions, and support decision-making and planning. The world model can transform sensor observations into structured environmental states, such as bird's-eye view (BEV), three-dimensional (3D) targets, lane lines, and traffic rules. Based on the current state and candidate actions, it predicts environmental changes such as the movement of vehicles, pedestrians, and obstacles, lane states, and traffic light changes over the next few seconds or even tens of seconds. The world model understands the causal relationship between actions and consequences, assessing the safety risks and environmental impacts of different driving decisions.

[0036] The VLA model for autonomous driving is an end-to-end intelligent model that integrates visual perception, language understanding, and action decision-making capabilities. The core objective of the VLA model is to directly map multimodal inputs (such as camera images, laser point cloud maps, navigation commands, and traffic rule texts) into control actions of autonomous vehicles (such as trajectory, speed, steering, and braking). The VLA model simultaneously possesses scene understanding, logical reasoning, and forward planning capabilities, representing a core paradigm for the transformation of autonomous driving from a "modular pipeline" to "end-to-end intelligence." By sharing a latent space, it uses image features, text commands, and historical action encodings as a unified representation, avoiding cross-module information loss and addressing the pain point of the traditional architecture's separation of "perception-prediction-planning." Leveraging the world knowledge and logical reasoning capabilities of a large language model, it understands fuzzy navigation commands, traffic rule texts, and complex scene descriptions, enabling interpretable decision-making.

[0037] The VLA model and the world model are collaborative partners with clearly defined roles. The VLA model focuses on the real-time decision-making loop on the vehicle side, ensuring timely responses during the driving process with low latency and high reliability; the world model undertakes the responsibility of cognition and training in the cloud, optimizing driving strategies by generating simulation scenarios and mining data value. The vehicle side seeks "speed," while the cloud side seeks "depth." The two perform their respective duties and work together to form a complete system of physical intelligent agents for autonomous driving.

[0038] End-to-end trajectory planning: The core logic of end-to-end autonomous driving is to directly generate future driving trajectories from sensor data and convert these trajectories into control commands that the vehicle can execute, such as acceleration or steering. In this embodiment, the method used is to directly acquire image P from the image sensor, such as acquiring a set of k surround view images at time i. , This means that if image j is acquired at time i, where j=1,2,…,k, the model will output a trajectory sequence in BEV format. Each trajectory point t=1,2,…,n, where n represents the number of future trajectory points. This represents the horizontal coordinate at time t in the future. Let P represent the longitudinal coordinates at time t in the future. The mapping process from image P to trajectory sequence can be formalized as follows: , Represents a trajectory sequence, O() represents a large model, Represents a set of images in a panoramic view. The table can be used to input information, including navigation instructions. and vehicle status parameters In this embodiment, the large model takes a panoramic image and task instructions as input, outputs future frames as a spatiotemporal thought chain (CoT), and outputs a trajectory sequence.

[0039] Multimodal Understanding and Integrated Generation Mechanism: This application integrates multimodal understanding and visual generation functions into a large model. The training objective of the multimodal understanding task is consistent with that of the standard large language model. The integrated generation task can be implemented using a dependency vector quantization variational autoencoder to convert images into discrete visual labels. Specifically, the process involves quantizing an A×B×N dimension image pixel matrix into an a×b dimension discrete visual label matrix using an image editor, where a=A / w, b=B / w, and w is the downsampling coefficient. These a×b discrete visual labels are then arranged according to a raster scan sequence and used to train a Transformer-based autoregressive large model. In the image generation stage, a general language modeling objective function can be used to generate subsequent visual labels one by one through autoregression, maximizing the probability of each visual label's occurrence.

[0040]

[0041] Where Ω represents the total loss, the smaller the Ω, the more accurate the large language model is in predicting visual label sequences, q n This represents the visual marker at index n. Let represent all visual tags with indices less than n, and △ be the set of parameters for the large language model. This indicates that, given all previous visual markers, the current position takes the value q. n The probability of the large language model. The large language model uses a decoder derived from a vector quantization variational autoencoder to restore the generated discrete visual tags to image pixel information, i.e., to the image itself.

[0042] Autonomous driving technology has undergone three key iterations: rule-driven, end-to-end, and VLA-based. Each iteration has addressed the core pain points of the previous generation but has also exposed new problems. Current autonomous driving technology suffers from the following issues:

[0043] (1) The model cannot understand physical causality.

[0044] The core problem that traditional autonomous driving perception systems address is the "spatial problem": using sensors such as cameras and LiDAR to identify various objects in the road, labeling their positions, sizes, and relative relationships, ultimately mapping the three-dimensional world to a two-dimensional BEV (Battery Electric Vehicle). The fatal flaw of this approach is the loss of the "temporal dimension"—it can only capture static spatial information and cannot perceive dynamic trends. This makes it difficult for large language models to establish causal relationships in the physical world.

[0045] (2) Large language models cannot achieve the transition from "being taught" to "learning by oneself".

[0046] The core challenge of autonomous driving has never been handling common scenarios, but rather covering long-tail scenarios (i.e., extreme scenarios). Long-tail scenarios have a low probability of occurrence, but once they happen, they are extremely dangerous. Relying on real-world road tests to collect training data for large language models is not only extremely costly, but some extreme scenarios cannot even be actively generated.

[0047] (3) The cost of “imagining the future” cannot be reduced to the point where it is available on the vehicle side.

[0048] The dynamic understanding of the world by the large language model is only the first step. How to efficiently utilize this dynamic understanding capability on the vehicle side, where technical resources are limited, is another engineering challenge. The computational cost of the world model during the deduction process is extremely high, making it almost impossible to run directly on the vehicle side.

[0049] To address the above problems, embodiments of this application provide a trajectory planning method, such as... Figure 1 As shown, the method includes the following steps:

[0050] Step S101: Obtain driving perception data of the vehicle in the current scene;

[0051] Step S102: Using the world knowledge built into the large language model, the lane lines in the future scene associated with the driving perception data are used as static physical constraints, and the traffic participants associated with the driving perception data in the vehicle drivable area defined by the lane lines are used as dynamic physical constraints to infer the driving perception data and obtain the future scene image.

[0052] Step S103: Using a large language model and a spatiotemporal thought chain as an intermediate reasoning carrier, reason about driving perception data and future scene images to obtain the vehicle's trajectory sequence.

[0053] In the technical solution provided in this application, the large language module has built-in world knowledge, enabling it to understand dynamic change patterns. The large language model utilizes this built-in world knowledge to infer driving perception data of the vehicle in the current scene, obtaining images of future scenes. Then, using a spatiotemporal thought chain as an intermediate reasoning carrier, the large language model further infers from the driving perception data and future scene images to obtain the vehicle's trajectory sequence. This reasoning process more closely resembles the simulation and deduction of the physical world, enabling it to perceive the dynamic change patterns of the physical world, establish correct causal relationships in the physical world, and improve the accuracy of trajectory sequence reasoning. Furthermore, the large language model, utilizing its built-in world knowledge, can autonomously generate simulated long-tail scenarios, improving the reliability of decision-making and enhancing the safety of autonomous driving.

[0054] The trajectory planning method provided in this application is applied to an autonomous driving system on a vehicle, which deploys a large language model. This large language model can be a VLA model or a Vision Language Model (VLM) model. The large language model contains built-in world knowledge, meaning it can serve as a world model.

[0055] In step S101 above, the autonomous driving system may include various vehicle sensors, such as cameras, LiDAR, vehicle speed sensors, and voice modules. Driving perception data may include data collected by various vehicle sensors, such as surround view images collected by cameras, radar data collected by LiDAR, vehicle speed data collected by vehicle speed sensors, and voice data collected by the voice module. During vehicle operation, various vehicle sensors collect data in real time, and the current spatiotemporal data collected by these sensors constitutes the vehicle's driving perception data for the current scenario.

[0056] In step S102 above, the autonomous driving system may further include a large language model, which can process multimodal driving perception data; that is, the large language model is a multimodal large model. The large language model has built-in world knowledge, can perceive the dynamic changes in the physical world, and has visual generation capabilities. For example, the large language model may have a vector quantization variational encoder, which can restore the predicted visual label sequence to image pixels. Traffic participants include, but are not limited to, other vehicles, pedestrians, and obstacles.

[0057] After acquiring driving perception data of the current scene, the autonomous driving system inputs this data into a large language model. The large language model can then obtain lane lines in the future scene associated with the driving perception data, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines. Thus, the large language model acts as a world model, utilizing built-in road structure common sense, traffic rule constraints, and other world knowledge, guided by both static and dynamic physical constraints, to infer from the driving perception data and predict a complete image of the future scene.

[0058] In step S103 above, the future scene image can represent the temporal relationships of the future world, and the dynamic change process of the visual content can intuitively reflect the temporal progression and the inherent logic of scene development.

[0059] After obtaining images of future scenes, the large language model uses spatiotemporal CoT as an intermediate inference carrier, enabling it to function as an inverse dynamics model. Based on current observation data (i.e., driving perception data) and future prediction results (i.e., images of future scenes), it completes trajectory planning and obtains the vehicle's trajectory sequence.

[0060] The probabilistic model for predicting trajectory sequences using a large language model can be expressed as:

[0061]

[0062] Where Ω represents the total probability; the larger the Ω, the more accurate the large language model's prediction of the trajectory sequence. R represents the trajectory sequence. , Let n represent the trajectory points, n=1,2,…,N, where N represents the number of trajectory points included in the trajectory sequence, and P represents the set of toroidal images. , Representing image i, Represents spacetime CoT, The table can be used to input information, including navigation instructions. and vehicle status parameters △ represents the parameter set of the large language model. This represents conditional probability.

[0063] In this embodiment, the large language model takes a surround view image and task instructions as input and outputs a predicted trajectory sequence. After obtaining the vehicle's trajectory sequence, the autonomous driving system (such as the large language model) can map the trajectory sequence into control actions to control the vehicle's movement.

[0064] In some embodiments, such as Figure 2 As shown, step S102 above may include:

[0065] Step S201: Using the world knowledge built into the large language model, reason about the driving perception data to obtain the first visual marker of the lane line in the future scene and the second visual marker of traffic participants in the vehicle driving area defined by the first visual marker.

[0066] Step S202: Using the world knowledge built into the large language model, the first visual label is used as a static physical constraint and the second visual label is used as a dynamic physical constraint to reason about the driving perception data and obtain the third visual label of the future scene.

[0067] Step S203: Generate a future scene image using the first visual marker, the second visual marker, and the third visual marker.

[0068] In the technical solution provided in this application embodiment, the large language model is explicitly guided to obtain detailed information of the complete future scene, namely the third visual marker, which fundamentally solves the problem of physical law violation that may occur when directly generating future frames, and improves the accuracy of the predicted future scene image.

[0069] In step S201 above, the large language model can map driving perception data to a latent space. Utilizing built-in road structure common sense, traffic rule constraints, and other world knowledge, it infers from the driving perception data within the latent space to obtain visual markings of lane lines in the future scene, such as the first visual marking M. L The first visual marker of the lane line, M. L It can impose static physical constraints on large language models, such as ensuring that vehicle trajectories do not exceed lane lines, and avoid generating scenarios that conflict with road topology.

[0070] Lane lines, as the core structural framework of a road scene, clearly define the boundaries of the drivable area for vehicles. The first visual marker M of the lane lines is obtained through inference. L Subsequently, based on the first visual marker M L The large language model utilizes built-in world knowledge such as road structure common sense and traffic rule constraints to continue reasoning on driving perception data in the latent space, obtaining the visual labels of traffic participants within the vehicle's drivable area, i.e., the second visual label M. D .

[0071] In this embodiment of the application, the second visual marker M D The visual label is a multi-dimensional bounding box. A multi-dimensional bounding box not only accurately preserves the spatial location information of the target but also implicitly contains the target's motion characteristics, such as the trajectory of a vehicle ahead or the direction a pedestrian crosses the road. The second visual label is M. D It can impose dynamic physical constraints on large language models, such as preventing collisions between different targets and ensuring that the direction of target motion conforms to the law of physical inertia.

[0072] The latent space is an abstract, compressed representation within the model. It is not the raw perceptual data such as images or point clouds from the real world, but rather a low-dimensional abstract vector that has been encoded. Extrapolating the future in the latent space is computationally efficient compared to the raw perceptual data level.

[0073] In step S202 above, after obtaining the first visual marker M L Second visual marker M D Subsequently, the large language model utilizes its built-in knowledge of road structure, traffic rules, and other world knowledge to label M using a first-person visual representation. L For static physical constraints, M is marked with a second visual symbol. D Guided by both static and dynamic physical constraints, and using driving perception data as a dynamic physical constraint, inference is performed to obtain a complete visual label of the future scene, namely the third visual label M. F This progressive generation process can be represented using mathematical expressions.

[0074] M L →M D →M F st C static (M L )∧C dynamic (M D M L )

[0075] Among them, C static (M L C represents the static physical constraints related to the rationality of lane topology. dynamic (M D M L ) represents the dynamic physical constraints related to the target motion and spatial compatibility.

[0076] In this embodiment, lane lines and multi-dimensional bounding boxes, these coarse-grained visual cues guide the large language model to focus on the drivable area and key target objects in the future scene, while constraining the generated results to meet physical plausibility requirements. Furthermore, in this embodiment, the large language model focuses on traffic participants within the lane lines and the drivable area defined by the lane lines, rather than reasoning about all targets in the driving scene. This reduces the computational cost of the large language model, enabling the technical solution provided in this embodiment to be applied to vehicles with limited computing power.

[0077] In step S203 above, the large language model can use a vector quantization variational encoder to decode the first visual marker, the second visual marker, and the third visual marker to generate a future scene image. This future scene image includes spatial relationships of the future world such as lane lines and multi-dimensional bounding boxes, as well as temporal relationships of the future world such as conventional future images (i.e., detailed information corresponding to the third visual marker).

[0078] Through the ordered generation sequence of steps S201 to S203, the large language model is explicitly guided to obtain scene information of the complete future scene, thus fundamentally solving the problem of physical law violations that may occur when directly generating future frames.

[0079] In some embodiments, step S203 may be: using a first visual marker, a second visual marker, and a third visual marker to determine a structural constraint framework for a future scene image; filling the structural constraint framework with fine-grained information to obtain a future scene image.

[0080] Large language models can utilize vector quantization variational encoders to reconstruct the first and second visual markers into image pixels, obtaining a structural constraint framework for the future scene image. This structural constraint framework can also be called the future unified frame. After obtaining the structural constraint framework of the future scene image, the large language model does not need to build the scene from scratch. It fills the structural constraint framework with fine-grained information such as road texture, target appearance details, and lighting conditions indicated by the third visual markers, thus obtaining the future scene image.

[0081] In this embodiment, the large language model first parses the scene structure information and then fills in the details. This progressive generation strategy from coarse to fine solves the problem of physical law violation that may occur when directly generating future frames from the root, ensuring the physical rationality of the generated scene and improving the realism of the image.

[0082] In this embodiment of the application, the large language model can use a vector quantization variational encoder to restore the third visual markers to image pixels, and the resulting structural constraint framework can be directly used as the future scene image without limitation.

[0083] In some embodiments, such as Figure 3 As shown, step S102 above may include:

[0084] Step S301: Obtain the future unified frame corresponding to the future scene. The future unified frame is generated using the first visual marker and the second visual marker. The first visual marker is the visual marker of the lane line in the future scene obtained by reasoning from historical driving perception data using the world knowledge built into the large language model. The first visual marker defines the vehicle driving area. The second visual marker is the visual marker of traffic participants in the vehicle driving area obtained by reasoning from historical driving perception data using the world knowledge built into the large language model.

[0085] Step S302: Using the world knowledge built into the large language model, the lane lines in the future unified frame are used as static physical constraints, and the traffic participants in the future unified frame are used as dynamic physical constraints to reason about the driving perception data and obtain the third visual label of the future scene.

[0086] Step S303: Generate a future scene image using the first visual marker, the second visual marker, and the third visual marker.

[0087] In the technical solution provided in this application embodiment, the large language model directly obtains the future unified frame, uses the lane lines and multi-dimensional bounding boxes in the future unified frame to represent the spatial relationship of the future world, constructs static physical constraints and dynamic physical constraints to generate the future scene map, without the need to generate lane lines and multi-dimensional bounding boxes separately, saving computational costs and improving model inference efficiency.

[0088] In step S301 above, the future unified frame can adopt... Figure 2 The results are obtained through steps S201-S203. The future unified frame is used to predict the future world state. The large language model employs... Figure 2 Within a preset time period after obtaining the future unified frame in steps S201 to S203, the future unified frame can be directly acquired without the need to separately generate lane lines and multi-dimensional detection boxes. The preset time period can be set according to actual needs, such as 1 minute, 5 minutes, 10 minutes, etc.

[0089] In step S302 above, the future unified frame includes future lane lines and multi-dimensional bounding boxes, etc. These coarse-grained visual cues can guide the large language model to focus on the drivable area of ​​the vehicle and key target objects in the future scene, while constraining the generated results to meet the physical rationality requirements.

[0090] The large language model utilizes built-in common sense about road structures, traffic rule constraints, and other world knowledge to unify lane lines (such as first visual markers M) in future frames. L ) as static physical constraints, with traffic participants (such as second visual markers M) in the future unified frame. DThe dynamic physical constraints, guided by both static and dynamic physical constraints, allow for reasoning based on the currently acquired driving perception data to obtain a complete visual label of the future scene, namely the third visual label M. F Based on the first visual marker M L Second visual marker M D Third visual marker M F Proceed to step S303. Step S303 is the same as step S203 above, and will not be described again here.

[0091] In some embodiments, the text codebook of the large language model includes visual tags generated by encoding image data of the driving scene using a vector quantization variational encoder.

[0092] In this embodiment, a vector quantization variational encoder encodes image data of a driving scene to obtain visual tags, and integrates these visual tags into the text codebook of a large language model, expanding the model's vocabulary coverage and extending the vocabulary space from a purely linguistic domain to a multimodal space encompassing both visual and textual modalities. This improvement enables the multimodal large language model to predict visual tags.

[0093] Corresponding to the trajectory planning method described above, this application also provides a model training method, such as... Figure 4 As shown, the method includes the following steps:

[0094] Step S401: Obtain the image dataset for autonomous driving and the instruction problem corresponding to each frame of image data in the image dataset;

[0095] Step S402: Using the world knowledge built into the large language model, reason about each frame of image data and the corresponding instruction problem in the image dataset, and sequentially obtain the first visual marker of the lane line in the future scene corresponding to each frame of image data, and the second visual marker of traffic participants in the vehicle driving area defined by the first visual marker.

[0096] Step S403: Using the world knowledge built into the large language model, the first visual label corresponding to each frame of image data is used as a static physical constraint, and the second visual label corresponding to each frame of image data is used as a dynamic physical constraint. Reasoning is performed on each frame of image data and the corresponding instruction problem to obtain the third visual label of the future scene corresponding to each frame of image data.

[0097] Step S404: Using the first visual marker, second visual marker and third visual marker corresponding to each frame of image data, generate a predicted future scene image corresponding to each frame of image data;

[0098] Step S405: Use the predicted future scene image corresponding to each frame of image data and the future scene image in the image dataset corresponding to each frame of image data to iteratively train the large language model.

[0099] In the technical solution provided in this application, the large language module has built-in world knowledge, enabling it to understand dynamic change patterns. The large language model utilizes this built-in world knowledge to infer driving perception data of the vehicle in the current scene, obtaining images of future scenes. Then, using a spatiotemporal thought chain as an intermediate reasoning carrier, the large language model further infers from the driving perception data and future scene images to obtain the vehicle's trajectory sequence. This reasoning process more closely resembles the simulation and deduction of the physical world, enabling it to perceive the dynamic change patterns of the physical world, establish correct causal relationships in the physical world, and improve the accuracy of trajectory sequence reasoning. Furthermore, the large language model, utilizing its built-in world knowledge, can autonomously generate simulated long-tail scenarios, improving the reliability of decision-making and enhancing the safety of autonomous driving.

[0100] Furthermore, in this embodiment, the large language model first parses the scene structure information and then fills in the details. This progressive generation strategy from coarse to fine solves the problem of physical law violations that may occur when directly generating future frames from the root, ensuring the physical rationality of the generated scene and improving the realism of the image, while also enhancing the model's scene understanding ability.

[0101] In step S401 above, the image dataset can be real driving video data or data taken from the RealDriveSim dataset. The RealDriveSim dataset is a Graph Visual Question Answering (GVQA) dataset, which can include one or more frames of images and corresponding instruction questions. In the image dataset, each frame of image data is labeled with lane lines and multi-dimensional bounding boxes. Instruction questions can include, but are not limited to, "What action should I take in relation to the vehicle in front?", "Please generate the future lane divider and the speed of the vehicle in front", "Please generate a spatial multi-dimensional detection box graph", and "Please predict what the next frame will be?". The answers to the instruction questions are control commands, lane line images, multi-dimensional bounding box images, and future frame images (i.e., the next frame image). One frame of image and its corresponding instruction question constitute a training sample, which can be represented as (P, L), where P represents the set of surround view images, and L represents the instruction question corresponding to the set of surround view images. The large language model processes the training samples and generates the corresponding response A (such as a future scene image), which can be represented by the expression A=M(P,L), where M() represents the large language model.

[0102] In step S402 above, the large language model possesses a visual Transformer-based encoder, capable of converting image data into continuous features. For each frame of image data in the image dataset, the autonomous driving system inputs the frame of image data and the corresponding instruction question into the large language model. The large language model can map driving perception data to a latent space, and using its built-in world knowledge, it infers the image data and the corresponding instruction question within the latent space to obtain the first visual label of the lane line in the future scene corresponding to the image data. After obtaining the first visual label of the lane line, based on the first visual label, the large language model continues to infer the image data within the latent space using its built-in world knowledge to obtain the second visual label of traffic participants within the vehicle's drivable area defined by the first visual label of the image data.

[0103] In step S403 above, for each frame of image data in the image dataset, after obtaining the first visual label and the second visual label corresponding to the image data, the large language model uses its built-in world knowledge, takes the first visual label corresponding to the image data as a static physical constraint, and takes the second visual label corresponding to the image data as a dynamic physical constraint, and performs reasoning on the image data and the corresponding instruction problem in the latent space to obtain the third visual label of the future scene corresponding to the image data.

[0104] In step S404 above, the large language model can use a vector quantization variational encoder to restore the first visual marker, second visual marker, and third visual marker to image pixels, thereby obtaining a predicted future scene image. That is, the autonomous driving system can generate a structural constraint framework for the future scene, and fill this framework with fine-grained information such as road texture, target appearance details, and lighting conditions to obtain the predicted future scene image corresponding to that image data. For example, using the first and second visual markers corresponding to each frame of image data, the structural constraint framework for the future scene image data corresponding to each frame of image data is determined; the fine-grained information indicated by the third visual marker corresponding to each frame of image data is then filled into the structural constraint framework to obtain the predicted future scene image corresponding to each frame of image data.

[0105] In step S405 above, the image dataset naturally includes the real future scene image corresponding to each frame of image data, eliminating the need for additional manual annotation and fully utilizing the quality of the generated results from massive video resources. The autonomous driving system can obtain the real future scene image corresponding to each frame of image data from the image dataset. Combined with the predicted future scene image corresponding to each frame of image data, the autonomous driving system can determine the prediction loss of the large language model. When the prediction loss indicates model convergence, model training ends. If the prediction loss indicates model non-convergence, the autonomous driving system adjusts the parameters of the large language model and re-executes step S402 until the model converges.

[0106] In this embodiment, the training process of the autonomous driving system for the large language model can be found in [reference needed]. Figure 5 As shown: The autonomous driving system provides instructions such as "What action should I take in relation to the vehicle in front?", "Please generate future lane markings and the speed of the vehicle in front", "Please generate a spatial multi-dimensional detection block diagram", and "Please predict what the next frame will be?", along with a set of surround-view images collected by the vehicle at a given moment (e.g., ...). Figure 5 The large language model takes the front, back, left, and right images as input and outputs control commands and corresponding answers, such as smooth braking and smooth stopping commands, lane line images, multi-dimensional bounding box images, and the next frame image, using a multi-dimensional spatiotemporal CoT method. This training method greatly improves the model's language understanding ability.

[0107] In some embodiments, the text codebook of the large language model includes visual tags generated by encoding image data of a driving scene using a vector quantization variational encoder. This improvement enables multimodal large language models to predict visual tags.

[0108] In this embodiment, the autonomous driving system can utilize an existing large language model, maintaining its architectural design unchanged and without modifying any structural components to ensure compatibility with pre-trained weights. When using an existing large language model, the autonomous driving system only needs to add visual tags generated by encoding image data of the driving scene using a vector quantization variational encoder to the text codebook of the large language model, enabling the large language model to have visual generation capabilities. This eliminates the need for retraining from scratch, significantly reducing training costs. A large language model with visual generation capabilities only needs to follow... Figure 4 The illustrated process trains a large language model, allowing it to retain its language understanding capabilities while activating its visual generation capabilities to predict future frames and capture the dynamic changes in the driving environment. Specifically, given an image / instruction input pair (p, l), the model predicts the visual label sequence {b1, b2, ..., b...} for future frames using an autoregressive generation method. a×rIts probability distribution can be expressed as:

[0109]

[0110] in, Represents the total probability product. The larger the value of b, the more accurate the large language model's prediction of visual label sequences. n This represents the visual marker at index n. Indicates the total number of indexes. Let represent all visual tags with indices less than n, and △ be the set of parameters for the large language model. This indicates that, given all previous visual markers, the current position has a value of b. n The probability of the large language model. The large language model uses a decoder derived from a vector quantization variational autoencoder to restore the generated discrete visual tags to image pixel information, i.e., to the image itself.

[0111] During training, the autonomous driving system can perform full fine-tuning of the large language model while freezing the visual encoder and vector quantization variational encoder to improve training efficiency.

[0112] In this embodiment, after training the large language model to predict future scene images, the autonomous driving system can further train its driving decision-making ability using GVQA task data. This enhances the large language model's ability to understand language interaction in autonomous driving's full-vision tasks, using the spatiotemporal thought chain as an intermediate reasoning basis. The training model generates a safe driving trajectory from "current observation data + future prediction results." For example, the autonomous driving system inputs an image dataset into the large language model, obtains the predicted future scene image corresponding to each frame of image data according to steps S402-S404, and then uses step S103 to predict the trajectory sequence corresponding to each frame of image data. Based on the predicted trajectory sequence and the vehicle's actual trajectory sequence in the image dataset, the prediction loss is determined. When the prediction loss indicates model convergence, model training ends; if the prediction loss indicates model non-convergence, the autonomous driving system adjusts the parameters of the large language model and re-executes step S402 until the model converges. This allows the large language model to further focus on the two core tasks of autonomous driving scene understanding and trajectory planning, improving its application performance in real-world driving scenarios. The autonomous driving system can distinguish different tasks through task prompts, enabling a single model to flexibly switch between scene understanding and trajectory planning modes without requiring additional architectural modifications.

[0113] In the technical solution provided in this application embodiment, after the autonomous driving system (i.e., large language model) is applied to the vehicle, the autonomous driving system can periodically or event-triggeredly execute the above-mentioned model training method to improve the model accuracy.

[0114] By employing the aforementioned model training and trajectory planning methods, autonomous driving systems are upgraded from simply "driving" to "understanding the world." Internally, a "world model" capable of extrapolating dynamic changes in the physical world is built, achieving a fundamental shift from "passively responding to the present" to "actively predicting the future." Specifically:

[0115] The first core breakthrough is upgrading the perception system from spatial representation to spatiotemporal representation, and introducing an innovative training paradigm of "next frame prediction (such as future scene images)." The learning focus of the large language model is no longer the "static state of the current world," but rather the "dynamic changes of the current state." Simultaneously, the perception encoder can be fully upgraded to a three-dimensional structure, deeply integrating multimodal data from cameras and LiDAR, completely preserving the height information, motion trajectory, and dynamic relationships of objects. This crucial information, often overlooked in two-dimensional bird's-eye views, is fundamental to understanding physical causal relationships. Once the large language model possesses spatiotemporal perception capabilities, it can make predictions more consistent with physical laws: such as the position of an accelerating vehicle in the next frame, the direction of a turning pedestrian, and the probability of non-motorized vehicles cutting in at intersections. These judgments no longer rely on perception accuracy but stem from a deep understanding of the dynamic structure of the world, making driving decisions more forward-looking and safer.

[0116] Secondly, during the training phase, the large language model first masters the ability to compress and decompress perceptual information through video data (such as image datasets); then it constructs a future prediction model in the latent space; and finally, it jointly trains the prediction ability with driving decisions. Through this process, the autonomous driving system acquires the cognitive ability to "imagine the future" while keeping the computational cost within the range of real-time vehicle operation, achieving a balance between "high cognition" and "efficient operation." In the trajectory planning phase, the embodiments of this application realize a fundamental shift from "predicting the trajectory" to "generating the trajectory": traditional trajectory planning is a typical "single path prediction," that is, given the current state, it outputs the most likely sequence of path points; while the trajectory planning provided by the embodiments of this application is a generative trajectory planning method, which draws on the generative logic of the large language model, allowing the model to autonomously generate the optimal driving trajectory. This shift brings a qualitative improvement in decision-making ability, such as the ability of the generative model to consider multiple scenario possibilities simultaneously and converge to the optimal solution in half a day through multiple rounds of iterative optimization.

[0117] Furthermore, this application's embodiments achieve a key efficiency breakthrough: upgrading scene generation from traditional step-by-step reconstruction to "feedforward generation," increasing speed by approximately two times and reducing training costs. More importantly, this application's embodiments enable autonomous driving systems to shift from "learning" to "exploration," no longer limited to learning from limited human-labeled data, but capable of generating never-before-seen scenes. Through trial and error, optimization, and iteration, they achieve self-evolution, truly realizing "autonomous learning" to solve complex problems. Additionally, the advanced performance of the model ultimately needs to be implemented on vehicle hardware. A precise mapping relationship needs to be established between the model and hardware performance, followed by large-scale architecture search. In vehicle scenarios, a "wider but shallower" model structure is superior to a deep structure, contrary to the design experience of large language models, reflecting that the vehicle's requirements for real-time performance far outweigh the pursuit of parameter scale. Through the method provided in this application's embodiments, the model architecture exploration cycle can be compressed from months to days, significantly improving the efficiency of technology iteration.

[0118] The value of this application's embodiments extends far beyond upgrading autonomous driving technology; it lays a solid foundation for developing a "general physical artificial intelligence (AI)." The core of physical intelligent agents is enabling AI to perceive, make decisions, and act in the real physical world. Training such AI requires three core conditions: a complete perception system, real-time decision-making needs, and large-scale real-world data. Automobiles happen to possess all three conditions, and millions of intelligent vehicles are already on the road, generating massive amounts of real-world driving scenario data daily. The large language model (such as the VLA model) obtained through the training method provided in this application's embodiments can be applied not only to the field of autonomous driving but also to the field of robot control. This is because autonomous driving and robot control are highly similar in their underlying logic: both require perceiving the intentions of three-dimensional space and the motion state of physical objects, and making optimal action decisions under real-time constraints. The difference between autonomous driving and robots lies primarily in the actuator level—one uses a steering wheel, brakes, and accelerator, while the other uses robotic arms and legs. When the underlying perception and decision-making models are generalized, the large language model obtained through the training method provided in this application's embodiments can be extended across fields to broader physical intelligent agent scenarios such as robotics and intelligent manufacturing.

[0119] The core significance of this application's embodiments lies not in whether it currently achieves an absolute technical route, but in that it points to a new direction for the industry: the future of autonomous driving will no longer be a competition of the performance of a single model, but a comprehensive competition of physical intelligent agent systems. The core of this competition is whose system can learn faster, evolve faster, and adapt more efficiently to the complex and ever-changing physical world. For example... Figure 6As shown in the comparison diagram of different CoTs, (1) the discrete text CoT adopts the discrete text mode, and the information provided by the expression "pedestrians and obstacles are on the right" obtained by inferring from the image data is insufficient. Based on this, the planned output control command is to go straight, which will cause a collision; (2) the image text CoT adopts the image text thinking mode, and there is a modal inconsistency problem between the image and the text "pedestrians and obstacles are in front" obtained by inferring from the image data. Based on this, the planned output control command is to go straight to the right, which may cause a collision; (3) the multidimensional spatiotemporal CoT, which is relied upon in the embodiments of this application, adopts the multidimensional spatiotemporal thinking mode, which can capture the spatiotemporal relationship in the future scene, such as lane lines, multidimensional bounding boxes, and the next frame image (i.e., the future scene map). Based on this, the planned output control command is to stop, which can effectively improve driving safety.

[0120] In summary, the technical solutions provided in this application have the following beneficial effects:

[0121] (1) A multi-dimensional spatial reasoning mechanism is proposed to realize a closed loop of pure visual end-to-end causal reasoning.

[0122] This application proposes a novel spatiotemporal thinking architecture that breaks the inherent pattern of cross-modal conversion in traditional VLM models. It constructs a closed-loop reasoning link within the entire visual space of "observation-thinking-decision-making," eliminating the need for text conversion throughout the process. This completely avoids the reasoning bottlenecks and accuracy losses caused by modal conversion, solves the problems of reasoning discontinuity and insufficient accuracy in traditional models, and enhances the reasoning coherence and stability of autonomous driving.

[0123] (2) It uses future scene diagrams as reasoning carriers, without intermediate text, and deeply integrates spatiotemporal information.

[0124] This application adopts a unified future scene map as the core reasoning carrier, abandons intermediate text symbol conversion, integrates dynamic evolution information of the time dimension and geometric structure information of the spatial dimension at the visual level, fully preserves the fine geometric and spatiotemporal correlation features of high-dimensional visual data, eliminates the loss of spatiotemporal relationship caused by text compression, and maximizes the use of perceptual information.

[0125] (3) The future scenario diagram has the dual core functions of a world model and an inverse dynamics model.

[0126] The future scene map designed in this application realizes functional reuse. The future scene map serves as both a world model to complete the global prediction of future scene states such as drivable areas and obstacle poses, and an inverse dynamics model to reverse-engineer the optimal driving trajectory by combining current observation data, thereby realizing the integration of perception prediction and decision output and simplifying the model reasoning architecture.

[0127] (4) Innovative pre-training paradigm, slightly expanded text codebook, and efficient unlocking of VLM visual generation capabilities.

[0128] The model training method proposed in this application is a pre-training method that unifies visual generation and understanding. It only slightly expands the existing VLM text codebook (i.e., vocabulary) without reconstructing the backbone network. While fully preserving the model's powerful semantic understanding capabilities, it unlocks visual generation capabilities in a low-cost and high-efficiency manner, balancing model performance and training resource consumption.

[0129] (5) Adopt a progressive generation strategy from coarse to fine to enhance physical realism.

[0130] This application's embodiments introduce a progressive generation mechanism. The model first learns the scene's physical constraint skeleton, such as lane lines and multi-dimensional bounding boxes, and then fills in the scene's pixel details. It follows a coarse-to-fine generation logic, ensuring that the constraint prediction results conform to traffic physics rules. This solves the problem that directly generating complex scenes can easily violate physical laws, thereby improving prediction accuracy and scene realism.

[0131] (6) Eliminate information loss and modal gap to improve the physical realism and accuracy of reasoning.

[0132] This application's embodiments completely solve the core drawbacks of traditional VLM, such as high-dimensional visual information loss and cross-modal semantic bias, through full-vision reasoning and hierarchical progressive generation design. At the same time, relying on the physical constraint generation mechanism, it significantly optimizes the physical rationality of future scene and trajectory prediction, and comprehensively improves the environmental understanding and decision-making accuracy of autonomous driving systems.

[0133] Corresponding to the trajectory planning method described above, this application also provides a trajectory planning device, such as... Figure 7 As shown, the device includes:

[0134] The acquisition module 701 is used to acquire driving perception data of vehicles in the current scene;

[0135] The first reasoning module 702 is used to use the world knowledge built into the large language model, take the lane lines in the future scene associated with the driving perception data as static physical constraints, and take the traffic participants associated with the driving perception data in the vehicle drivable area defined by the lane lines as dynamic physical constraints, to reason about the driving perception data and obtain the future scene image.

[0136] The second reasoning module 703 is used to use a large language model and a spatiotemporal thought chain as an intermediate reasoning carrier to reason about driving perception data and future scene images to obtain the vehicle's trajectory sequence.

[0137] In some embodiments, the first reasoning module 702 is specifically used to infer driving perception data using the world knowledge built into the large language model, and sequentially obtain a first visual marker of lane lines in the future scene, and a second visual marker of traffic participants within the vehicle drivable area defined by the first visual marker; using the world knowledge built into the large language model, with the first visual marker as a static physical constraint and the second visual marker as a dynamic physical constraint, to infer driving perception data to obtain a third visual marker of the future scene; and using the first visual marker, the second visual marker, and the third visual marker, to generate an image of the future scene.

[0138] In some embodiments, the first inference module 702 is specifically used to obtain a future unified frame corresponding to the future scene. The future unified frame is generated using a first visual marker and a second visual marker. The first visual marker is a visual marker of lane lines in the future scene obtained by inferring from historical driving perception data using the world knowledge built into the large language model. The first visual marker defines the drivable area of ​​the vehicle. The second visual marker is a visual marker of traffic participants within the drivable area of ​​the vehicle obtained by inferring from historical driving perception data using the world knowledge built into the large language model. Using the world knowledge built into the large language model, with lane lines in the future unified frame as static physical constraints and traffic participants in the future unified frame as dynamic physical constraints, the driving perception data is inferred to obtain a third visual marker of the future scene. The future scene image is generated using the first visual marker, the second visual marker, and the third visual marker.

[0139] In some embodiments, the first reasoning module 702 is specifically used to determine the structural constraint framework of the future scene image using a first visual marker and a second visual marker; and to fill the structural constraint framework with fine-grained information indicated by a third visual marker to obtain the future scene image.

[0140] In some embodiments, the text codebook of the large language model includes visual tags generated by encoding image data of the driving scene using a vector quantization variational encoder.

[0141] In the technical solution provided in this application, the large language module has built-in world knowledge, enabling it to understand dynamic change patterns. The large language model utilizes this built-in world knowledge to infer driving perception data of the vehicle in the current scene, obtaining images of future scenes. Then, using a spatiotemporal thought chain as an intermediate reasoning carrier, the large language model further infers from the driving perception data and future scene images to obtain the vehicle's trajectory sequence. This reasoning process more closely resembles the simulation and deduction of the physical world, enabling it to perceive the dynamic change patterns of the physical world, establish correct causal relationships in the physical world, and improve the accuracy of trajectory sequence reasoning. Furthermore, the large language model, utilizing its built-in world knowledge, can autonomously generate simulated long-tail scenarios, improving the reliability of decision-making and enhancing the safety of autonomous driving.

[0142] Corresponding to the above-described model training method, this application also provides a model training apparatus, such as... Figure 8 As shown, the device includes:

[0143] The acquisition module 801 is used to acquire the image dataset for autonomous driving and the instruction problem corresponding to each frame of image data in the image dataset.

[0144] The first reasoning module 802 is used to use the world knowledge built into the large language model to reason about each frame of image data and the corresponding instruction problem in the image dataset, and sequentially obtain the first visual marker of the lane line in the future scene corresponding to each frame of image data, and the second visual marker of traffic participants in the vehicle drivable area defined by the first visual marker.

[0145] The second reasoning module 803 is used to use the world knowledge built into the large language model, take the first visual label corresponding to each frame of image data as a static physical constraint, and take the second visual label corresponding to each frame of image data as a dynamic physical constraint, to reason about each frame of image data and the corresponding instruction problem, and obtain the third visual label of the future scene corresponding to each frame of image data.

[0146] The third inference module 804 is used to generate a predicted future scene image corresponding to each frame of image data by utilizing the first visual marker, the second visual marker and the third visual marker corresponding to each frame of image data.

[0147] The training module 805 is used to iteratively train the large language model using the predicted future scene image corresponding to each frame of image data and the future scene image in the image dataset corresponding to each frame of image data.

[0148] In some embodiments, the third inference module 804 is specifically used to determine the structural constraint framework of the future scene image data corresponding to each frame of image data by using the first visual marker and the second visual marker corresponding to each frame of image data; and to fill the structural constraint framework corresponding to each frame of image data with the fine-grained information indicated by the third visual marker corresponding to each frame of image data to obtain the predicted future scene image corresponding to each frame of image data.

[0149] In some embodiments, the text codebook of the large language model includes visual tags generated by encoding image data of the driving scene using a vector quantization variational encoder.

[0150] In the technical solution provided in this application, the large language module has built-in world knowledge, enabling it to understand dynamic change patterns. The large language model utilizes this built-in world knowledge to infer driving perception data of the vehicle in the current scene, obtaining images of future scenes. Then, using a spatiotemporal thought chain as an intermediate reasoning carrier, the large language model further infers from the driving perception data and future scene images to obtain the vehicle's trajectory sequence. This reasoning process more closely resembles the simulation and deduction of the physical world, enabling it to perceive the dynamic change patterns of the physical world, establish correct causal relationships in the physical world, and improve the accuracy of trajectory sequence reasoning. Furthermore, the large language model, utilizing its built-in world knowledge, can autonomously generate simulated long-tail scenarios, improving the reliability of decision-making and enhancing the safety of autonomous driving.

[0151] Furthermore, in this embodiment, the large language model first parses the scene structure information and then fills in the details. This progressive generation strategy from coarse to fine solves the problem of physical law violations that may occur when directly generating future frames from the root, ensuring the physical rationality of the generated scene and improving the realism of the image, while also enhancing the model's scene understanding ability.

[0152] This application also provides an autonomous driving system, such as... Figure 9 As shown, the system includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904. The processor 901, communication interface 902, and memory 903 communicate with each other via the communication bus 904. The memory 903 stores computer programs; the processor 901, when executing the program stored in the memory 903, implements any of the aforementioned trajectory planning methods or any of the aforementioned model training methods.

[0153] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0154] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0155] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0156] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0157] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described trajectory planning methods or any of the above-described model training methods.

[0158] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the above-described trajectory planning methods or any of the above-described model training methods.

[0159] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0160] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, systems, media, and program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0162] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A trajectory planning method, characterized in that, The method includes: Acquire driving perception data of vehicles in the current scene; Using the world knowledge built into the large language model, the driving perception data is inferred to obtain a future scene image by taking the lane lines in the future scene associated with the driving perception data as static physical constraints and the traffic participants associated with the driving perception data in the vehicle drivable area defined by the lane lines as dynamic physical constraints. Using a large language model and a spatiotemporal thought chain as an intermediate reasoning carrier, the driving perception data and the future scene image are reasoned to obtain the trajectory sequence of the vehicle.

2. The method according to claim 1, characterized in that, The step of using the world knowledge built into the large language model, taking lane lines in the future scene associated with the driving perception data as static physical constraints, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, to infer the future scene image from the driving perception data includes: By using the world knowledge built into the large language model, reasoning is performed on the driving perception data to obtain the first visual marker of the lane line in the future scene, and the second visual marker of traffic participants in the vehicle driving area defined by the first visual marker. Using the world knowledge built into the large language model, with the first visual marker as a static physical constraint and the second visual marker as a dynamic physical constraint, reasoning is performed on the driving perception data to obtain a third visual marker for the future scene. A future scene image is generated using the first visual marker, the second visual marker, and the third visual marker.

3. The method according to claim 1, characterized in that, The step of using the world knowledge built into the large language model, taking lane lines in the future scene associated with the driving perception data as static physical constraints, and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, to infer the future scene image from the driving perception data includes: A unified future frame corresponding to a future scene is obtained. The unified future frame is generated using a first visual marker and a second visual marker. The first visual marker is a visual marker of lane lines in the future scene obtained by reasoning from historical driving perception data using the world knowledge built into the large language model. The first visual marker defines the drivable area of ​​the vehicle. The second visual marker is a visual marker of traffic participants in the drivable area of ​​the vehicle obtained by reasoning from historical driving perception data using the world knowledge built into the large language model. Using the world knowledge built into the large language model, with lane lines in the future unified frame as static physical constraints and traffic participants in the future unified frame as dynamic physical constraints, the driving perception data is inferred to obtain the third visual label of the future scene. A future scene image is generated using the first visual marker, the second visual marker, and the third visual marker.

4. The method according to claim 2 or 3, characterized in that, The step of generating a future scene image using the first visual marker, the second visual marker, and the third visual marker includes: Using the first visual marker and the second visual marker, a structural constraint framework for the future scene image is determined; The future scene image is obtained by filling the fine-grained information indicated by the third visual marker within the structural constraint framework.

5. The method according to any one of claims 1-3, characterized in that, The text codebook of the large language model includes visual tags generated by encoding image data of driving scenarios using a vector quantization variational encoder.

6. A model training method, characterized in that, The method includes: The problem of obtaining the image dataset for autonomous driving, and the instructions corresponding to each frame of image data in the image dataset; Using the world knowledge built into the large language model, reasoning is performed on each frame of image data and the corresponding instruction problem in the image dataset, and the first visual marker of the lane line in the future scene corresponding to each frame of image data and the second visual marker of traffic participants in the vehicle driving area defined by the first visual marker are obtained sequentially. Using the world knowledge built into the large language model, the first visual label corresponding to each frame of image data is used as a static physical constraint, and the second visual label corresponding to each frame of image data is used as a dynamic physical constraint. Reasoning is performed on each frame of image data and the corresponding instruction problem to obtain the third visual label of the future scene corresponding to each frame of image data. Using the first visual marker, second visual marker, and third visual marker corresponding to each frame of image data, a predicted future scene image corresponding to each frame of image data is generated; The large language model is iteratively trained using the predicted future scene image corresponding to each frame of image data and the future scene image in the image dataset corresponding to each frame of image data.

7. The method according to claim 6, characterized in that, The step of generating a predicted future scene image corresponding to each frame of image data using the first visual marker, the second visual marker, and the third visual marker corresponding to each frame of image data includes: Using the first and second visual markers corresponding to each frame of image data, the structural constraint framework of the future scene image data corresponding to each frame of image data is determined; By filling the structural constraint framework corresponding to each frame of image data with the fine-grained information indicated by the third visual marker corresponding to each frame of image data, the predicted future scene image corresponding to each frame of image data is obtained.

8. The method according to claim 6 or 7, characterized in that, The text codebook of the large language model includes visual tags generated by encoding image data of driving scenarios using a vector quantization variational encoder.

9. A trajectory planning device, characterized in that, The device includes: The acquisition module is used to acquire driving perception data of vehicles in the current scene; The first reasoning module is used to use the world knowledge built into the large language model, with lane lines in the future scene associated with the driving perception data as static physical constraints and traffic participants associated with the driving perception data within the drivable area defined by the lane lines as dynamic physical constraints, to reason about the driving perception data and obtain a future scene image. The second reasoning module is used to use a large language model and a spatiotemporal thought chain as an intermediate reasoning carrier to reason about the driving perception data and the future scene image to obtain the trajectory sequence of the vehicle.

10. A model training device, characterized in that, The device includes: The acquisition module is used to acquire the image dataset for autonomous driving, and the instruction problem corresponding to each frame of image data in the image dataset; The first reasoning module is used to use the world knowledge built into the large language model to reason about each frame of image data and the corresponding instruction question in the image dataset, and sequentially obtain the first visual marker of the lane line in the future scene corresponding to each frame of image data, and the second visual marker of traffic participants in the vehicle drivable area defined by the first visual marker. The second reasoning module is used to use the world knowledge built into the large language model, with the first visual label corresponding to each frame of image data as a static physical constraint and the second visual label corresponding to each frame of image data as a dynamic physical constraint, to reason about each frame of image data and the corresponding instruction problem, and obtain the third visual label of the future scene corresponding to each frame of image data. The third inference module is used to generate a predicted future scene image corresponding to each frame of image data by utilizing the first visual marker, the second visual marker and the third visual marker corresponding to each frame of image data. The training module is used to iteratively train the large language model using the predicted future scene image corresponding to each frame of image data and the future scene image in the image dataset corresponding to each frame of image data.

11. An autonomous driving system, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images