A method for generating autonomous driving scenarios based on a large language model

Abstract

This invention discloses a method for generating autonomous driving scenes based on a large language model, relating to the field of autonomous driving technology. The method uses a trajectory generator to fine-tune a large language model by inputting scene descriptions and user prompts, generating discrete trajectory representations and restoring them to continuous trajectories. A high-precision map generator uses the trajectory sequence as a condition and a diffusion transformer network to generate a topologically consistent high-precision map. A multi-view video generator encodes 3D bounding boxes, the high-precision map, and the first frame image into structured features, combines them with text features, and inputs them into a video diffusion model to generate multi-view driving videos. This invention overcomes the limitations of traditional black-box probabilistic sampling and solves the problems of map and trajectory logic disconnect and insufficient spatiotemporal consistency in multi-view videos in existing technologies, achieving fine-grained and controllable generation of autonomous driving scenes using natural language.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a method for generating traffic scene videos based on a large language model and a diffusion model for autonomous driving world models. Background Technology

[0002] As autonomous driving technology evolves towards a data-driven end-to-end architecture, building world models capable of generating high-fidelity, interactive driving scenarios has become a key path to solving the problem of long-tail data scarcity. Existing 2D scene evolution world models are mainly divided into three categories: methods based on autoregressive sequence modeling, conditional generation methods based on structured prior guidance, and multi-view collaborative generation methods based on attention mechanism interaction.

[0003] However, existing technologies still face the following three major technical bottlenecks:

[0004] (1) The autoregressive sequence modeling method generates data based on "black box" sampling of data probability distribution. This results in the model only generating the most statistically likely scenarios, or only being able to perform coarse-grained style control through simple category labels (such as "rainy day" and "night"). This leads to a lack of user interaction and the inability to understand the user's specific intentions because users cannot accurately define the dynamic interaction logic in the scenario through natural language instructions or specific parameters. Due to the lack of effective human-computer interaction interfaces and fine-grained control methods, the generated driving scenarios are often unpredictable. It is impossible to construct targeted corner cases to test the boundary capabilities of autonomous driving algorithms under specific conditions, which greatly limits the practical value of world models in autonomous driving research and development testing.

[0005] (2) Structured prior guidance methods often use CNN architecture, which makes it difficult to capture topological information with sparse and vectorized features such as lane lines and road boundaries, often resulting in problems such as lane line breaks. At the same time, the lack of an effective "trajectory-map" conditional guidance mechanism often leads to spatial semantic misalignment in the generated video, such as vehicles crossing lane lines or passing through medians, which seriously undermines the effectiveness of the simulation scene.

[0006] (3) Multi-view collaborative generation methods based on attention mechanisms are essentially "soft constraints" in their cross-view attention, making it difficult to guarantee pixel-level geometric accuracy within overlapping view domains and prone to positional drift, deformation, or ghost artifacts. Furthermore, as the number of viewpoints increases, the computational complexity of the cross-view attention mechanism increases quadratically, limiting its spatiotemporal modeling capabilities when generating long video sequences. This lack of geometric and spatiotemporal consistency makes it difficult for the generated scene data to meet the stringent requirements of multi-sensor fusion-based perception algorithms (such as BEV Fusion) for the spatial geometric accuracy of the input data.

[0007] Therefore, there is an urgent need for a new solution that can integrate natural language semantic reasoning, possess global topological capture capabilities, and ensure strict spatiotemporal alignment from multiple perspectives. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a method for generating autonomous driving scenarios based on a large language model, comprising the following steps:

[0009] Trajectory generation steps: Construct a "prompt-scene description-future trajectory" training sample by extracting structured scene descriptions based on autonomous driving datasets and user-input prompt words, input it into a supervised fine-tuned large language model (LLM), and output a discrete trajectory representation; perform recovery processing on the discrete trajectory representation to obtain a continuous bird's-eye view (BEV) trajectory sequence;

[0010] High-precision map generation steps: The continuous bird's-eye view trajectory sequence is used as a conditional embedding and input into a Diffusion Transformer (DiT) network. A high-precision map (HDMap) that is spatially semantically consistent with the continuous bird's-eye view trajectory sequence is generated through a reverse diffusion denoising process.

[0011] Multi-view video generation steps: Based on the continuous bird's-eye view trajectory sequence and the agent category, generate a three-dimensional bounding box sequence corresponding to each moment; encode the three-dimensional bounding box, high-precision map, and the first frame image and T-frame noisy video into bounding box embedding vectors, high-precision map embedding vectors, and video embedding vectors with injected first frame information, respectively; concatenate the above embedding vectors along the channel dimension to obtain a structured feature vector, and then input it together with the text feature vector into the SVD (Stable VideoDiffusion) multi-view video generator; generate a multi-view autonomous driving scene video consistent with the semantics of prompt words, trajectory evolution, and road topology through a video diffusion model, thereby forming a complete technical link consisting of "natural language prompts - scene description - trajectory generation - map generation - multi-view video generation".

[0012] Furthermore, in the trajectory generation step:

[0013] The vehicle's historical motion state, the state of surrounding traffic participants, and map semantic information are extracted from the autonomous driving dataset to construct the structured scene description. The structured scene description and user-input prompts are used as input, and the vehicle trajectory in the future time domain is used as a supervision label to perform supervised fine-tuning on the pre-trained large language model, so that the large language model can directly output the corresponding discrete trajectory representation of the vehicle based on the current scene description and prompts. The discrete trajectory representation includes at least one of the following: a motor type label, key waypoints, and a quantized sequence of future trajectory points.

[0014] Furthermore, the trajectory generation step specifically includes:

[0015] First, a training sample set is constructed based on the nuScenes dataset to extract the vehicle's historical motion state, the state of surrounding traffic participants, and map semantic information, and unify them into a bird's-eye view coordinate system centered on the vehicle to form a structured scene description. Then, the pre-trained large language model is supervised and fine-tuned by combining user prompts and corresponding future trajectory labels to convert the future trajectory into a discrete trajectory representation suitable for the language model to generate, including motor type labels, key waypoints, and a quantized sequence of future trajectory points. During inference, the large language model outputs the discrete trajectory representation based on the input prompts and scene description, and then the trajectory recovery module performs inverse quantization, interpolation, and smooth fitting to obtain a physically feasible and traffic semantically satisfied continuous bird's-eye view trajectory sequence.

[0016] Furthermore, the high-precision map generation step specifically includes:

[0017] The input continuous bird's-eye view trajectory sequence is rasterized and semantically encoded. Trajectory conditional embedding vectors are generated through 2D convolution, global average pooling, and linear projection. Simultaneously, time step information is sinusoidally encoded and input into a multilayer perceptron to generate temporal conditional embedding vectors. These two vectors are then fused to form a unified conditional embedding vector. Noise is added to the semantically encoded high-precision map, and the noisy map features, along with the conditional embedding vectors, are input into a diffusion transformer-based network to predict the noise. Then, a DDPM scheduler is used to perform backdiffusion iterations to recover clean, vectorized lane lines and road topology matching the trajectory constraints from the pure noise, resulting in the high-precision map.

[0018] Furthermore, adding noise to the semantically encoded high-precision map includes the following steps:

[0019] Introduce a pre-defined variance scheduling sequence ,definition , The signal retention coefficients at time step t are the coefficients that are retained. The larger the value, the more features of the original high-precision map are preserved. Let be the noise variance coefficient at the t-th time step of noise addition, and define the cumulative product. , The signal retention coefficients for the s-th noisy step, where s is the noisy step; the noisy high-precision map at any time step t. It can be calculated directly using the formula:

[0020]

[0021] in This represents a noisy high-precision map at time step t, where B is the batch size, 3 is the number of channels, H is the spatial height, and W is the spatial width. A high-precision map after semantic encoding. Let be a random noise tensor sampled from a standard normal distribution, whose dimension is 1 / 2. Maintain consistency Controlling the retention strength of the original signal, The injection intensity of noise is controlled; the high-precision map after adding noise is then processed by two-dimensional convolution, with C output channels. h The kernel size is k m =4, sliding step size is s m =4, thus obtaining the feature vector after convolution. After flattening and rearranging, and adding position embedding, a high-precision map embedding vector is obtained. ,in .

[0022] Furthermore, in the multi-view video generation step, the bounding box embedding vector construction method is as follows:

[0023] The continuous bird's-eye view trajectory sequence generates a 3D bounding box sequence for each frame according to the preset 3D bounding box size for the agent category. The corner points of the bounding boxes are then projected onto the bird's-eye view, resized, and normalized to form a 19-channel bounding box sequence image. Where B represents the batch and T represents the number of frames. The length and width of the resized sequence image are given, respectively. The bounding box sequence image is input into the downsampler, and the features downsampled from each viewpoint are concatenated according to K viewpoints to obtain the final bounding box embedding vector. , These are the feature length and width output after downsampling, respectively, and K is the number of viewpoints.

[0024] Furthermore, in the multi-view video generation step, the high-precision map embedding vector construction method is as follows:

[0025] The high-precision map generated in the high-precision map generation step is rasterized and semantically encoded into a semantic image. 3 represents road boundaries, lane lines, and sidewalks. These are the length and width of the continuous bird's-eye view plane, which are then resized and normalized to obtain a high-precision map sequence. B represents the batch number, and T represents the frame number. The length and width of the resized sequence image are given respectively. The high-precision map sequence is input into the downsampler to obtain the final high-precision map embedding vector. , These are the feature length and width output after downsampling, respectively, and K is the number of viewpoints.

[0026] Furthermore, in the multi-view video generation step, the method for constructing the video embedding vector by injecting the first frame information is as follows:

[0027] The first frame of the video is resized and input into a variational autoencoder (VAE). After being copied T times, the encoded image feature vector is obtained. The T-frame noisy video is resized and input into the variational autoencoder (VAE) to obtain the encoded video feature vector. ,Will and Concatenating along the feature dimension yields a video embedding vector in which information from the first frame is injected into each frame. .

[0028] Furthermore, in the multi-view video generation step, the structured feature vectors are... Text feature vectors The time step t is input into the SVD (Stable Video Diffusion) model, and after training, the denoised latent vector is obtained. The latent vectors are input into the variational autoencoder (VAE) to decode the multi-view video generated by the diffusion model. .

[0029] Furthermore, the text feature vector It is extracted from the video description, recombined based on keywords describing the weather or time, and encoded using the CLIP text encoder.

[0030] The beneficial effects of this invention are:

[0031] This invention proposes a trajectory generation method for a large language model based on scene description-supervised fine-tuning. It constructs a structured scene description by extracting the vehicle's historical motion state, the states of surrounding traffic participants, and map semantic information from the dataset. This description is then combined with user prompts to train the large language model, enabling it to directly output discrete trajectory representations based on the "prompts + scene description." This transforms complex traffic interaction scenarios into controllable spatiotemporal motion priors. This method avoids the instability issues caused by directly regressing high-precision continuous floating-point trajectory points using a large language model. Furthermore, it improves the interpretability and controllability of trajectory generation through hierarchical output of maneuver types, critical path points, and quantized trajectory points.

[0032] This invention designs a high-precision map generation method constrained by trajectory features. Utilizing a DiffusionTransformer network, it element-wise fuses the semantic encoding of the trajectory from a rasterized continuous bird's-eye view with the time-step embedding vector, serving as a guiding condition for the diffusion process. This design achieves inverse denoising through reparameterization techniques and a DDPM scheduler, enabling the reconstruction of vectorized lane lines and road topology conforming to trajectory constraints from a noisy state. This effectively solves the problem of the disconnect between map and agent trajectory logic in traditional generative models.

[0033] This invention proposes a multi-view video diffusion generation method based on multi-dimensional injection of structured features. By concatenating 3D bounding box sequences, rasterized high-precision map embedding vectors, and VAE-encoded features injected with first-frame information along the channel dimension, a highly constraining structured conditional feature is constructed. Combined with text feature vectors recombined with keywords, this method enables the SVD model to diffuse and generate multi-view driving scene videos under unified spatiotemporal constraints, significantly improving the visual fidelity and multi-view consistency of the video under complex weather conditions and at different times of day. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0035] Figure 2 A flowchart illustrating the steps involved in generating a high-precision map.

[0036] Figure 3 This is a schematic diagram of the time-conditional embedded module (MLP) network process.

[0037] Figure 4 This is a schematic diagram of the downsampler in the multi-view video generation process. Detailed Implementation

[0038] This invention provides a method for generating autonomous driving scenarios based on a large language model, comprising the following steps:

[0039] Trajectory generation steps: Construct a "prompt-scene description-future trajectory" training sample by extracting structured scene descriptions based on autonomous driving datasets and user-input prompt words, input it into a supervised fine-tuned large language model (LLM), and output a discrete trajectory representation; perform recovery processing on the discrete trajectory representation to obtain a continuous bird's-eye view (BEV) trajectory sequence;

[0040] High-precision map generation steps: The continuous bird's-eye view trajectory sequence is used as a conditional embedding and input into a Diffusion Transformer (DiT) network. A high-precision map (HDMap) that is spatially semantically consistent with the bird's-eye view trajectory sequence is generated through a reverse diffusion denoising process.

[0041] Multi-view video generation steps: Based on the continuous bird's-eye view trajectory sequence and the agent category, generate a sequence of three-dimensional bounding boxes corresponding to each moment to represent the spatial occupancy relationship of the vehicle and surrounding traffic participants in the time sequence process; encode the three-dimensional bounding boxes, high-precision map, and the first frame image and T-frame noisy video into bounding box embedding vectors, high-precision map embedding vectors, and video embedding vectors injected with the first frame information, respectively; concatenate the above embedding vectors to obtain a structured feature vector, and then input it together with the text feature vector into the SVD (Stable Video Diffusion) multi-view video generator; through the video diffusion model, generate a multi-view autonomous driving scene video consistent with the semantics of the prompt words, trajectory evolution, and road topology, thus forming a complete technical link consisting of "natural language prompts - scene description - trajectory generation - map generation - multi-view video generation".

[0042] See Figure 1 As shown, the trajectory generation step specifically includes the following steps:

[0043] First, a trajectory generation sample set for training a large language model is constructed based on the nuScenes dataset. For each moment in the scene sample, the vehicle's historical motion state, the states of surrounding traffic participants, and map semantic information are extracted. The current vehicle position is set as the local coordinate origin, and the current heading is set as the coordinate alignment direction, thus unifying the historical trajectory, future trajectory, and the states of surrounding traffic participants into the vehicle's central bird's-eye view coordinate system. The vehicle's historical motion state includes at least the position, speed, acceleration, and heading angle information at the historical moment. The states of surrounding traffic participants include at least the category, relative position, relative speed, and orientation information of targets such as vehicles, pedestrians, and bicycles. The map semantic information includes at least road boundaries, lane lines, lane attributes, intersection areas, zebra crossings, stop lines, and traffic signal status. Subsequently, the above structured scene information is organized into a unified structured scene description, which, along with user prompts, serves as input. The corresponding future time-domain vehicle trajectories are used as supervision labels to form training samples, providing supervised fine-tuning for the pre-trained large language model, enabling it to learn the mapping relationship from "prompts + scene description" to "future trajectory representation." To improve the stability of large language models for trajectory generation tasks and avoid numerical drift and trajectory jitter caused by directly regressing high-precision continuous floating-point coordinates, this invention does not directly output continuous trajectory points. Instead, it converts future trajectories into discrete trajectory representations suitable for language model generation. These discrete trajectory representations include at least one or more of the following: maneuver type labels, critical path points, and quantized future trajectory point sequences.

[0044] In this embodiment, there are N agents in the scene within the prediction time step. The large language model outputs discrete trajectory representations based on the input prompts and scene descriptions, where N represents the number of agents (including the vehicle and surrounding interactive agents), and T represents the prediction time step. For the ... Individual agents ( ) at the t-th time step ( trajectory points Defined as a triple:

[0045]

[0046] In the formula , This indicates the position of the intelligent agent in the unified bird's-eye view (BEV) world coordinate system, represented by the horizontal and vertical coordinates. This represents the heading angle of the intelligent agent.

[0047] To adapt the output to the discrete generation mechanism of the large language model, continuous trajectory points are quantized and encoded according to a preset bird's-eye view grid, mapping each trajectory point from continuous coordinates to discrete grid indices. During the inference phase, the large language model outputs discrete trajectory representations based on the input prompts and scene descriptions. Then, the trajectory recovery module performs inverse quantization, interpolation, and smooth fitting on the discrete trajectory representations, ultimately obtaining a physically feasible continuous bird's-eye view trajectory sequence that satisfies traffic semantic constraints. , It is all The complete trajectory sequence is composed of It is a trajectory point in a complete trajectory sequence; this continuous bird's-eye view trajectory sequence not only serves as a conditional input for the high-precision map generation step, but is also used for subsequent 3D bounding box generation and multi-view video generation, thereby providing a unified, stable and controllable motion prior for the entire autonomous driving world model.

[0048] See Figure 2 As shown, the high-precision map generation process specifically includes the following steps:

[0049] The input is a sequence of continuous bird's-eye view trajectories output from the trajectory generation step. Create a continuous bird's-eye view space of H×W (H is the spatial height, W is the spatial width), rasterize the continuous bird's-eye view trajectory, and then semantically encode it. ,in, This is a sequence of continuous bird's-eye view trajectories after semantic encoding. B represents the batch size, and 3 indicates that vehicles, other vehicles, and pedestrians are represented using different channels. The sequence is processed through a two-dimensional convolutional layer, with the number of output channels of the two-dimensional convolutional network set to [value missing]. The kernel size is The sliding step size is The fill size is The trajectory features obtained after convolution are obtained. , The height and width of the convolution output are given by , where , Global average pooling is performed on the trajectory features after convolution to obtain the pooled trajectory features. Trajectory features after further flattening Perform a linear projection on it, and let the number of channels after projection be . Finally, the trajectory conditional embedding vector is obtained. .

[0050] The other input is the time step t, which is then encoded by sine and cosine and fed into the input. Figure 3 The multilayer perceptron shown contains two fully connected layers, with a SiLU activation function between them. The final output time-conditional embedding vector is... .

[0051] The conditional embedding vector c is obtained by adding the trajectory conditional embedding vector and the time conditional embedding vector element by element.

[0052]

[0053] Secondly, the high-definition map (HDMap) is rasterized within a continuous H×W bird's-eye view space and then semantically encoded to obtain a semantically encoded high-definition map. The three channels are defined as road boundaries, lane lines, and sidewalks.

[0054] Furthermore, noise is added to the semantically encoded high-precision map. The specific process is as follows:

[0055] To improve computational efficiency and avoid obtaining the noisy state at step t through t iterations of sampling, this invention employs a reparameterization technique for direct sampling; a pre-defined variance scheduling sequence is introduced. ,definition , The signal retention coefficients at time step t are the coefficients that are retained. The larger the value, the more features of the original high-precision map are preserved. Let be the noise variance coefficient at the t-th time step of noise addition, and define the cumulative product. , Here, represents the signal retention coefficients at the s-th noisy step, where s is the noisy step; based on the additivity property of the Gaussian distribution, the noisy high-precision map at any time step t... The high-precision map in the initial state (i.e., the high-precision map at time t=0) is directly calculated using the formula:

[0056]

[0057] in This represents a noisy high-precision map at time step t. Let be a random noise tensor sampled from a standard normal distribution, whose dimension is 1 / 2. To maintain consistency, I represents the identity matrix. Controlling the retention strength of the original signal, The injection intensity of noise is controlled; the high-precision map after adding noise is then processed by two-dimensional convolution, with C output channels. h The kernel size is k m =4, sliding step size is s m =4, thus obtaining the feature vector after convolution. After flattening and rearranging, and adding position embedding, a high-precision map embedding vector is obtained. ,in .

[0058] The conditional embedding vector c and the high-precision map embedding vector are combined. The input is the noise predicted from the diffusion converter network. Then, the DDPM scheduler is used to execute the backdiffusion process, with the sampling process starting at time step t=T and continuing until t=0; at any time step t, the noisy high-precision map at the current time is used. and network prediction noise The recursive equation derived from the Bayesian posterior probability formula is used to calculate the feature state of the previous time step. The update formula for this inverse denoising is defined as:

[0059]

[0060] in, The characteristic state at time t; To control the variance term for sampling randomness, The value is usually taken as or , used to simulate random disturbances in the inverse process; z is random noise that follows a standard normal distribution, z ~ N(0, I) (when t=0, z=0);

[0061] Using the above recursive formula After several iterations, when t=0, a clean, high-precision map feature with all noise removed is finally obtained. The data is then decoded and restored to vectorized lane lines and road topology, i.e., a high-precision map.

[0062] See Figure 1 As shown, the multi-view video generation process specifically includes the following steps:

[0063] First, the continuous bird's-eye view trajectory sequence is used to generate a sequence of 3D bounding boxes for each frame according to the preset 3D bounding box size for the agent category. Eight corner points of the 3D bounding boxes are taken, and points with a z-axis smaller than 0.5m are clipped. These are then projected onto a 2D plane to obtain 2D polygons. These 2D polygons are then drawn on a 19-channel continuous bird's-eye view plane according to the agent category to obtain the initial bounding box planar image. ,in These are the length and width of the continuous bird's-eye view plane, respectively. Their dimensions are adjusted and normalized to the range [-0.5, 0.5], then packaged into batches to obtain the bounding box sequence map. Where B represents the batch and T represents the number of frames. These represent the length and width of the resized sequence image, respectively. Input the bounding box sequence diagram as follows Figure 4 The downsampling method shown concatenates the features downsampled from each viewpoint according to K viewpoints to obtain the final bounding box embedding vector. , These are the feature length and width output after downsampling, respectively. K represents the number of viewpoints, and K=6.

[0064] Secondly, the high-precision map generated in the high-precision map generation step is rasterized and semantically encoded into a semantic image. 3 represents road boundaries, lane lines, and sidewalks. These are the length and width of the continuous bird's-eye view plane, respectively. They are resized and normalized to the range [-1, 1], and then stitched together according to K viewpoints, packaged into batches, to obtain a high-precision map sequence. B represents the batch number, and T represents the frame number. The length and width of the resized sequence image are given respectively. The high-precision map sequence is input into the downsampler to obtain the final high-precision map embedding vector. , These are the feature length and width output after downsampling, respectively, and K is the number of viewpoints.

[0065] Next, the first frame of the video is resized and input into the variational autoencoder (VAE). After being copied T times, the encoded image feature vector is obtained. The T-frame noisy video is resized and input into the variational autoencoder (VAE) to obtain the encoded video feature vector. ,Will and Concatenating along the feature dimension yields a video embedding vector in which information from the first frame is injected into each frame. .

[0066] Embed the bounding box vector High-precision map embedding vector and the video embedding vector that injects information from the first frame Structured feature vectors are obtained by concatenating along the channel dimension. ;

[0067] The video description is extracted and recombined based on keywords, including weather or time keywords such as rainy day, sunny day, and night. These keywords are then encoded into text feature vectors using the CLIP text encoder. .

[0068] Finally, the structured feature vectors Text feature vectors The input at time step t is the SVD (Stable Video Diffusion) model, which is then trained to obtain the denoised latent vector. The latent vectors are input into the variational autoencoder (VAE) to decode the multi-view video generated by the diffusion model. .

Claims

1. A method for generating autonomous driving scenarios based on a large language model, characterized in that: Includes the following steps: Trajectory generation steps: Construct a "prompt-scene description-future trajectory" training sample by extracting structured scene descriptions based on autonomous driving datasets and user-input prompt words, input it into a supervised fine-tuned large language model, and output a discrete trajectory representation; perform recovery processing on the discrete trajectory representation to obtain a continuous bird's-eye view trajectory sequence; High-precision map generation steps: The continuous bird's-eye view trajectory sequence is used as a conditional embedding and input into a diffusion transformer network. A high-precision map that is spatially semantically consistent with the continuous bird's-eye view trajectory sequence is generated through a reverse diffusion denoising process. Multi-view video generation steps: Based on the continuous bird's-eye view trajectory sequence and the agent category, generate a three-dimensional bounding box sequence corresponding to each moment; encode the three-dimensional bounding box, high-precision map, first frame image of the video, and T-frame noisy video into bounding box embedding vector, high-precision map embedding vector, and video embedding vector with injected first frame information, respectively; concatenate the above embedding vectors along the channel dimension to obtain a structured feature vector, and then input it together with the text feature vector into the SVD multi-view video generator; generate a multi-view autonomous driving scene video consistent with the semantics of prompt words, trajectory evolution, and road topology through the video diffusion model.

2. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: In the trajectory generation step: The vehicle's historical motion state, the state of surrounding traffic participants, and map semantic information are extracted from the autonomous driving dataset to construct the structured scene description. The structured scene description and user-input prompts are used as input, and the vehicle trajectory in the future time domain is used as a supervision label to perform supervised fine-tuning on the pre-trained large language model, so that the large language model can directly output the corresponding discrete trajectory representation of the vehicle based on the current scene description and prompts. The discrete trajectory representation includes at least one of the following: a motor type label, key waypoints, and a quantized sequence of future trajectory points.

3. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: The trajectory generation step specifically includes: First, a training sample set is constructed based on the nuScenes dataset to extract the vehicle's historical motion state, the state of surrounding traffic participants, and map semantic information, and unify them into a bird's-eye view coordinate system centered on the vehicle to form a structured scene description. Then, the pre-trained large language model is supervised and fine-tuned by combining user prompts and corresponding future trajectory labels to convert the future trajectory into a discrete trajectory representation suitable for the language model to generate, including motor type labels, key waypoints, and a quantized sequence of future trajectory points. During inference, the large language model outputs the discrete trajectory representation based on the input prompts and scene description, and then the trajectory recovery module performs inverse quantization, interpolation, and smooth fitting to obtain a physically feasible and traffic semantically satisfied continuous bird's-eye view trajectory sequence.

4. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: The high-precision map generation steps specifically include: The input continuous bird's-eye view trajectory sequence is rasterized and semantically encoded. Trajectory conditional embedding vectors are generated through 2D convolution, global average pooling, and linear projection. Simultaneously, time step information is sinusoidally encoded and input into a multilayer perceptron to generate temporal conditional embedding vectors. These two vectors are then fused to form a unified conditional embedding vector. Noise is added to the semantically encoded high-precision map, and the noisy map features, along with the conditional embedding vectors, are input into a diffusion transformer-based network to predict the noise. Then, a DDPM scheduler is used to perform backdiffusion iterations to recover clean, vectorized lane lines and road topology matching the trajectory constraints from the pure noise, resulting in the high-precision map.

5. The method for generating autonomous driving scenarios based on a large language model according to claim 4, characterized in that: Adding noise to a semantically encoded high-precision map includes the following steps: Introduce a pre-defined variance scheduling sequence ,definition , The signal retention coefficients at time step t are the coefficients that are retained. The larger the value, the more features of the original high-precision map are preserved. Let be the noise variance coefficient at the t-th time step of noise addition, and define the cumulative product. , The signal retention coefficients for the s-th noisy step, where s is the noisy step; the noisy high-precision map at any time step t. It can be calculated directly using the formula: in This represents a noisy high-precision map at time step t, where B is the batch size, 3 is the number of channels, H is the spatial height, and W is the spatial width. A high-precision map after semantic encoding. Let be a random noise tensor sampled from a standard normal distribution, whose dimension is 1 / 2. To maintain consistency, I represents the identity matrix. Controlling the retention strength of the original signal, The injection intensity of noise is controlled; the high-precision map after adding noise is then processed by two-dimensional convolution, with C output channels. h The kernel size is k m =4, sliding step size is s m =4, thus obtaining the feature vector after convolution. After flattening and rearranging, and adding position embedding, a high-precision map embedding vector is obtained. ,in .

6. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: In the multi-view video generation step, the bounding box embedding vector construction method is as follows: The continuous bird's-eye view trajectory sequence generates a 3D bounding box sequence for each frame according to the preset 3D bounding box size for the agent category. The corner points of the bounding boxes are then projected onto the bird's-eye view, resized, and normalized to form a 19-channel bounding box sequence image. Where B represents the batch and T represents the number of frames. The length and width of the resized sequence image are given, respectively. The bounding box sequence image is input into the downsampler, and the features downsampled from each viewpoint are concatenated according to K viewpoints to obtain the final bounding box embedding vector. , These are the feature length and width output after downsampling, respectively, and K is the number of viewpoints.

7. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: In the multi-view video generation step, the high-precision map embedding vector construction method is as follows: The high-precision map generated in the high-precision map generation step is rasterized and semantically encoded into a semantic image. 3 represents road boundaries, lane lines, and sidewalks. These are the length and width of the continuous bird's-eye view plane, which are then resized and normalized to obtain a high-precision map sequence. B represents the batch number, and T represents the frame number. The length and width of the resized sequence image are given respectively. The high-precision map sequence is input into the downsampler to obtain the final high-precision map embedding vector. , These are the feature length and width output after downsampling, respectively, and K is the number of viewpoints.

8. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: In the multi-view video generation step, the method for constructing the video embedding vector by injecting the first frame information is as follows: The first frame of the video is resized and input into a variational autoencoder (VAE). After being copied T times, the encoded image feature vector is obtained. B represents the batch number, and T represents the frame number. These represent the feature length and width of the downsampled output, respectively, where K is the number of viewpoints. The T frames of noisy video are resized and input into the variational autoencoder (VAE) to obtain the encoded video feature vector. ,Will and Concatenating along the feature dimension yields a video embedding vector in which information from the first frame is injected into each frame. .

9. The method for generating autonomous driving scenarios based on a large language model according to claim 1, characterized in that: In the multi-view video generation step, the structured feature vectors are... Text feature vectors Inputting the SVD model at time step t, and obtaining the denoised latent vectors after training. B represents the batch number, and T represents the frame number. These represent the feature length and width of the downsampled output, respectively, where K is the number of viewpoints. The latent vectors are input into the variational autoencoder (VAE) to decode the multi-view video generated by the diffusion model. , These represent the length and width of the resized sequence image, respectively.

10. The method for generating autonomous driving scenarios based on a large language model according to claim 9, characterized in that: The text feature vector It is extracted from the video description, recombined based on keywords describing the weather or time, and encoded using the CLIP text encoder.

Images