Track generation method based on natural language control

Abstract

The invention relates to a trajectory generation method based on natural language control. The method comprises the following steps: extracting an interaction relationship in a data set scene; carrying out feature coding on input data belonging to different modals, coding a vehicle initial track and a map topology into track features and map features by applying a line segment encoder, and converting interaction relation text description into text features by applying a text embedding layer of a large language model; applying a random local attention module to obtain a text-track fusion feature and a text-map fusion feature; splicing the text-track fusion features, the text-map fusion features and the text features to jointly form interaction features, and inputting the interaction features into a large language model to obtain behavior output; the behavior description of the vehicle is output through the large language model; performing feature fusion by applying a random local attention module again; splicing the fusion features and the text features, and inputting the spliced features and the text features into a large language model to obtain track output; and carrying out model training.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence and autonomous driving, relates to trajectory generation technology, and specifically provides a method for generating vehicle trajectories based on natural language control. Background Art

[0002] With the continuous development of autonomous driving technology, core tasks such as trajectory prediction, scene perception, and decision-making and planning have higher requirements for the quality and quantity of traffic scene data. However, the method of collecting real-scene data through sensors requires extremely high time and economic costs. Therefore, methods for assisting in the production or generation of vehicle trajectory data are used in the industry. Such generation methods have extremely high application value in the field of autonomous driving because they can customize to meet business needs, have high data production efficiency, and do not require attention to safety issues.

[0003] The current mainstream trajectory production / generation methods can be summarized into three categories: First, scene editing is performed based on a graphical interface. Although such methods still require the high participation of domain experts and apply a high-fidelity graphics rendering engine to simulate the real world, the data production efficiency is still limited by manual work. Second, customized control is performed based on programming languages. Such methods are also limited by the participation of manual work and require domain experts to write a large amount of code to interact with the simulation environment and then control the vehicle movement to obtain the driving trajectory. Third, real-scene data is applied to train a generative model in combination with a neural network. With the rapid development of deep learning, the performance of such methods has been continuously improved, and currently, it is the mainstream generation method. The existing generation methods mainly use content such as pictures, traffic rules, and text as conditions to directly guide the training process, but the controllability of the generated trajectories is not strong, and unreasonable driving behaviors are likely to occur when vehicle interactions are involved, resulting in low authenticity of the trajectories. This is because the one-stage training ignores the huge modality differences between conditions such as text and trajectories, resulting in the model being unable to learn the behavioral logic followed by real human driving, and simply using information such as text as conditions cannot relatively accurately control the trajectories.

[0004] Related Literature:

[0005] [1]Touvron H,Lavril T,Izacard G,et al.LLaMA:Open and EfficientFoundation Language Models[J].arXiv preprint arXiv:2302.13971,2023.

[0006] [2] Gao J, Sun C, Zhao H, et al. VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020:11525-11533.

[0007] [3] Hu E J, Shen Y, Wallis P, et al. LoRA: Low-Rank Adaptation of Large Language Models[J]. arXiv preprint arXiv:2106.09685, 2021.

[0008] [4] Ettinger S, Cheng S, Caine B, et al. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021:9710-9719.

[0009] [5] Bergamini L, Ye Y, Scheel O, et al. SimNet: Learning Reactive Self-driving Simulations from Real-world Observations[C] / / 2021 IEEE International Conference on Robotics and Automation(ICRA). IEEE, 2021:5119-5125.

[0010] [6] Suo S, Regalado S, Casas S, et al. TrafficSim: Learning To Simulate Realistic Multi-Agent Behaviors[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:10400-10409.

[0011] [7] Xu D, Chen Y, Ivanovic B, et al. BITS: Bi-level Imitation for Traffic Simulation [C] / / 2023 IEEE International Conference on Robotics and Automation(ICRA). IEEE, 2023:2929-2936.

[0012] [8] Zhong Z, Rempe D, Xu D, et al. Guided Conditional Diffusion for Controllable Traffic Simulation[C] / / 2023 IEEE International Conference on Robotics and Automation(ICRA). IEEE, 2023:3560-3566.

[0013] [9] Zhong Z, Rempe D, Chen Y, et al. Language-Guided Traffic Simulation via Scene-Level Diffusion[C] / / Conference on Robot Learning. PMLR, 2023:144-177.

[0014]

[10] Tan S, Ivanovic B, Weng X, et al. Language Conditioned Traffic Generation [C] / / Conference on Robot Learning. PMLR, 2023:2714-2752.

[0015]

[11] Vaswani, Ashish, et al. Attention is all you need[C] / / Advances in Neural Information Processing Systems. 2017. Summary of the Invention

[0016] The present invention provides a vehicle trajectory generation method based on natural language control, which can control the generation of real, diverse, and controllable vehicle trajectories that conform to driving behavior logic, effectively improving the understanding ability of the generation model for driving logic. The method of the present invention also performs context feature fusion through the proposed attention structure, enhancing the model's understanding ability of the scene.

[0017] The present invention is realized through the following technical solutions:

[0018] A trajectory generation method based on natural language control, characterized by including the following steps:

[0019] Step 1: Extract the interactive relationship text description, vehicle initial trajectory, and map topology in the dataset scene as inputs;

[0020] Step 2: Perform feature encoding on the input data belonging to different modalities. Apply a line segment encoder to encode the vehicle initial trajectory and map topology into trajectory features and map features, and apply the text embedding layer of the large language model to convert the interactive relationship text description into text features;

[0021] Step 3: After obtaining the text features, trajectory features, and map features, apply a random local attention module to fuse the text features with the trajectory features and the text features with the map features respectively, and then obtain text-trajectory fusion features and text-map fusion features;

[0022] Among them, for the two parts of the original features that need to be fused by the random local attention module, the fusion features of the two parts of the original features are obtained, including the following steps: Through cross-attention calculation, obtain the correlation scores between the two parts of the original features of different interaction objects; Select the relevant segments with the top correlation scores as the local enhanced features of the original features; Use the obtained local enhanced features of the original features, introduce a random variable of normal distribution as a random perturbation coefficient, and weight the original features to obtain the fusion features of the two parts of the original features;

[0023] Step 4: Concatenate the text-trajectory fusion features, text-map fusion features, and text features to jointly form interaction features, and input them into the large language model to realize the fusion of different modality features in the first stage and obtain the behavior output;

[0024] Step 5: The large language model outputs a behavioral description of the vehicle, and the corresponding implicit features are intercepted from the regression layer of the large language model as behavioral features;

[0025] Step 6: Apply the randomly local attention module proposed in Step three again to fuse the interaction features obtained in Step four and the behavioral features obtained in Step five, and denote the fused features as interaction-behavior fusion features;

[0026] Step 7: Concatenate the interaction-behavior fusion features obtained in Step six with the text features and input them into the large language model to achieve the fusion of different modality features in the second stage and obtain the trajectory output;

[0027] Step 8: Use the dataset to train the model. Calculate the cross-entropy loss between the trajectory output by the large language model in Step seven and the real trajectory in the dataset during training, and use the gradient descent method to update the model parameters to finally obtain a trained neural network model;

[0028] Step 9: Use the trained neural network model to output the vehicle trajectory in the current scenario.

[0029] Further, in Step two, perform feature encoding on the input data belonging to different modalities. Apply the Polyline Encoder to encode the initial vehicle trajectory and the map topology into trajectory features and map features, and apply the text embedding layer of the large language model to convert the interactive relationship text description into text features, including:

[0030] (1) Feature encoding of text: Tokenize the input text description I to generate a sequence containing L tokens, and then map each token to a d i -dimensional vector through the text embedding layer ε of the large language model, so as to obtain a text embedding matrix of size [L, d i as the text feature F I ;

[0031] (2) Feature encoding of map topology: Use a line-based vectorization representation method for the map input M. Each road element is composed of a line segment, that is, an ordered vertex sequence. Through the line segment encoder Convert each line segment into a d m -dimensional vector to obtain a feature matrix representing the map topology as the map feature F M ;

[0032] (3) Feature encoding of vehicle trajectory: For the historical trajectories X of N vehicles in the scenario, regard each trajectory as composed of line segments between trajectory points, and convert it into a trajectory feature matrix through the line segment encoder as the trajectory feature F X .

[0033] Furthermore, for the two parts of original features A and B that need to be fused, the random local attention module obtains the fused feature of the two parts of original features, including the following calculation steps:

[0034] (1) Calculate the feature correlation score matrix S between the original features A and B through the cross-attention formula. First, calculate the feature dot product A·B T , and then divide it by the square root of the feature dimension d for numerical scaling. Finally, convert the dot product result into a probability distribution through softmax, and regard this as the correlation score between features;

[0035] (2) Apply the top K algorithm to select the top K segments with the highest correlation scores from the correlation score matrix S as the local enhanced features S topK = top k (S);

[0036] (3) Utilize the local enhanced features S topK , introduce a random variable of the normal distribution as a random perturbation coefficient, and weight the original features A and B to obtain the final fused feature

[0037] Furthermore, the large language model is LLaMA.

[0038] Furthermore, the fusion of different modality features in the two stages is performed through the regression head of the large language model for text mapping, and the cross-entropy loss is used to constrain the difference between the generated text and the real text during the training process.

[0039] Furthermore, the method of using the cross-entropy loss to constrain the difference between the generated text and the real text during the training process is as follows:

[0040] (1) For the behavior output, use the cross-entropy loss L B to calculate the difference between the generated trajectory B and the real value , and update the model parameters based on the backpropagation mechanism;

[0041] (2) For the trajectory output, use the cross-entropy loss L T to calculate the difference between the generated trajectory T and the real value , and update the model parameters based on the backpropagation mechanism.

[0042] Furthermore, in the dataset training of step eight, the LoRA parameter fine-tuning method is used, and the modules for updating parameters include the LoRA low-rank matrix module and the random local attention module of the large language model.

[0043] The substantial feature of the present invention is to improve the structure of the LLaMA model so that it can accept other modal data besides text as input, and introduce behavioral logic to construct a two-stage translation strategy of "text-behavior-trajectory". Apply the Polyline Encoder [2] to encode maps and trajectories, perform feature fusion through the randomly local attention module proposed by the present invention, convert the fused features into the form of token embeddings that can be processed by the large language model, and convert the trajectory generation problem into the "next token" prediction problem of the large language model. Furthermore, a real, diverse, and controllable trajectory generation effect that meets the text description is achieved. The beneficial effects of the technical solution provided by the present invention are:

[0044] 1. Existing trajectory generation methods based on text, images, rules, etc. directly learn the direct mapping from text to trajectory, ignoring the complex human driving logic contained in the trajectory, resulting in weak authenticity of the generated trajectory. The present invention introduces "behavioral logic" as an intermediate feature of trajectory generation, constructs a two-stage translation strategy of "text-behavior-trajectory", and utilizes the basic understanding ability of the pre-trained large language model for traffic scenarios to make the large language model more accurately understand and generate vehicle trajectories during fine-tuning training on high-quality datasets, thereby achieving a more realistic generation effect.

[0045] 2. Existing generation methods perform poorly in the controllability of generating trajectories, mainly because there is no good correlation between control conditions such as text and scene context features (map topology, vehicle initial trajectory, etc.). The present invention realizes a controllable and diverse generation effect by fusing text features with map and vehicle trajectory features through the proposed randomly local attention module. Brief Description of the Drawings

[0046] Figure 1 is a flowchart of a trajectory generation method based on natural language control;

[0047] Figure 2 is a model structure diagram of a trajectory generation method based on natural language control;

[0048] Figure 3 is a comparison of the experimental results of the method of the present invention and the existing optimal method. Detailed Embodiment

[0049] The technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those of ordinary skill in the art based on the technical solutions in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0050] The model adopted by the present invention is improved based on the LLaMA-7B model. First, the LLaMA model is briefly introduced below. As a large language model, its core principle is the self-attention mechanism, which can capture the dependencies existing in the sequence data. The working principles of large language models are similar: first, the input token sequence is converted into a sequence of word vectors through a word embedding matrix, and then it is successively passed through multiple decoder layers. Normalization is performed on the final output, and the result is mapped to tokens, thereby achieving the prediction effect of "the next token".

[0051] LLaMA-7B is a high-performance language generation model developed by Meta. It is a Decoder-Only model based on the Transformer architecture, containing approximately 7 billion parameters. In each decoder layer, the input hidden layer state is normalized by RMSNorm and then enters the multi-head attention module. The output of the attention module and the hidden layer state before normalization form a residual connection. Then, the new hidden layer state is normalized by RMSNorm and sent to the feed-forward neural network layer. Its output also forms a residual connection and finally serves as the output of the decoder layer. Stacking such decoder layers together forms a large language model with a Decoder-Only architecture. In LLaMA-7B, there are a total of 32 decoder layers, the hidden layer dimension is 4096, the number of multi-head attention heads is 32, RoPE is used as the positional encoding, and SwiGLU is used as the activation function.

[0052] The present invention does not change the internal structure of the original LLaMA-7B large language model, but instead adds a Polyline Encoder [2] as a feature encoder for maps and trajectories, and proposes a random local attention module to complete the fusion of map, trajectory features and text features. The fused features will be concatenated to the end of the original text input to form a new input sequence as the input of the large language model. After a two-stage translation process, the large language model outputs the final trajectory. The entire LoRA [3] fine-tuning training process mainly updates the parameters of the proposed random local attention module and the LoRA low-rank matrix module. The core idea of LoRA is to simulate the change amount of parameters by adding a low-rank matrix as a residual term of the original parameters while keeping most of the parameters of the pre-trained model unchanged, so as to adapt to specific tasks. Since the present invention does not involve the improvement of LoRA, it will not be introduced in detail.

[0053] See Figure 2 , the following mainly introduces the improvement points of the present invention, namely the proposed random local attention module and the two-stage translation process.

[0054] (1) Data Input.

[0055] Extract the map data, trajectory data, and annotation information of each scenario from the WOMD official dataset, and apply the corresponding modality encoder to encode their features. The complete data input includes: the map topology M and the initial trajectory set X of each vehicle in the scenario, as well as the text description I of the vehicle interaction relationship. The encoding process is as follows:

[0056] (1) The annotation information of the vehicle interaction relationship in the dataset mainly describes the interaction relationship between O pairs of vehicles among N vehicles in the current scenario and is recorded in the form of a text string (for example: "Vehicle A gives way to Vehicle B"). In the text processing of the large language model, first, the input text description I is tokenized to generate a sequence containing L tokens, and then each token is mapped to a 4096-dimensional vector through the text embedding layer ε of the LLaMA large language model, so as to obtain a text embedding matrix of size [L, 4096] as the text feature F. I 。

[0057] (2) The map topology M in the dataset adopts the widely used vectorized representation method based on polylines. Each road element (lane centerline, lane boundary, road boundary, crosswalk, etc.) is composed of polylines (ordered vertex sequences), and through the classic Polyline Encoder [2] each polyline is converted into a 512-dimensional vector, and finally a feature matrix representing the map topology is obtained as the map feature F. M 。

[0058] (3) The trajectories of each vehicle in the dataset are stored in the form of trajectory points. For the historical trajectories X of N vehicles in the scenario, each trajectory can also be regarded as composed of polylines between trajectory points. Therefore, through the classic Polyline Encoder [2] it is converted into a trajectory feature matrix as the trajectory feature F. X 。

[0059] (2) Build a neural network model.

[0060] The following mainly introduces the two-stage translation strategy and the functional modules that play roles in different stages.

[0061] In the "Interaction - Behavior" translation of the first stage, we denote the text description of the vehicles as I, which describes the interaction relationships between O pairs of vehicles among N vehicles in the current scenario. Each pair of interaction relationships can be simply written in the form of "Car A overtakes Car B". In this method, these descriptions can also be replaced by richer text content. Input K segments of text into the large language model The model will output the behavior description B of N vehicles in text form. In the text form, the behaviors will be represented by content similar to "[Car A] Change Lane: Both the left and right sides have ample road widths...; Keep Still:...".

[0062] The text description I of the vehicles is converted into interaction features through the text embedding layer ε of the large language model where C represents the feature dimension. The map topology in the current traffic scenario is represented by where E represents the number of line segments in the map. Each line segment contains D points, and each point contains U attributes (e.g., x, y coordinates, orientation, speed limit, etc.). The initial trajectories of N vehicles in the scenario are represented by where S represents the number of trajectory points in each initial trajectory at S moments, and each trajectory point contains four attributes (x coordinate, y coordinate, orientation, speed scalar). The scenario map and the initial trajectories of the vehicles together constitute the spatial constraints of the current scenario. The map and the trajectories are encoded by the Polyline Encoder [2] and then converted into map features a and trajectory features

[0063]

[0064] To ensure the consistency of vehicle trajectories and the map, and to prevent vehicles from driving out of the map boundary or violating traffic rules, the present invention proposes a random local attention module to fuse the text description I, the initial trajectory X, and the map M features:

[0065]

[0066] where, and respectively represent the correlation scores between O pairs of vehicle interaction descriptions and N initial trajectories, and between O pairs of vehicle interaction descriptions and E map line segments. C represents the feature dimension. This correlation score is calculated using the classic attention formula

[11] First, calculate the dot product of two feature vectors, and then divide by Scale the dot product matrix and finally obtain the correlation scores through softmax. The top K represents selecting the association relationships corresponding to the top K scores, and these selected correlation scores will be used to weight the initial trajectory feature F X and the map feature F M and add them to the interaction feature F I as follows:

[0067]

[0068] By setting the values of K < N and M, finally, the feature represents the local correlation of interaction and map space constraints. This local feature enables each pair of vehicle interactions to occur at the most reasonable position on the map. In addition, the and in the formula are random variables following a Gaussian distribution. The introduction of such random variables can randomly perturb the correlation scores and , so that the feature representing the o-th pair of interactions has a certain position randomness, thereby improving the diversity of the results.

[0069] Given the interaction feature , the large language model maps the result to a text segment of behavior B through the regression head. In Figure 2 , represents the regression head, which has a cross-attention architecture containing adjustable parameters:

[0070]

[0071] where σ represents a fully connected layer without a linear activation function. The regression head follows the autoregressive paradigm and uses the interaction feature and the already output behavior text to update the final behavior text B. The already output behavior text queries the context information in the interaction feature to ensure that the output behavior description is aligned with the interaction. In the training stage, the difference between the behavior description B output by the regression head and the ground truth B is calculated through the cross-entropy loss function L , and the parameters are updated through backpropagation.

[0072] In the "behavior-trajectory" translation in the second stage, the model translates the behavioral text description of the vehicle in B into the trajectories of N vehicles During the training process, vehicle trajectories are texturized for the training and inference of large language models. To control token consumption, each floating-point number is uniformly retained to two decimal places. The texturized trajectory points still contain four attribute values. In Figure 2 (b), the translation process of the second stage is introduced. The text segment of behavior B is input into the large language model , and the behavior features are also transformed by the text embedding layer ε :

[0073] F B = ε(B).

[0074] Here, random local attention is also used to capture the local correlation between the behavior feature F B and the interaction feature , and the correlation score is expressed as

[0075]

[0076] where K < O. Similarly, the correlation score is applied to weight the interaction feature :

[0077]

[0078] In the above formula, variables from the Gaussian distribution are also randomly sampled. By calculating the behavior feature of the nth vehicle, the random local attention module can focus on the most relevant interaction features and the map information contained therein. The random variable will also perturb the behavior feature so that the model can generate vehicle trajectories that match the text description and map constraints, while also having diversity.

[0079] The behavior features of N vehicles are input into the regression head to generate the trajectories in T:

[0080]

[0081] where represents the output trajectory. During the generation process, it will query the behavior information in . During training, the cross-entropy loss function L T is used to calculate the difference between the generated trajectory T and the ground truth , and the parameters are updated through backpropagation.

[0082] (III) Training the neural network model

[0083] The above neural network model is trained, and the cross-entropy loss is used to constrain the prediction of behavior and trajectory text during the two-stage training process. It should be emphasized that during the fine-tuning training for the large language model, only the parameters of the proposed random local attention module and LoRA module are trained, and the parameters of the segment encoder and the pre-trained large language model are frozen. This can reduce the video memory occupancy and improve the training efficiency at the same time, enabling the large language model to better learn the correlation between trajectory and scene context features and adapt to the trajectory generation task.

[0084] (4) Generate a trajectory that conforms to the text description

[0085] After fine-tuning training with a high-quality dataset, the model can output a vehicle trajectory that conforms to the given text description for the input combination of text description, map, and initial trajectory. The model can also adapt to other datasets, as long as they are organized in the format introduced in the data input. In addition, thanks to the large language model's basic understanding of world knowledge, the model can also support precise trajectory parameter control through text. Due to the randomness of the random local attention module, the diversity of the generated trajectories is also better than other methods in multiple predictions for the same input.

[0086] The feasibility of the method of the present invention is verified below with specific examples, as detailed in the following description: 2]

[0087] The present invention uses the WOMD dataset [4] , which is a large real-world scenario dataset containing rich traffic scenarios and vehicle interactions. Each scenario in the dataset has corresponding vehicle trajectory and map information. The dataset provides 1-second historical trajectories and 8-second future trajectories, with a sampling frequency of 10HZ. The entire dataset contains 487K training scenarios, approximately 44K validation scenarios, and 44K test scenarios. Each scenario has at most 8 objects for which the trajectory needs to be predicted.

[0088] In Figure 3 , the present invention is compared with the recent image-based, rule-based, and text-based trajectory generation methods SimNet [5] , TrafficSim [6] , BITS [7] , CTG [8] , LCTGen

[10] , CTG++ [9] in terms of the authenticity and diversity of the generated trajectories.

[0089] Authenticity: Using the paper CTG++ [9]The scene-level authenticity metric defined in [reference] represents authenticity by calculating the Wasserstein distance between the generated trajectory and the normalized histogram of the ground truth trajectory. It mainly compares four attribute differences: longitudinal acceleration magnitude (LO), latitudinal acceleration magnitude (LA), jerk (JE), yaw rate (YR), and the average of the differences (AVG).

[0090] Diversity: Use the scene-level diversity metric defined in paper BITS [7] The scene-level diversity metric defined in [reference] uses kernel density estimation to calculate the trajectory density distribution for each scene. Then, it calculates the Wasserstein distance (WD) between each pair of density distributions and takes the mean as the final metric. To calculate diversity, for the same input, each model is inferred three times, and the generated results of the three inferences are used for calculation.

[0091] In Figure 3 the last row of [reference], the performance improvements of the present invention in terms of the authenticity and diversity of the generated trajectories compared to other methods are shown, achieving better performance in each metric.

[0092] Therefore, the method of the present invention has a powerful ability to understand vehicle interactions and output their behaviors and driving logics, enabling the model of the present invention to generate realistic and diverse trajectory data.

Claims

1. A trajectory generation method based on natural language control, characterized in that, It includes the following steps: Step 1: Extract the interactive relationship text description, vehicle initial trajectory, and map topology in the dataset scenario as inputs; Step 2: Perform feature encoding on the input data belonging to different modalities. Apply a line segment encoder to encode the vehicle initial trajectory and map topology into trajectory features and map features, and apply the text embedding layer of the large language model to convert the interactive relationship text description into text features; Step 3: After obtaining the text features, trajectory features, and map features, apply the random local attention module to fuse the text features with the trajectory features and the text features with the map features respectively, thereby obtaining text-trajectory fusion features and text-map fusion features; Among them, for the two parts of the original features that need to be fused by the random local attention module, the fusion features of the two parts of the original features are obtained, including the following steps: Obtain the correlation scores between the two parts of the original features of different interactive objects through cross-attention calculation; Select the relevant segments with the top correlation scores as the local enhancement features of the original features; Use the obtained local enhancement features of the original features, introduce a random variable of normal distribution as a random perturbation coefficient, and weight the original features to obtain the fusion features of the two parts of the original features; Step 4: Concatenate the text-trajectory fusion features, text-map fusion features, and text features to jointly form interactive features, and input them into the large language model to achieve the fusion of different modality features in the first stage and obtain behavior outputs; Step 5: Output the behavior description of the vehicle by the large language model, and intercept the corresponding implicit features from the regression layer of the large language model as behavior features; Step 6: Apply the random local attention module proposed in Step 3 again to fuse the interactive features obtained in Step 4 and the behavior features obtained in Step 5, and record the fused features as interactive-behavior fusion features; Step 7: Concatenate the interactive-behavior fusion features obtained in Step 6 with the text features and input them into the large language model to achieve the fusion of different modality features in the second stage and obtain trajectory outputs; Step 8: Use the dataset for model training. Calculate the cross-entropy loss between the trajectory output by the large language model in Step 7 and the real trajectory in the dataset during training, and use the gradient descent method to update the model parameters to finally obtain a trained neural network model; Step 9: Use the trained neural network model to output the vehicle trajectory in the current scenario.

2. The trajectory generation method based on natural language control according to claim 1, wherein In Step 2, when performing feature encoding on the input data belonging to different modalities, applying a line segment encoder to encode the vehicle initial trajectory and map topology into trajectory features and map features, and applying the text embedding layer of the large language model to convert the interactive relationship text description into text features, includes: (1) Feature encoding of the text: The input text description I is tokenized to generate a sequence containing L tokens. Then, through the text embedding layer ε of the large language model, each token is mapped to a d i -dimensional vector, thus obtaining a text embedding matrix of size [L, d i , which serves as the text feature F I ; (2) Feature Encoding of Map Topology: For the map input M, a line-based vectorization representation method is adopted. Each road element is composed of line segments, that is, an ordered vertex sequence. Through the line segment encoder each line segment is converted into a d m -dimensional vector, obtaining a feature matrix representing the map topology, which is used as the map feature F M ; (3) Feature Encoding of Vehicle Trajectories: For the historical trajectories X of N vehicles in the scene, each trajectory is regarded as composed of line segments between trajectory points, and through the line segment encoder is converted into a trajectory feature matrix as the trajectory feature F X .

3. A trajectory generation method based on natural language control according to claim 1, characterized in that For the two parts of the original features A and B that need to be fused by the random local attention module, the fusion features of the two parts of the original features are obtained, including the following calculation steps: (1) Calculate the feature correlation score matrix S between the original features A and B through the cross-attention formula. First, calculate the dot product of the features A·B T , and then divide it by the square root of the feature dimension d for numerical scaling. Finally, convert the dot product result into a probability distribution through softmax and regard this as the correlation score between the features; (2) Apply the topK algorithm to select the top K segments with the highest correlation scores from the correlation score matrix S as the local enhanced features S of the original features topK = top k (S); (3) Utilize the local enhanced feature S topK , introduce a random variable of normal distribution as the random perturbation coefficient, weight the original features A and B, and obtain the final fused feature 4. A trajectory generation method based on natural language control according to claim 1, characterized in that, The large language model is LLaMA.

5. A trajectory generation method based on natural language control according to claim 1, characterized in that The fusion of different modality features in both stages is performed through the regression head of the large language model for text mapping, and the difference between the generated text and the real text is constrained by the cross-entropy loss during the training process.

6. A trajectory generation method based on natural language control according to claim 5, characterized in that The method of using the cross-entropy loss to constrain the difference between the generated text and the real text during the training process is: (1) For the behavior output, use the cross-entropy loss L B Calculate the difference between the generated trajectory B and the ground truth and update the model parameters based on the backpropagation mechanism; (2) For trajectory output, use the cross-entropy loss L T Calculate the difference between the generated trajectory T and the ground truth and update the model parameters based on the backpropagation mechanism.

7. A trajectory generation method based on natural language control according to claim 1, characterized in that In the dataset training of step eight, using the LoRA parameter fine-tuning method, the modules for updating parameters include the LoRA low-rank matrix module and the random local attention module of the large language model.

Images