Multi-modal large model training and deployment method for automatic driving

Abstract

The invention discloses a multi-modal large model training and deployment method for automatic driving, and the method comprises the following steps: S1, constructing a multi-modal data set which covers an image frame sequence, a navigation instruction, a vehicle state and thinking chain reasoning data; s2, using the data set to supervise and finely adjust the VLA large model of the cloud; s3, continuing to perform optimization by using reinforcement learning in the simulation environment; and S4, through knowledge distillation, the capability of the cloud large model is migrated to the vehicle-end lightweight VLA large model for deployment. According to the multi-modal large model training and deployment method for automatic driving provided by the invention, the comprehensive performance and generalization ability of an automatic driving system can be improved, and reasonable explanation can be made for the made track to realize a reliable decision; the method is superior to a traditional scheme in the aspects of model performance, generalization, real-time performance, deployment cost and the like, and shows remarkable comprehensive technical advantages.

Description

Technical Field

[0001] This invention relates to a method for training and deploying large models, and more particularly to a method for training and deploying multimodal large models for autonomous driving. Background Technology

[0002] Autonomous driving systems need to handle complex environments and gain a deep understanding of traffic scenarios, but current technologies struggle to meet the challenging demands of autonomous driving. Traditional autonomous driving methods often employ modular designs, dividing autonomous driving tasks into perception, localization, planning, and control. While offering advantages such as strong interpretability and simple independent module verification, they also suffer from information bottlenecks and error accumulation. In modular schemes, errors from upstream modules are amplified at each level, and human-defined rules are insufficient to cover various long-tail scenarios, limiting the system's adaptability and robustness. In recent years, end-to-end learning methods have attempted to directly map sensor data to control output using a single neural network, reducing module interface conflicts and potentially increasing performance ceilings. However, end-to-end models are essentially black-box decision-making, lacking interpretability, failing to meet safety verification requirements, and demanding extremely high data coverage of a large number of complex scenarios. While they alleviate the long-tail problem of modularization to some extent, they still cannot truly solve the problem.

[0003] With the rapid development of multimodal large models and large language models, the Vision-Language-Action (VLA) paradigm has become a new trend in autonomous driving research. VLA models connect visual input, language commands, and action outputs into a closed-loop system through unified modeling. VLA possesses cross-modal understanding and multi-task generalization capabilities, performing well in various application scenarios. Compared to traditional algorithms, it can solve more long-tail problems and is expected to become the foundation for higher-level intelligent agents. Furthermore, many scholars believe that autonomous driving is upgrading from purely data-driven to knowledge-driven, and the introduction of large models such as LLM and VLM is a key step in realizing this paradigm shift. Against this backdrop, traditional modular design and end-to-end approaches cannot meet the ever-increasing demands for intelligence. To fully utilize the reasoning and knowledge representation capabilities of large models, innovation in model design and training methods is necessary.

[0004] As can be seen from the above, it is necessary and valuable to use multimodal large models to improve autonomous driving capabilities. It can not only improve adaptability to extreme and long-tail scenarios by leveraging pre-trained knowledge, but also provide reasonable explanations for its output planning. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for training and deploying multimodal large models for autonomous driving, which can not only improve the overall performance and generalization ability of autonomous driving system, but also make reasonable interpretations of the trajectory and achieve reliable decision-making.

[0006] To address the aforementioned technical problems, this invention provides a method for training and deploying a multimodal large model for autonomous driving, comprising the following steps: S1, constructing a multimodal dataset covering image frame sequences, navigation commands, vehicle states, and thought chain inference data; S2, using the dataset to perform supervised fine-tuning of the cloud-based VLA large model; S3, continuing optimization using reinforcement learning in a simulation environment; S4, transferring the capabilities of the cloud-based large model to a lightweight vehicle-side VLA large model for deployment through knowledge distillation.

[0007] Further, step S1 includes: S11, simultaneously acquiring multiple sensor data and control signals in a simulated or real vehicle environment; S12, cleaning and formatting the acquired raw data, cropping and scaling the image sequence to a uniform resolution, and aligning the data acquisition time; S13, labeling the thought chain reasoning data, including scene and key object descriptions, driving intention descriptions, and outputting the driving trajectory.

[0008] Further, step S11 uses control commands and driving trajectories provided by the simulation environment, or records control inputs via CAN bus and vehicle driving trajectories via GPS / IMU in a real vehicle scenario; step S12 uses differential equation modeling to fill in or align vehicle speed and acceleration with the timestamps of image frames over time; or matches the modal data through averaging / interpolation; the preprocessed data sample format is as follows: , It is the image sequence at time t. It is the navigation command at time t. It is the state of the vehicle at time t. The thought chain reasoning data at time t; step S13 gives the driving intention based on the given vehicle's own state, driving scenario and key objects, and outputs the driving trajectory for the next 5 seconds.

[0009] Further, step S2 includes: S21, constructing training input and supervision data: the training samples are in the following form: It is the image sequence at time t. It is the navigation command at time t. It is the state of the vehicle at time t. This is the reasoning data of the thought chain at time t; S22, let the model policy be... , Let the model parameters be the COT sequence output by the model. Then the loss function is defined as: ； Where C, N, and S represent the image, navigation command, and vehicle status, respectively; S23, Model Training and Optimization: During training, the image, command, and status are input into the model, the predicted probability of the output action is calculated, and compared with the expert labels.

[0010] Furthermore, weight decay is added as a regularization term to the loss function to prevent overfitting. Step S23 uses mini-batch stochastic gradient descent, with each batch containing several synchronized multimodal samples. During training, learning rate scheduling and gradient clipping are used to maintain training stability. The training termination condition is model convergence or stable performance on the validation set.

[0011] Further, step S3 includes: S31, defining the state space and action space: setting the input data of the model to include the current image frame sequence C, navigation instructions N, and vehicle state S, the state space Q is equivalent to the set of multimodal data observations; the action space A is defined as the planned trajectory of the vehicle in the next five seconds, and the action space A is the set of driving trajectories output by the VLA model; S32, designing a reward network to evaluate the actions, and calculating the weighted average using the following formula: NC represents collision-free driving for road users, DAC represents compliance with drivable zones, EP represents self-progression, TTC represents time to collision, and C represents comfort; S33, iteratively optimize the strategy: for a specific scenario, input q, Using the old strategy network G trajectories were generated by sampling. Each trajectory is scored using a reward network. Calculate the advantage of each trajectory .

[0012] Furthermore, the reward network is an output head of the last layer of the VLA large model, outputting a reward value r.

[0013] Furthermore, in step S4, the cloud-based large model is used as the teacher model, and the lightweight VLA large model on the vehicle side is used as the student model, and the deployment is carried out through a multi-level knowledge distillation process.

[0014] Furthermore, the teacher model and student model adopt a completely consistent model structure and transfer teacher knowledge through multiple distillation objectives, specifically including: soft label alignment: using the class probability distribution output by the teacher model under the same input as a soft label, so that the student model matches the output distribution of the teacher model; intermediate layer feature alignment: using L2 loss or cosine similarity loss, so that the hidden feature map of the corresponding layer of the student model is close to the feature map of the teacher model; attention map alignment: extracting the attention weight map of each layer in the Transformer, constructing a loss to make the attention map of the student model similar to that of the teacher model.

[0015] Furthermore, the total loss during the distillation process in step S4 is a weighted sum of the losses of each target; the distillation loss for soft label alignment is given by the following formula: ; Denotes KL divergence, Let represent the category probability distributions output by the teacher model and the student model, respectively; the intermediate feature layer distillation loss function is as follows: ;in and Let be the teacher model features and student model features of the i-th layer, respectively; the attention distillation loss is given by the following formula: ;in and Let represent the Transformer attention matrices for the teacher and student models, respectively; the total distillation loss is a linear combination of these components. ;in , Control the contributions of different objectives.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The multimodal large model training and deployment method for autonomous driving provided by the present invention constructs a large model with more parameters in the cloud and strengthens it through supervised fine-tuning (SFT) and relative policy optimization (GRPO), and then transfers the capabilities to the vehicle-side lightweight model through knowledge distillation. This ensures the accuracy of the model while taking into account the deployment efficiency on the edge, effectively solving the problems encountered by the current traditional algorithms. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the training and deployment process of a multimodal large model for autonomous driving according to the present invention. Figure 2 This is a schematic diagram of the multimodal large model structure used in this invention. Detailed Implementation

[0018] The present invention will now be further described with reference to the accompanying drawings and embodiments.

[0019] Figure 1 This is a schematic diagram of the structure of the multimodal large model training and deployment method for autonomous driving according to the present invention.

[0020] Please see Figure 1 This invention provides a method for training and deploying a multimodal large-scale model for autonomous driving, including dataset construction, supervised fine-tuning of the VLA model (SFT), reinforcement learning, and model distillation. As shown in the figure below, firstly, a multimodal training dataset is constructed; then, the VLA large-scale model in the cloud is supervisedly fine-tuned using the dataset; next, reinforcement learning is performed; and finally, knowledge distillation is used to transfer the capabilities of the cloud-based large-scale model to a lightweight VLA large-scale model on the vehicle for deployment.

[0021] The main process of this invention will be described in detail below.

[0022] I. Dataset To train the VLA model described above, a multimodal dataset needs to be constructed, encompassing four types of information: image frame sequences, navigation commands, vehicle status, and thought chain of reasoning (COT) data. The construction process generally includes the following steps: Data Acquisition: Simultaneously acquire data from multiple sensors and control signals in simulation or real-vehicle environments. Taking autonomous driving scenarios as an example, simulators (such as CARLA) or real vehicle platforms can be used to set up cameras, GPS / IMU sensors, and onboard CAN bus recording modules. The system acquires continuous image frames at a fixed frequency (e.g., 10 frames per second). Navigation commands can be obtained through map navigation instructions during vehicle operation or through manual annotations (e.g., straight ahead, left turn, right turn). Simultaneously, vehicle status (position, speed, acceleration, throttle, brake, steering wheel angle, etc.) is recorded. Vehicle status and driving trajectory are derived from expert driver demonstrations. In simulations, control commands and driving trajectories provided by the simulation environment can be directly used. In real-vehicle scenarios, control inputs (accelerator pedal, brake, steering wheel angle, etc.) can be recorded via the CAN bus, and the vehicle's driving trajectory can be recorded via GPS / IMU.

[0023] Data preprocessing: The collected raw data is cleaned and formatted. Image sequences need to be cropped and scaled to a uniform resolution. Since the sampling rates of different modalities may vary, time axis alignment is required: This can be achieved by using differential equation modeling to interpolate or align vehicle speed and acceleration over time to the timestamps of image frames; alternatively, mean / interpolation can be used to match the data across modalities. The preprocessed data sample format is as follows: ; It is the image sequence at time t. It is the navigation command at time t. It is the state of the vehicle at time t. This is the thought chain (COT) inference data at time t, which needs to be labeled.

[0024] COT (Consciousness-Oriented) reasoning data annotation: Annotation consists of three parts: 1. Scene and key object description, 2. Driving intention description, and 3. Driving trajectory. The first part requires correctly describing the current driving scene based on the current image, and identifying objects that may affect the vehicle's movement (e.g., pedestrians, vehicles, traffic lights, speed limit signs, ground traffic markings, etc.). 2. Based on the given vehicle state, driving scene, and key objects, the driving intention is given. 3. The driving trajectory for the next 5 seconds is output. This data can be directly discretized from the expert-demonstrated driving trajectory into the vehicle's coordinate system to obtain a series of trajectory points. Where x and y represent the positions in the vehicle coordinate system. This indicates the orientation angle. The first and second parts need to be implemented through manual annotation, or by using a large model for auxiliary annotation and then manually modifying it.

[0025] In summary, by collecting data, preprocessing data, and annotating with experts, a complete multimodal training set was constructed, which can support supervised fine-tuning and reinforcement learning of large VLA models.

[0026] II. VLA Model and Supervised Fine-Tuning The VLA model itself can utilize open-source visual language large-scale models such as Qwen-VL and llama3.2-Vision. In general, the model structure is as follows: Figure 2 As shown below, a detailed explanation will follow.

[0027] VLA Model: The model input includes images, navigation commands, and vehicle status. Navigation commands are the planned instructions for the current scene (straight ahead, left turn, right turn, etc.). Vehicle status includes the current vehicle speed and the driving trajectory of the previous 5 seconds. Image data is processed by a visual encoder to obtain a visual token, which is then fused with historical visual tokens through a mutual attention mechanism to obtain a fused visual token. Navigation commands and vehicle status are processed by a text encoder to obtain a text token. The fused visual token and text token are fed into the VLA large model, and the final output is the COT token. The COT token contains three parts: 1. Scene and key object description, 2. Driving intention description, and 3. Driving trajectory. 1 and 2 are collectively referred to as language output, and 3 is referred to as trajectory output. Obtaining the trajectory output enables actual vehicle control. It can be described by the following formula: ； {L,T} represent the language output and trajectory output, respectively, while CNS represent the input image, navigation command, and vehicle status, respectively. For the Visual-Language-Motion (VLA) large model.

[0028] Supervised fine-tuning (SFT): Based on the VLA model pre-training, supervised fine-tuning (SFT) is required to enable the model to learn specific perception-decision mappings. SFT uses the dataset constructed above to perform behavior cloning training. Its main steps and key points include: 1. Training Input and Supervision Data Construction: The training sample format in the SFT stage is as follows: ; It is the image sequence at time t. It is the navigation command at time t. It is the state of the vehicle at time t. This is the thought chain (COT) inference data at time t. The supervised data comes from labeled COT thought chain inference data. The data should cover a variety of driving scenarios and command types to enhance the model's generalization ability.

[0029] 2. Loss Function: The goal of SFT is to maximize the probability of the model generating COT inference data, which is equivalent to minimizing the negative log-likelihood loss. Let the model policy be... , Let the model parameters be the COT sequence output by the model. Then the loss function can be defined as: ； Where C, N, and S represent the image, navigation commands, and vehicle status, respectively. A regularization term (such as weight decay) can also be added to the loss function design to prevent overfitting.

[0030] 3. Model Training and Optimization: During training, modal inputs such as images, commands, and states are fed into the model, and the predicted probabilities of the output actions are calculated and compared with expert labels. Common optimizers are Adam or AdamW for fine-tuning the VLA model. Mini-batch stochastic gradient descent is typically used, with each batch containing several synchronized multimodal samples. During training, techniques such as learning rate scheduling (e.g., linear decay) and gradient clipping can be used to maintain training stability. Training termination conditions can be model convergence or stability of metrics on the validation set.

[0031] The SFT stage fully leverages expert data for supervised learning of the model, achieving behavior cloning. During training, the model directly mimics the expert's decision-making patterns without requiring additional reward feedback. The diversity of expert data determines the model's robustness to complex scenarios; therefore, the data construction should cover as many roads, traffic rules, and navigation tasks as possible. In summary, the SFT stage aims to use supervised learning to help the model fit the state-to-action mapping. Its training objective function mainly consists of negative log-likelihood loss, and the model parameters are iteratively updated using an Adam-type optimizer.

[0032] III. Strengthen Learning After supervised fine-tuning, this invention further optimizes the model using reinforcement learning in a simulation environment to improve its performance in extreme scenarios that open-loop training cannot cover. The Group Relative Policy Optimization (GRPO) algorithm is used for reinforcement training, and its process and key points include: State space and action space definitions: In the GRPO algorithm, each state is the input data of the model, including the current image frame sequence C, navigation instructions N, and vehicle state S. Therefore, the state space Q is equivalent to the set of multimodal data observations. The action space A is defined as the planned trajectory of the vehicle in the next five seconds, which is the set of driving trajectories output by the VLA model.

[0033] Reward Network Construction: A reward network needs to be designed to evaluate the quality of actions. This step can directly use the PDMS score proposed by NAVSIM as supervision to train the reward network. The PDMS calculation formula is shown below: ； It represents the collision-free driving (NC) and drivable area compliance (DAC) of road users, as well as a weighted average of factors including self-progression (EP), time to collision (TTC), and comfort (C). This comprehensively evaluates the quality of a trajectory. The reward network does not require separate design; it only needs to be designed as an output head in the last layer of the original VLA large model, outputting the reward value r.

[0034] Policy update process (GRPO algorithm): The policy is iteratively optimized using Group Relative Policy Optimization. Its basic process is as follows: For a given scenario, input q... Using the old strategy network G trajectories were generated by sampling. Each trajectory is scored using a reward network. Calculate the advantage of each trajectory For the current policy network Optimize using the following formula: ； ； ； It is the KL divergence. and These are hyperparameters that control the cutoff range and the KL divergence weights, respectively. It is the mean of the scores for this set of trajectories. This represents the standard deviation of the trajectory scores. The method proposed above effectively optimizes the VLA model through reinforcement learning, thereby further improving the model's performance in complex driving tasks.

[0035] IV. Model Distillation To transfer knowledge from a cloud-trained VLA large model (teacher model) to a more streamlined VLA large model (student model) for deployment, this invention designs a multi-layered knowledge distillation process. Its key components include: Architecture alignment strategy: The teacher model and the student model need to maintain a certain degree of correspondence in their architecture. In this invention, the teacher model and the student model adopt a completely consistent model structure, except that the student model has fewer parameters than the teacher model, making it suitable for deployment on the edge.

[0036] Distillation Objectives: This invention designs multiple distillation objectives to transfer teacher knowledge. These include: soft-logit alignment, intermediate-layer feature alignment, and attention map alignment. Soft-logits: The class probability distribution output by the teacher model under the same input is used as a soft label, enabling the student model to match the teacher model's output distribution. Intermediate-layer feature alignment: L2 loss or cosine similarity loss is used to make the hidden feature maps of corresponding layers in the student model closely resemble those of the teacher model. Attention map alignment: Attention weight maps of each layer in the Transformer are extracted, and a loss is constructed to make the student model's attention map similar to the teacher model's. These distillation objectives can be used individually or in combination; multiple objectives help students comprehensively learn the teacher's knowledge.

[0037] Loss function: The total loss during the distillation process is typically a weighted sum of the losses for each objective. The distillation loss for soft label alignment can be expressed as follows: ； Denotes KL divergence, Let represent the class probability distributions output by the teacher model and the student model, respectively. The intermediate feature layer distillation loss function can be written as follows: ； in and Let be the teacher model features and student model features of the i-th layer, respectively. The attention distillation loss can be written as the following formula: ； in and Let represent the Transformer attention matrices for the teacher and student models, respectively. In summary, the total distillation loss is a linear combination of the components: ; in , Control the contributions of different objectives.

[0038] Overall Distillation Optimization Strategy: During the training phase, the parameters of the teacher model are typically frozen first, using its output as a fixed "teacher signal." The student model is trained on the same or similar data as the SFT phase, minimizing both the original task loss and the distillation loss. The same dataset as the SFT phase can be used for distillation to maintain task consistency, or more diverse data can be introduced to improve distillation performance. The entire training process is similar to joint training: in each mini-batch, the student model learns to predict expert labels and also learns the teacher's implicit knowledge through comparison with the teacher. After distillation, the student model can approximate the performance of the teacher model with a smaller number of parameters, accelerating inference and reducing deployment costs.

[0039] The scope of protection of this invention includes, but is not limited to, the following: VLA model structure design: Based on the visual-language multimodal large model, a unified visual-language-action model framework is constructed, which includes key components such as visual encoder, text encoder, and LLM; this structure supports the fusion of multi-frame images and natural language commands to directly output vehicle trajectory and driving intention.

[0040] SFT Supervised Fine-Tuning Process: Supervised learning is used in the cloud to fine-tune the model, including building multimodal driving scenario data, defining loss functions, and specific optimization strategies and training details; this process enhances the pre-trained model's ability to jointly reason about visual and linguistic inputs in autonomous driving scenarios.

[0041] The GRPO reinforcement fine-tuning method employs the Group Relative Policy Optimization (GRPO) algorithm to fine-tune the VLA model using reinforcement learning. This includes steps such as state representation, reward function design, and a policy update mechanism based on within-group relative scores. The GRPO algorithm eliminates the state-value network of the PPO algorithm, reducing training overhead. This fine-tuning method, combined with SFT, enables the model to achieve superior decision-making performance in real-world driving environments.

[0042] Knowledge distillation scheme: Implement knowledge distillation between the cloud-based teacher model and the vehicle-based student model, including the design of distillation objectives based on the teacher model (such as distillation of soft logits, feature layers, and attention maps) and the design of corresponding loss functions, to ensure that the student model learns the reasoning ability and feature representation of the teacher model; this enables the vehicle-based model to achieve the driving capabilities of the cloud model under limited computing resources.

[0043] Overall Technical Solution: The overall technical solution includes the VLA training and deployment process, specifically comprising cloud-based model SFT fine-tuning, GRPO reinforcement learning, and on-device model deployment via knowledge distillation. Each stage forms a closed loop, continuously optimizing model performance and generalization ability. The above points embody the innovative ideas and technological combinations of this invention and can be considered key aspects for patent protection.

[0044] This invention offers significant advantages over traditional autonomous driving methods. Firstly, in terms of performance and generalization ability, due to the large scale of the cloud-based model and its integration of multimodal large-model inference capabilities, this invention achieves more comprehensive scene understanding and decision-making capabilities, demonstrating superior performance across various driving scenarios. Through SFT+GRPO training, this invention can understand more complex traffic scenarios and advanced commands, generating reliable driving trajectories. Furthermore, this invention excels in multimodal fusion accuracy and safety. Through a unified multimodal architecture and aligned training, the system can more accurately map visual information to language commands, reducing misunderstandings and erroneous executions. Even in extreme conditions such as rain, fog, and nighttime, the distilled model maintains high accuracy in recognizing and predicting the behavior of others, improving the system's robustness and safety. Compared to traditional modular and purely end-to-end methods, this invention combines the advantages of both: maintaining a certain level of interpretability (explaining the reasons for the planned path) while utilizing large-model knowledge to improve path planning quality, thus achieving a better balance between safety and generalization. Finally, this invention exhibits significant advantages in cloud-vehicle collaboration. The cloud-based large-scale model acts as the "teacher," utilizing powerful computing capabilities and massive amounts of simulation / real-world data for training. The vehicle-side model, after distillation, inherits the "experience" and reasoning abilities of the teacher model, while maintaining low operating costs, thus realizing the "large-scale model on-vehicle" pathway. This "training ground-combat soldier" model accelerates technology deployment, allowing the vehicle-side to continuously benefit from the iterative updates of cloud-based algorithms. In summary, this invention outperforms traditional solutions in terms of model performance, generalization, real-time performance, and deployment costs, demonstrating significant comprehensive technical advantages.

[0045] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. A method for training and deploying a multimodal large model for autonomous driving, characterized in that, Includes the following steps: S1. Construct a multimodal dataset, covering image frame sequences, navigation commands, vehicle status, and thought chain reasoning data; S2. Use the dataset to perform supervised fine-tuning of the large VLA model in the cloud; S3. Continue to optimize using reinforcement learning in the simulation environment; S4. Migrate the capabilities of the cloud-based large model to the lightweight VLA large model on the vehicle side for deployment through knowledge distillation.

2. The method for training and deploying a multimodal large model for autonomous driving as described in claim 1, characterized in that, Step S1 includes: S11. Simultaneously acquire multiple sensor data and control signals in a simulated or real vehicle environment; S12. Clean and format the collected raw data, crop and scale the image sequence to a uniform resolution, and align the data acquisition time. S13. Label the thought chain reasoning data, including scene and key object descriptions, driving intention descriptions, and output driving trajectories.

3. The method for training and deploying a multimodal large model for autonomous driving as described in claim 2, characterized in that, Step S11 uses control commands and driving trajectories provided by the simulation environment, or records control inputs via CAN bus and vehicle driving trajectories via GPS / IMU in a real vehicle scenario; Step S12 uses differential equation modeling to fill in or align vehicle speed and acceleration with the timestamps of image frames over time; or it uses averaging / interpolation to match the modal data; the preprocessed data sample format is as follows: , It is the image sequence at time t. It is the navigation command at time t. It is the state of the vehicle at time t. This is the reasoning data of the thought chain at time t; Step S13 gives the driving intention based on the given vehicle's own state, driving scenario and key objects, and outputs the driving trajectory for the next 5 seconds.

4. The method for training and deploying a multimodal large model for autonomous driving as described in claim 1, characterized in that, Step S2 includes: S21. Construction of Training Input and Supervision Data: The training sample format is as follows: It is the image sequence at time t. It is the navigation command at time t. It is the state of the vehicle at time t. This is the reasoning data of the thought chain at time t; S22, Let the model strategy be , Let the model parameters be the COT sequence output by the model. Then the loss function is defined as: ； Where C, N, and S represent the image, navigation command, and vehicle status, respectively; S23. Model Training and Optimization: During training, images, instructions, and state inputs are fed into the model, the predicted probability of the output action is calculated, and compared with the expert labels.

5. The method for training and deploying a multimodal large model for autonomous driving as described in claim 4, characterized in that, The loss function incorporates weight decay as a regularization term to prevent overfitting. Step S23 employs mini-batch stochastic gradient descent, with each batch containing several synchronized multimodal samples. During training, learning rate scheduling and gradient clipping are used to maintain training stability. Training terminates when the model converges or the metrics stabilize on the validation set.

6. The method for training and deploying a multimodal large model for autonomous driving as described in claim 1, characterized in that, Step S3 includes: S31. Definition of State Space and Action Space: The input data of the model is defined as the current image frame sequence C, navigation command N, and vehicle state S. The state space Q is equivalent to the set of multimodal data observations. The action space A is defined as the planned trajectory of the vehicle in the next five seconds. The action space A is the set of driving trajectories output by the VLA model. S32. Design a reward network to evaluate actions, and calculate the weighted average using the following formula: ； NC stands for Road User Collision-Free Driving, DAC stands for Drivable Area Compliance, EP stands for Self-Progression, TTC stands for Time to Collision, and C stands for Comfort. S33. Iteratively optimize the strategy: For a given scenario, input q, Using the old strategy network G trajectories were generated by sampling. Each trajectory is scored using a reward network. Calculate the advantage of each trajectory .

7. The method for training and deploying a multimodal large model for autonomous driving as described in claim 6, characterized in that, The reward network is an output head of the last layer of the VLA large model, outputting a reward value r.

8. The method for training and deploying a multimodal large model for autonomous driving as described in claim 1, characterized in that, Step S4 uses the cloud-based large model as the teacher model and the vehicle-side lightweight VLA large model as the student model, and deploys them through a multi-layered knowledge distillation process.

9. The method for training and deploying a multimodal large model for autonomous driving as described in claim 8, characterized in that, The teacher and student models employ a completely identical model structure and transfer teacher knowledge through multiple distillation objectives, specifically including: Soft label alignment: The class probability distribution output by the teacher model under the same input is used as a soft label to match the output distribution of the student model. Intermediate layer feature alignment: L2 loss or cosine similarity loss is used to make the hidden feature maps of the corresponding layers of the student model close to the feature maps of the teacher model; Attention map alignment: Extract the attention weight maps of each layer in the Transformer and construct a loss that makes the attention map of the student model similar to that of the teacher model.

10. The method for training and deploying a multimodal large model for autonomous driving as described in claim 9, characterized in that, The total loss during the distillation process in step S4 is the weighted sum of the losses of each target; the distillation loss for soft label alignment is given by the following formula: ； Denotes KL divergence, These represent the category probability distributions output by the teacher model and the student model, respectively. The intermediate feature layer distillation loss function is as follows: ； in and These are the teacher model features and student model features of the i-th layer, respectively; Attention distillation loss is given by the following formula: ； in and Let represent the Transformer attention matrices for the teacher and student models, respectively; the total distillation loss is a linear combination of these components. ； in , Control the contributions of different objectives.

Images