An end-to-end automatic driving training method and system based on a multi-modal large model

Abstract

The application discloses an end-to-end automatic driving training method and system based on a multi-modal large model. The method collects driving data through real vehicles or shadow mode, and deploys to a closed-loop simulation environment for evaluation to detect failure scenarios, and then classifies the root causes and generates an improvement plan containing a to-be-reinforced training stage identifier; based on the failure scenarios and their root causes, the world model imagination expansion and directional scenario variation are performed in parallel to generate imagination data and synthetic data; at least using the data, according to the improvement plan, the corresponding stage is selected from a multi-stage progressive training strategy for retraining, and iteration is performed until convergence. The application realizes the closed-loop evolution of automatic model failure detection, directional training data generation and targeted retraining, significantly reduces manual intervention, and improves the generalization and planning safety in the long-tail scenario.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology for autonomous driving, and in particular to an end-to-end autonomous driving training method and system based on a multimodal large model. Background Technology

[0002] With the widespread application of Vision-Language-Action (VLA) multimodal large models in the field of autonomous driving, end-to-end autonomous driving systems have initially achieved unified modeling of visual perception, language reasoning, and trajectory planning, demonstrating significant advantages in generalization and human alignment. However, existing VLA model training methods generally adopt the supervised fine-tuning (SFT) paradigm, which relies on pre-collected labeled datasets for one-time or limited training rounds, after which the model parameters are fixed and deployed. This offline training mode results in the model lacking the closed-loop evolutionary ability to continuously mine its own failure scenarios from real road data or simulation interactions and generate targeted training data for self-improvement. Specifically, when existing systems detect planning errors, collision risks, or inference errors in long-tail scenarios during simulation evaluation or real-vehicle testing, they cannot automatically transform the failure feedback into effective training signals. They cannot accurately locate the capability dimensions that need to be strengthened based on the root causes of failure, nor can they generate novel training samples that cover the failure boundary region at low cost. Furthermore, they lack a decision-making mechanism to call specific training stages for targeted retraining on demand. As a result, the performance improvement of the model heavily relies on manual analysis of failure cases and manual expansion of the dataset, which is inefficient and costly. Especially in the face of rare long-tail scenarios, the improvement is slow and it is difficult to achieve continuous self-evolution of the model. Summary of the Invention

[0003] In view of the above-mentioned shortcomings in the current field of autonomous driving artificial intelligence technology, this invention provides an end-to-end autonomous driving training method and system based on a multimodal large model. It automatically detects and classifies failure scenarios, performs world model imagination expansion and directional scenario mutation in parallel to generate training data for failure boundary regions, and dynamically selects the corresponding training stage for targeted retraining according to the improvement plan. Through iteration, a closed-loop evolution mechanism is formed, enabling the VLA model to continuously improve its performance without human intervention.

[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0005] An end-to-end autonomous driving training method based on a multimodal large model includes:

[0006] Acquire driving data of the current target multimodal large model in real vehicle or shadow mode;

[0007] The current target multimodal large model is deployed to a closed-loop simulation environment for evaluation. Failure scenarios are detected based on the evaluation results, and root cause classification is performed on the failure scenarios. An improvement plan is generated based on the root cause classification results. The improvement plan includes the training phase identifiers to be strengthened.

[0008] Based on the failure scenarios and their root cause classification, a dual-track data generation process is performed: the world model is invoked to imagine and expand the failure scenarios to generate imagined data, and the root causes of the failure scenarios are used to perform targeted scenario mutations and verify them with the world model to generate synthetic data.

[0009] Using at least the imagined data and the synthetic data, and based on the training stage identifier to be strengthened in the improvement plan, at least one corresponding training stage is selected from the preset multi-stage progressive training strategy to retrain the current target multimodal large model, thereby obtaining an updated multimodal large model.

[0010] Repeat the above process using the updated multimodal large model as the new current target multimodal large model until the iteration termination condition is met.

[0011] According to one aspect of the present invention, the multimodal large model adopts a dual-mode thinking mechanism, which adaptively routes to a fast thinking path or a slow thinking path according to the complexity of the scene during reasoning; the fast thinking path directly outputs the trajectory, while the slow thinking path first generates a structured reasoning chain and then outputs the trajectory, wherein the structured reasoning chain includes five reasoning domains: environmental perception, key object recognition, risk assessment, behavioral decision-making, and trajectory association.

[0012] According to one aspect of the invention, the output trajectory maps the trajectory to control commands that satisfy physical executability through an adapter based on a cyclic gating unit and a bicycle kinematic model, and performs dynamic constraint checks, collision checks, and drivable area verification through a safety guardrail. If any check fails, a preset rule planner is triggered to backtrack.

[0013] According to one aspect of the present invention, the multimodal large model performs hierarchical cross-attention alignment of 2D visual features, 3D visual features and language embeddings through a hierarchical visual-language alignment layer, and fuses 2D and 3D visual features through an instance-aware token fusion module with multi-head attention and gating mechanism.

[0014] According to one aspect of the present invention, the multi-stage progressive training strategy includes a reinforcement learning training stage, which employs group relative advantage optimization, samples multiple output trajectories for the same inference cue, calculates a multi-dimensional verifiable reward including trajectory quality, meta-action consistency, collision penalty and inference quality, calculates the relative advantage of each output relative to the group baseline within the group, and updates the model parameters in combination with KL divergence constraints.

[0015] According to one aspect of the present invention, the imagination unfolding adopts a quick forced multi-step unfolding strategy: starting from the current state of the failed scenario, an action sequence is generated by Gaussian vocabulary sampling, and the dense reward of the subsequent state is predicted by autoregression, and the unfolding trajectory of the reward sequence that satisfies the preset growth trend is used as the imagination data.

[0016] According to one aspect of the present invention, the targeted scene mutation determines the scene parameters to be mutated based on the failure root cause, perturbs at least one of dynamic obstacle trajectory, light intensity, weather type and road surface physical properties, and inputs the mutated scene into the world model for verification, and selects the mutated scene whose predicted reward meets a preset threshold as the synthetic data.

[0017] According to one aspect of the present invention, when acquiring the driving data, the model prediction uncertainty, sample distribution sparseness and scene complexity of each sample are calculated by an active learning strategy, and priority weights are generated in combination. High-value samples are selected for subsequent retraining based on the priority weights.

[0018] According to one aspect of the present invention, the method further includes online mapping and pose optimization steps: extracting semantic landmarks from the 3D point cloud, fitting cylindrical landmarks and planar landmarks using the RANSAC algorithm with normal vector constraints; constructing a factor map including lane line factors, road boundary factors, pole-shaped landmark factors, traffic sign surface factors, road surface plane factors, dynamic obstacle boundary factors, and intersection topology factors for global pose optimization; during incremental updates, determining candidate nodes based on spatial Euclidean distance and topological association depth, and determining the final optimization region based on the reduction in information entropy of each candidate node before and after the update, and performing local marginalization optimization.

[0019] An end-to-end autonomous driving training system based on a multimodal large model includes:

[0020] The data acquisition module acquires driving data collected by the current target multimodal large model in real vehicle or shadow mode;

[0021] The evaluation and analysis module deploys the current target multimodal large model to a closed-loop simulation environment for evaluation, detects failure scenarios based on the evaluation results, performs root cause classification on the failure scenarios, and generates an improvement plan based on the root cause classification results. The improvement plan includes the training phase identifiers to be strengthened.

[0022] The dual-track data generation module performs dual-track data generation based on the failure scenarios and their root cause classifications: it calls the world model to imagine and expand the failure scenarios to generate imagined data, and performs targeted scenario mutations based on the root causes of the failure scenarios and verifies them through the world model to generate synthetic data.

[0023] The training module, using at least the imagined data and the synthetic data, selects at least one corresponding training stage from a preset multi-stage progressive training strategy according to the training stage identifier to be strengthened in the improvement plan, and retrains the current target multimodal large model to obtain an updated multimodal large model.

[0024] The iteration module repeats the above process using the updated multimodal large model as the new current target multimodal large model until the iteration termination condition is met.

[0025] The advantages of this invention are as follows: It can automatically detect failure scenarios and complete root cause classification based on closed-loop simulation evaluation; it can generate diverse training data covering the failure boundary region at low cost through world model imagination expansion and targeted scenario mutation in parallel; and it can dynamically select the corresponding training stage from the multi-stage progressive training strategy for targeted retraining according to the improvement plan, forming a complete adaptive evolutionary closed loop of "evaluation → generation → retraining", which significantly reduces the dependence of model iteration on manual analysis and data labeling; at the same time, the progressive introduction of structured inference chain supervised fine-tuning and reinforcement learning alignment based on group relative advantage optimization and multi-dimensional rewards enables deep synergistic optimization of model inference quality and trajectory safety, effectively improving the system's generalization ability and planning safety in long-tail scenarios, and realizing the continuous self-evolution of the autonomous driving model under limited supervision signals. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating an end-to-end autonomous driving training method based on a multimodal large model as described in this invention.

[0028] Figure 2 This is a schematic diagram of the process of an end-to-end autonomous driving training system based on a multimodal large model according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, an end-to-end autonomous driving training method based on a multimodal large model includes the following steps:

[0031] Step S1: Obtain driving data collected from the current target multimodal large model in real vehicle or shadow mode;

[0032] In this embodiment, the current target VLA model is first deployed on the vehicle platform in shadow mode. The shadow mode means that the VLA model runs in parallel with the vehicle's original rule planner, and the trajectory planning results output by the model are only used for recording and offline evaluation, and do not actually participate in vehicle control.

[0033] During vehicle operation, the onboard sensor suite (including surround-view cameras, lidar, millimeter-wave radar, and a GNSS / IMU integrated navigation system) acquires multi-view images, 3D point clouds, target echo data, and vehicle pose information in real time. The VLA model running in shadow mode performs forward inference on each frame of multimodal perception input, generating a planned trajectory and the corresponding inference chain output.

[0034] The system's built-in VehicleDataCollector module calculates the bias score and difficulty score for each collected sample in real time. The bias score measures the difference between the VLA model's output trajectory and the vehicle's actual driving trajectory; a larger difference indicates a more challenging scenario for the current model. The difficulty score considers factors such as traffic participant density, road topology complexity, lighting, and weather conditions. When either the bias score or the difficulty score exceeds a preset threshold, the sample is marked as a high-value sample and stored. Simultaneously, the system uses the ActiveLearningManager module to calculate priority weights based on uncertainty, rarity, bias, and complexity to further assist in selecting high-value samples.

[0035] Step S2: Deploy the current target multimodal large model to a closed-loop simulation environment for evaluation, detect failure scenarios based on the evaluation results, perform root cause classification on the failure scenarios, and generate an improvement plan based on the root cause classification results. The improvement plan includes the training phase identifiers to be strengthened.

[0036] The target VLA model is deployed in a closed-loop simulation environment (such as CARLA or NavSim) and parallel simulation evaluation is performed on a pre-defined large-scale scenario set. The scenario set includes at least regular driving scenarios, standard regulatory test scenarios, known long-tail scenarios, and failed scenarios accumulated in historical iterations.

[0037] During the simulation, the system automatically records multi-dimensional performance indicators for each test scenario. These indicators include collision rate, average displacement error (ADE), final displacement error (FDE), route completion rate, and comfort (measured by the rate of change of acceleration).

[0038] After the simulation, the FailureScenarioMiner module analyzes the simulation results for each scenario, identifying and marking any scenario where any core indicator fails to reach a preset threshold as a failure scenario. Subsequently, the module performs root cause classification for each failure scenario, with the specific classification system including perception errors, prediction errors, planning errors, reasoning errors, and long-tail scenarios. Specifically: perception errors include target detection omissions, false positives, semantic segmentation errors, and target tracking loss; prediction errors include deviations from the safety margin in predicting the future trajectories of surrounding agents and errors in intention prediction; planning errors include the vehicle's planned trajectory deviating from the drivable area, violating traffic rules, and causing collisions; reasoning errors include logical breaks in the CoT inference chain and inappropriate risk assessment; long-tail scenarios refer to composite failures that are difficult to categorize into a single category, including severe weather, sudden obstacles, and rare road topologies.

[0039] Based on the root cause category of the failure scenario, the system automatically generates a corresponding improvement plan, which includes at least the training phase type identifier to be strengthened and the data focus strategy.

[0040] Step S3: Based on the failure scenarios and their root cause classification, perform dual-track data generation: call the world model to imagine and expand the failure scenarios to generate imagined data, and perform targeted scenario mutation based on the root causes of the failure scenarios and verify it through the world model to generate synthetic data;

[0041] After obtaining the failure scenarios and their root cause classification results, the system executes a dual-track data generation strategy in parallel, which specifically includes the following steps:

[0042] Step S31: World model imagination generation;

[0043] The WorldModelImaginationTrainer module starts with the current state of a failed scenario and invokes a pre-defined world model within the system to perform imaginative expansion. Specifically, this module employs a fast, forced single-step rollout strategy, combined with Gaussian vocabulary sampling and autoregressive dense reward prediction, to generate imaginative exploration data covering the failure boundary region. This approach eliminates the need to rerun the complete scenario in a real or simulation environment, resulting in extremely low computational costs.

[0044] Step S32: Intelligent mutation synthesis generation;

[0045] The IntelligentDataSynthesizer module determines key mutation parameters of a scene based on the root causes of failure. If the root cause is a deviation in the prediction of dynamic obstacle trajectories, the module perturbs the speed, acceleration, and lane-changing timing of surrounding vehicles in the failed scene. If the root cause is perception degradation due to lighting or weather conditions, the module adjusts the scene's lighting intensity or adds weather effects such as rain, snow, or fog. If the root cause is a change in the road surface adhesion coefficient, the module modifies the road surface friction coefficient. After mutation, the generated mutated scene needs to be input into the world model again for validity verification. Only mutated scenes that pass the verification are selected as valid augmented synthetic data.

[0046] Step S4: Using at least the imagined data and the synthetic data, based on the training stage identifier to be strengthened in the improvement plan, select at least one corresponding training stage from the preset multi-stage progressive training strategy, and retrain the current target multimodal large model to obtain the updated multimodal large model.

[0047] The candidate training data, imagination exploration data, and enhanced synthetic data obtained in the preceding steps are merged to construct an incremental training set.

[0048] Based on the training stage identifiers to be reinforced in the improvement plan generated in the second step, the system dynamically selects at least one corresponding training stage from the preset multi-stage progressive training strategy to perform targeted retraining. The multi-stage progressive training strategy includes the following stages:

[0049] Step S41: Base model pre-training;

[0050] InternVL2 or Qwen2.5-VL was selected as the base VLM, and pre-trained with SigLIP 2D visual encoder and BEVFormer 3D visual encoder;

[0051] Step S42: Alignment training;

[0052] Hierarchical vision-language alignment training freezes the backbone network and trains only the HierarchicalVLAlignment module. This module achieves the alignment of 2D / 3D visual tokens and language embeddings step by step through a multi-layer cross-attention mechanism and FFN. At the same time, the InstanceAwareTokenFusion module performs instance-aware adaptive fusion of 2D and 3D visual features with multi-head attention and gating mechanism.

[0053] Step S43: Perform supervised fine-tuning;

[0054] SFT is adapted for the driving domain, and supervised fine-tuning is performed using approximately 3 million multi-task QA data points in the driving domain.

[0055] Step S44: Construct a structured reasoning chain;

[0056] RP-CoT Reasoning-Planning Chain Thinking (SFT) utilizes the RPCotDataConstructor module to construct a structured reasoning chain dataset, which includes five reasoning domains: environmental perception, key objects, risk assessment, behavioral decision-making, and trajectory association.

[0057] Step S45: Perform secondary fine-tuning;

[0058] End-to-end trajectory planning fine-tuning is performed using a comprehensive loss function, which includes at least L2 loss, collision loss, smoothing loss, lane departure loss, and comfort loss.

[0059] Step S46: Perform three fine-tuning adjustments;

[0060] GRPO / RLVR reinforcement learning fine-tuning employs group relative advantage optimization and multi-dimensional verifiable rewards to deeply align reasoning and planning.

[0061] Step S4 involves retraining only the stages identified in the improvement plan, while the remaining stages remain unchanged. After retraining is complete, the system outputs the updated VLA model.

[0062] Step S5: Repeat the above process using the updated multimodal large model as the new current target multimodal large model until the iteration termination condition is met.

[0063] The updated VLA model obtained in S4 is used as the new current target model, and the entire process from step one to step four above is repeated. In each iteration, as the model performance continues to improve, the distribution of failure scenarios continuously migrates to longer-tailed and more complex regions, thereby driving the system to continuously generate more difficult and diverse training data, forming an adaptive evolutionary closed loop of "real vehicle / shadow mode acquisition → closed-loop simulation evaluation → failure scenario mining → world model imagination training → intelligent data synthesis → targeted retraining".

[0064] When preset iteration termination conditions are met, the system stops iterating and outputs the final VLA model. These iteration termination conditions include, but are not limited to: simulation evaluation metrics no longer showing significant improvement after multiple consecutive iterations, the number of failure scenarios falling below a preset threshold, or reaching a preset maximum number of iterations. This final model can then be deployed in a real vehicle for online inference and control.

[0065] Example 2

[0066] In a preferred embodiment of the present invention, the VLA model self-iterative training system further includes online mapping and pose optimization functions to provide high-precision environmental priors for the training process.

[0067] The system extracts semantic landmarks from LiDAR point clouds. The semantic segmentation network first outputs discrete semantic point clouds with category labels, and then uses the RANSAC (Random Sample Consensus) algorithm with fused normal vector constraints to fit cylindrical landmarks and planar landmarks respectively.

[0068] For cylindrical landmarks (such as utility poles, tree trunks, and traffic signposts), let the extracted set of columnar point clouds for specific semantic categories be . The corresponding set of surface normal vectors is The parametric model of the cylinder is as follows: ,in A point on the axis of the cylinder; The unit direction vector of the cylinder axis satisfies ; The radius of the cylindrical section.

[0069] any point Geometric distance residual to the fitted cylinder surface for:

[0070]

[0071] The residual of the included normal angle is:

[0072]

[0073] For all A point is considered an interior point if it simultaneously meets the following dual threshold restrictions. :

[0074]

[0075] After N iterations, the model with the most interior points is selected. The Levenberg-Marquardt algorithm is then used to perform nonlinear least-squares refitting on the set of interior points to optimize the objective function.

[0076]

[0077] Let the set of areal landmarks be The planar parametric model is defined as follows: ,in The unit normal vector of the plane ( ), The intercept from the origin to the plane.

[0078] point algebraic distance residual to the plane for: ;

[0079] Normal deviation residual for: ;

[0080] The system constructs a factor graph containing seven types of factors for global pose optimization: lane line vector factor, road boundary factor, pole landmark factor, traffic sign surface factor, road surface plane factor, dynamic obstacle avoidance boundary factor, and intersection topology connection factor.

[0081] For the Class factor, its calibrated information matrix The unified representation is:

[0082]

[0083] in, This is the nominal basic information matrix of the sensor (determined by physical hardware parameters such as Gaussian white noise and ranging accuracy). The corresponding semantic category static weight factor (reflects the spatial physical stability of the landmark as a global anchor object).

[0084] The confidence factor for dynamic observation is calculated using the following formula:

[0085]

[0086] in, This represents the Softmax probability value of the VLA model network for this type of semantic segmentation / detection. The average reprojection / distance residual of the point cloud fitting; This represents the actual distance from the sensor to the object.

[0087] During incremental updates, the system determines candidate affected nodes based on a joint approach of spatial Euclidean distance and topological association depth. Spatially, this involves the physical radius of the current location. Map nodes within the graph are activated; topologically, a breadth-first search (BFS) is performed through the edges of the factor graph, with a maximum topological traversal depth of [value missing]. =3. For candidate nodes, calculate the change in the logarithm of their marginal covariance matrix before and after incorporating new observations, i.e., the reduction in Shannon entropy:

[0088]

[0089] The final optimization region was determined as follows ,in Only nodes whose information entropy reduction reaches a threshold participate in local optimization, which ensures both local map consistency and controls computational overhead.

[0090] The aforementioned online HD Map construction and factor graph optimization provide high-precision environmental priors for VLA model training, and exist as a preferred auxiliary mechanism in the closed-loop training framework of this invention.

[0091] Example 3

[0092] This embodiment details the specific process by which the VLA model of the present invention performs complete forward inference and outputs a safe trajectory after receiving multimodal input.

[0093] First, the onboard sensor array acquires multi-view surround view images (usually 6 to 12 views) and vehicle status information (including current vehicle position, speed, acceleration, heading angle, etc.).

[0094] The multi-view images are first input into the visual encoding layer. This layer executes two encoding paths in parallel: the first path extracts two-dimensional visual features from each frame using a SigLIP 2D encoder; the second path projects the multi-view image features onto a bird's-eye view space using a BEVFormer 3D encoder to extract three-dimensional visual features. The features output from both paths are then input into the InstanceAwareTokenFusion module. This module uses a multi-head attention mechanism and a gating fusion strategy to adaptively fuse the 2D and 3D visual features with instance awareness, outputting a fused visual token sequence.

[0095] Subsequently, the fused visual token sequence and the pre-set language embeddings (including scene description templates, driving instructions, etc.) are input into the HierarchicalVLAlignment module. This module achieves deep alignment between 2D / 3D visual tokens and language embeddings step by step through a multi-layer cross-attention mechanism and a feedforward network (FFN), and outputs the aligned multimodal feature representation.

[0096] The aligned multimodal features are then input into the SceneComplexityRouter module. This module adaptively selects the inference path based on the scene complexity assessment results: for regular driving scenarios, it routes to the fast thinking path, where the trajectory decoder directly generates a trajectory token and outputs the planned trajectory; for long-tail scenarios or complex interaction scenarios, it routes to the slow thinking path. In the slow thinking path, the model first generates a structured RP-CoT inference chain, which includes five inference domains: environmental perception, key objects, risk assessment, behavioral decision-making, and trajectory association. Each inference domain is deduced step by step in causal order to form a complete inference chain, which is then used by the trajectory decoder to generate a trajectory token.

[0097] The generated trajectory token is input into the decoder corresponding to the TrajectoryTokenizer, and restored to a sequence of trajectory points in continuous space. This sequence of trajectory points is then processed by the GRUbicycleAdapter, which, based on a gated loop unit (GRU) and a bicycle kinematics model, maps the original sequence of trajectory points into an executable trajectory that satisfies the vehicle's physical kinematic constraints. As a supplementary safeguard, the system also performs dynamic constraint checks, collision checks, and drivable area verification on the trajectory through the SafetyGuardrail module. If any check fails, a preset rule planner is triggered to backtrack and output the final safe trajectory.

[0098] At this point, the VLA model has completed the entire forward inference process, and the output safety trajectory is used by the downstream vehicle lateral and longitudinal control modules for execution.

[0099] Example 4

[0100] This embodiment details the specific method for constructing a structured inference chain dataset using the RPCotDataConstructor module. This dataset is used in Phase 3 (RP-CoT Inference-Planning Chain Thinking SFT) of the multi-stage progressive training strategy described in this invention.

[0101] First, complete information for each scenario is extracted from the labeled driving scenario dataset. This information includes: multi-view image sequences, LiDAR point clouds, high-precision maps, trajectories of surrounding intelligent agents (including historical and future data), traffic light status, vehicle status, and driving decision labels.

[0102] The RPCoTDataConstructor module constructs a structured inference chain step by step according to five inference domains:

[0103] The first domain—Environmental Perception—extracts static environmental information from the scene, including lane line type and location, road boundaries, intersection topology, and drivable areas, as well as dynamic environmental information, including the current location, speed, orientation, and category of surrounding traffic participants such as vehicles, pedestrians, and cyclists. The output of this domain is a structured environmental perception description text.

[0104] Second Domain – Key Objects: Building upon the first domain, traffic participants and static obstacles that have a significant impact on the vehicle's current driving decisions are selected through attention mechanisms or importance scoring, marked as key objects, and their possible future movement trends are briefly described.

[0105] The third domain—risk assessment—combines key object information from the second domain to quantify or classify the collision risk, driving safety margin, and risk level in each direction of travel in the current scenario, and outputs structured risk assessment results.

[0106] Fourth Domain – Behavioral Decision: Based on the reasoning results of the first three domains, determine the high-level driving behaviors that the vehicle should take. The high-level driving behaviors include at least one or more combinations of following, lane keeping, lane changing, overtaking, yielding, passing, stopping and waiting, and emergency braking.

[0107] The fifth domain—trajectory association—maps the high-level driving behavior determined in the fourth domain into specific spatiotemporal trajectory planning, that is, generating a sequence of expected trajectory points consistent with the semantics of the behavior decision, along with information on the curvature, speed, and acceleration constraints of the trajectory.

[0108] The outputs of the five domains are concatenated in sequence to form a complete structured RP-CoT sample. In a preferred embodiment of the present invention, two types of samples are constructed, namely short CoT samples (containing only key reasoning steps) and long CoT samples (containing complete five-domain detailed reasoning), according to different scene complexity, to adapt to training requirements of different difficulty levels.

[0109] The completed RP-CoT dataset was used for supervised fine-tuning of the VLA model in Phase 3, enabling the model to learn end-to-end causal mapping capabilities from perceptual input to structured inference chain and then to trajectory output.

[0110] Example 5

[0111] This embodiment details the GRPO (Group Relative Policy Optimization) reinforcement learning fine-tuning method used in Phase 5 of the multi-stage progressive training strategy described in this invention.

[0112] This embodiment aims to fine-tune the VLA model after training in the preceding phases (Phase 0 to Phase 4) using reinforcement learning, in order to achieve depth alignment between inference quality and trajectory planning.

[0113] First, a batch of inference prompts are sampled from the training dataset. Each prompt contains multimodal perception input of the current scene, vehicle status, and driving task description.

[0114] For each Prompt, the current policy model generates GG complete outputs through multiple samplings, each output containing a structured inference chain and a corresponding planning trajectory. In a preferred embodiment of the invention, the value of GG ranges from 4 to 16.

[0115] For each generated output, the system calculates a multi-dimensional verifiable reward, which includes at least a combination of the following components:

[0116] (1) Trajectory quality reward: Evaluate the similarity between the planned trajectory and the expert trajectory, including endpoint displacement error, average displacement error and trajectory shape similarity;

[0117] (2) Meta-action consistency reward: assess the consistency between high-level behavioral decisions (i.e., the behavioral decision domain in the CoT inference chain) and the underlying planning trajectory, i.e. whether the movement intention implied by the behavioral decision is correctly implemented in the trajectory;

[0118] (3) Collision penalty: If the planned trajectory collides with an obstacle or intrudes into the safe distance in the simulation environment, a negative reward is applied;

[0119] (4) Reasoning quality reward: Evaluate the logical coherence within each reasoning domain in the CoT reasoning chain, the causal consistency between each domain, and the reasonable correlation between the reasoning conclusion and the final trajectory output;

[0120] (5) Repetition penalty: Penalize the degradation of semantic repetition or trajectory pattern repetition in the generated output to encourage output diversity.

[0121] For GG outputs under the same prompt, the system calculates the relative advantage of each output relative to the average baseline within the group. That is, the superiority or inferiority is determined by the relative comparison between outputs within the group, without the need to train an additional value network.

[0122] Subsequently, the system updates the policy model parameters by incorporating KL divergence constraints. The KL divergence constraints are used to prevent the model from deviating too far from the reference policy (i.e., the model saved at the end of Phase 4) during the reinforcement learning fine-tuning process, thereby maintaining the stability and security of training.

[0123] In a preferred embodiment of the present invention, the group relative advantage optimization is supplemented by the RLVR (Reinforcement Learning from Verifiable Rewards) strategy, which verifies that the reward design is a deterministic calculation based on rules or physical simulators, without relying on manual preference labeling, thereby achieving low-cost reinforcement learning alignment.

[0124] The aforementioned GRPO / RLVR reinforcement learning fine-tuning stage enables the model to autonomously improve inference quality and trajectory safety through continuous trial and error and reward feedback.

[0125] Example 6

[0126] This embodiment details the specific implementation of the world model imagination training in the self-iterative training closed loop described in this invention.

[0127] The WorldModelImaginationTrainer module performs the training, which aims to generate imagination exploration data covering the failure boundary region at low cost, using failure scenarios as seeds, thereby expanding the distribution range of the training set.

[0128] First, a complete state sequence of a failed scenario is extracted from the failed scenario library. Starting with the current state stst of this failed scenario, the WorldModelImaginationTrainer module initiates the imagination unfolding process.

[0129] During deployment, this module does not rely on real-world interaction or complete simulation replay, but directly calls the world model for forward prediction. In this invention, the world model is an environment transition prior learned synchronously during the VLA model training process, possessing the ability to predict future states and rewards based on current observations.

[0130] In each expansion step, the module generates multiple candidate actions using a Gaussian vocabulary sampling strategy, rather than employing deterministic action selection. The covariance parameter of the Gaussian vocabulary sampling controls the breadth of exploration, enabling the system to generate diverse behavioral trajectories, rather than being limited to the original behavioral patterns of failure scenarios.

[0131] The world model predicts the next state and its corresponding dense reward signal based on the current state and candidate actions. This dense reward signal is output at every time step, rather than just a single reward at the end of the trajectory, thus providing fine-grained feedback information for intermediate states.

[0132] The unfolding process continues for a preset number of steps (preferably 3 to 5 steps in this invention) to form multiple complete unfolding trajectories. The module calculates the cumulative reward or average reward for each unfolding trajectory and selects trajectories whose rewards meet preset criteria as valid imaginative exploration data output.

[0133] The core advantage of the aforementioned quick forced single-step rollout strategy lies in its ability to avoid error accumulation during long-term rollout by forcing each prediction to be based on the actual state (i.e., each step uses the actual state of the previous step rather than the model's own predicted state as input), thus effectively ensuring the quality and reliability of the generated data. Simultaneously, the Gaussian vocabulary sampling mechanism ensures data diversity, preventing the generated data from falling into patterns that are too similar to the original failure scenarios.

[0134] Example 7

[0135] An end-to-end autonomous driving training system based on a multimodal large model includes:

[0136] The data acquisition module M1 acquires driving data collected by the current target multimodal large model in real vehicle or shadow mode;

[0137] The evaluation and analysis module M2 deploys the current target multimodal large model to a closed-loop simulation environment for evaluation, detects failure scenarios based on the evaluation results, performs root cause classification on the failure scenarios, and generates an improvement plan based on the root cause classification results. The improvement plan includes the training phase identifiers to be strengthened.

[0138] The dual-track data generation module M3 performs dual-track data generation based on the failure scenarios and their root cause classifications: it calls the world model to imagine and expand the failure scenarios to generate imagined data, and performs targeted scenario mutations based on the root causes of the failure scenarios and verifies them through the world model to generate synthetic data.

[0139] Training module M4, using at least the imagined data and the synthetic data, selects at least one corresponding training stage from a preset multi-stage progressive training strategy according to the training stage identifier to be strengthened in the improvement plan, and retrains the current target multimodal large model to obtain an updated multimodal large model.

[0140] The iteration module M5 repeats the above process using the updated multimodal large model as the new current target multimodal large model until the iteration termination condition is met.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An end-to-end autonomous driving training method based on a multimodal large model, characterized in that, include: Acquire driving data of the current target multimodal large model in real vehicle or shadow mode; The current target multimodal large model is deployed to a closed-loop simulation environment for evaluation. Failure scenarios are detected based on the evaluation results, and root cause classification is performed on the failure scenarios. An improvement plan is generated based on the root cause classification results. The improvement plan includes the training phase identifiers to be strengthened. Based on the failure scenarios and their root cause classification, a dual-track data generation process is performed: the world model is invoked to imagine and expand the failure scenarios to generate imagined data, and the root causes of the failure scenarios are used to perform targeted scenario mutations and verify them with the world model to generate synthetic data. Using at least the imagined data and the synthetic data, and based on the training stage identifier to be strengthened in the improvement plan, at least one corresponding training stage is selected from the preset multi-stage progressive training strategy to retrain the current target multimodal large model, thereby obtaining an updated multimodal large model. Repeat the above process using the updated multimodal large model as the new current target multimodal large model until the iteration termination condition is met.

2. The end-to-end autonomous driving training method based on a multimodal large model according to claim 1, characterized in that, The multimodal large model adopts a dual-mode thinking mechanism, which adaptively routes to a fast thinking path or a slow thinking path according to the complexity of the scene during reasoning. The fast thinking path directly outputs the trajectory, while the slow thinking path first generates a structured reasoning chain and then outputs the trajectory. The structured reasoning chain includes five reasoning domains: environmental perception, key object recognition, risk assessment, behavioral decision-making, and trajectory association.

3. The end-to-end autonomous driving training method based on a multimodal large model according to claim 2, characterized in that, The output trajectory is mapped to control commands that meet physical executability through an adapter based on a cyclic gating unit and a bicycle kinematic model. Dynamic constraint checks, collision checks, and drivable area verification are performed through safety guardrails. If any check fails, a preset rule planner is triggered to backtrack.

4. The end-to-end autonomous driving training method based on a multimodal large model as described in claim 1, characterized in that, The multimodal large model performs hierarchical cross-attention alignment of 2D visual features, 3D visual features and language embeddings through a hierarchical visual-language alignment layer, and fuses 2D and 3D visual features through an instance-aware token fusion module with multi-head attention and gating mechanism.

5. The end-to-end autonomous driving training method based on a multimodal large model according to claim 1, characterized in that, The multi-stage progressive training strategy includes a reinforcement learning training stage, which employs group relative advantage optimization. Multiple output trajectories are sampled for the same inference cue, and a multi-dimensional verifiable reward including trajectory quality, meta-action consistency, collision penalty, and inference quality is calculated. Within the group, the relative advantage of each output relative to the group baseline is calculated, and the model parameters are updated in combination with KL divergence constraints.

6. The end-to-end autonomous driving training method based on a multimodal large model according to claim 1, characterized in that, The imagination unfolding adopts a quick forced multi-step unfolding strategy: starting from the current state of the failed scenario, an action sequence is generated through Gaussian vocabulary sampling, and the dense reward of the subsequent state is predicted through autoregression. The unfolding trajectory of the reward sequence that satisfies the preset growth trend is used as the imagination data.

7. The end-to-end autonomous driving training method based on a multimodal large model according to claim 1, characterized in that, The targeted scene mutation determines the scene parameters to be mutated based on the failure root causes, perturbs at least one of dynamic obstacle trajectories, light intensity, weather type and road surface physical properties, and inputs the mutated scene into the world model for verification, and selects the mutated scenes whose predicted rewards meet the preset threshold as the synthetic data.

8. The end-to-end autonomous driving training method based on a multimodal large model according to claim 1, characterized in that, When acquiring the driving data, the model prediction uncertainty, sample distribution sparseness, and scene complexity of each sample are calculated through an active learning strategy, and priority weights are generated in combination. High-value samples are selected for subsequent retraining based on the priority weights.

9. The end-to-end autonomous driving training method based on a multimodal large model according to claim 1, characterized in that, It also includes online mapping and pose optimization steps: extracting semantic landmarks from 3D point clouds, fitting cylindrical landmarks and planar landmarks using the RANSAC algorithm with normal vector constraints; constructing a factor map containing lane line factors, road boundary factors, pole landmark factors, traffic sign surface factors, road surface factors, dynamic obstacle boundary factors, and intersection topology factors for global pose optimization; during incremental updates, candidate nodes are determined based on spatial Euclidean distance and topological association depth, and the final optimization area is determined based on the reduction of information entropy of each candidate node before and after the update, and local marginalization optimization is performed.

10. An end-to-end autonomous driving training system based on a multimodal large model, characterized in that, The end-to-end autonomous driving training method based on the multimodal large model according to any one of claims 1 to 9 includes: The data acquisition module acquires driving data collected by the current target multimodal large model in real vehicle or shadow mode; The evaluation and analysis module deploys the current target multimodal large model to a closed-loop simulation environment for evaluation, detects failure scenarios based on the evaluation results, performs root cause classification on the failure scenarios, and generates an improvement plan based on the root cause classification results. The improvement plan includes the training phase identifiers to be strengthened. The dual-track data generation module performs dual-track data generation based on the failure scenarios and their root cause classifications: it calls the world model to imagine and expand the failure scenarios to generate imagined data, and performs targeted scenario mutations based on the root causes of the failure scenarios and verifies them through the world model to generate synthetic data. The training module, using at least the imagined data and the synthetic data, selects at least one corresponding training stage from a preset multi-stage progressive training strategy according to the training stage identifier to be strengthened in the improvement plan, and retrains the current target multimodal large model to obtain an updated multimodal large model. The iteration module repeats the above process using the updated multimodal large model as the new current target multimodal large model until the iteration termination condition is met.

Images