Automatic driving large model training optimization method based on multi-scene data balancing

Abstract

The application is an automatic driving large model training optimization method based on multi-scene data balance. It includes the following steps: (1) constructing a real vehicle high-speed driving scene library; (2) constructing a visual language automatic driving large model, based on the real vehicle high-speed driving scene library, using an iterative training framework, designing a multi-task joint loss function and a weight self-adaptive adjustment strategy, achieving multi-task target balance, and obtaining the trained visual language automatic driving large model; (3) performing dynamics simulation for the automatic driving working condition, and based on the simulation data, constructing a high-fidelity simulation data test set; (4) according to the high-fidelity simulation data test set, performing hyperparameter optimization and lightweight processing on the trained visual language automatic driving large model, and obtaining a scene data balanced automatic driving large model. The application realizes the acceleration of the large model inference speed and the reduction of resource occupation, thereby significantly improving the discrimination ability of the large model for dynamic working conditions and high-risk scenes.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and specifically relates to an autonomous driving large model training optimization method based on multi-scene data balancing. Background Art

[0002] In recent years, the deep integration of artificial intelligence, big data, and data-driven technologies has ushered in an unprecedented intelligent transformation in the transportation industry. The performance of large-scale autonomous driving models directly impacts vehicle safety and operational efficiency. In practical applications, autonomous vehicles must navigate changing road conditions, complex traffic environments, and dynamic changes in their own state. This requires the autonomous driving system to quickly understand the driving scenario and make appropriate driving decisions, placing high demands on large-scale autonomous driving models.

[0003] Large visual language models have been introduced into autonomous driving systems, but they face the problem of skewed data distribution when driving a real vehicle at high speeds (greater than 60 km / h), a high proportion of data on straight-line driving conditions, and a relative scarcity of data on dynamic conditions and complex scenarios. This results in the autonomous driving system's insufficient ability to recognize high-risk and long-tail scenarios. At the same time, traditional training and optimization methods are limited by the strong nonlinearity of vehicle dynamics characteristics, resulting in insufficient model generalization capabilities. Moreover, due to the lack of data on dangerous driving scenarios in real vehicles, traditional methods conduct model testing in a simulation environment, which has a large deviation from the actual road conditions and poor optimization effect on the model. Summary of the Invention

[0004] The purpose of the present invention is to provide an optimization method for training a large autonomous driving model based on multi-scene data balance, so as to solve the problems of skewness of existing high-speed driving data and poor generalization performance of large autonomous driving models.

[0005] The technical solutions for achieving the purpose of the present invention are:

[0006] The method for optimizing large-scale autonomous driving model training based on multi-scenario data balance includes the following steps:

[0007] Step (1): Collect driving data, use a deep learning model to extract multi-dimensional features from the driving data, use a hierarchical clustering algorithm to stratify the real vehicle driving data into scenarios, and embed an information entropy weighting mechanism to adjust the weights of high-risk and long-tail scenario data to achieve scenario enhancement and build a data-enhanced real vehicle high-speed driving scenario library;

[0008] Step (2): Build a large model of visual language autonomous driving, based on a real car high-speed driving scene library, adopt an iterative training framework for training, design a multi-task joint loss function and weight adaptive adjustment strategy to achieve multi-task goal balance, and obtain a trained large model of visual language autonomous driving;

[0009] Step (3): Perform dynamic simulation for the straight line, turning, emergency braking, lane change, overtaking, and ramp merging conditions of the autonomous vehicle during high-speed driving; based on the simulation data, combine the physics engine and digital twin to build a high-fidelity simulation data test set;

[0010] Step (4): Based on the high-fidelity simulation data test set of step (3), the trained visual language autonomous driving large model obtained in step (2) is subjected to hyperparameter optimization and lightweight processing to obtain a large autonomous driving large model with balanced scene data.

[0011] Furthermore, step (1) includes the following steps:

[0012] Step (11): Preprocess the real vehicle driving data: obtain the time series data of vehicle speed, steering angle, vehicle body posture and acceleration from the vehicle-mounted sensors, and collect the spatial data of road topology and obstacle distance provided by the vehicle-mounted visual sensor; synchronize the data using timestamps, and perform spatial calibration with the help of inertial measurement unit (IMU) information to form a multimodal data matrix in a unified format;

[0013] Step (12): A hybrid architecture of a deep learning neural network based on a self-attention mechanism and a text encoder is used to input red, green, and blue channel images and a text description of the driving scene. The self-attention mechanism is used to perform joint feature learning on the pre-processed multimodal data. After the embedding layer, position encoding, and multi-head self-attention layer, the temporal dynamic information and spatial relationship information are fused and the fused features are output.

[0014] Step (13): Introduce a hierarchical clustering method to classify the fused features and assign a label to each sample scene to achieve real-time judgment and dynamic labeling of driving status;

[0015] Step (14): For the predicted output at each moment in the model, calculate the category probability distribution and use information entropy to measure model uncertainty; including:

[0016] Information entropy calculation:

[0017]

[0018] in, For the The input data of samples, For samples Belong to category The probability, information entropy It reflects the uncertainty within the scenario;

[0019] Weight definition and embedding: information entropy Incorporate the loss function and define the sample weights:

[0020]

[0021] Where, To adjust hyperparameters;

[0022] The modified scene data clustering hierarchical model loss function is:

[0023]

[0024] in, The total number of tasks to optimize the model for data distribution, Input for the model The predicted output of is the true label, is the traditional cross entropy loss function, is the dynamically calculated weight;

[0025] Step (15): Construct a risk quantification module, dynamically adjust the weights of high-risk and long-tail scenarios in the driving data based on the statistical results of the information entropy of each scenario, and feed it back to the data collection and large-scale model retraining process to achieve continuous optimization of data distribution, generate a multi-level scenario library based on balanced real-car driving data, and use it for large-scale autonomous driving model training.

[0026] Furthermore, step (2) includes the following steps:

[0027] Step (21): Design the traffic scenario and driving decision generation joint loss function, expressed as:

[0028]

[0029] in, is the overall joint loss function used to guide model parameters Updates; is the total number of tasks; For the The weight coefficient of each task is used to balance the importance of each task; For the The loss function of each task depends on the current model parameters By weighting the loss functions of each task and summing them up, a unified loss target is formed, providing overall guidance for large model training;

[0030] Step (22): Design a task weight adaptive adjustment strategy to dynamically balance the contribution of each task in training. Based on the performance of the current iteration, design the following weight update formula:

[0031]

[0032] in, is the total number of tasks, is the overall joint loss function, Represents a hyperparameter that controls the sensitivity of the loss value to the weight. Represents the index of each task;

[0033] Step (23): An iterative training mechanism is introduced, and scene equilibrium data is added as feedback in each iteration cycle; the model parameters are continuously optimized, and the feedback data helps the model to correct the recognition bias of complex scenes in a timely manner; in each round of iteration, the model parameters are updated using the joint loss function and the gradient descent method. The formula is as follows:

[0034]

[0035] in, Indicates the The model parameters at the iteration, Represents the learning rate, which controls the step size of parameter updates. represents the gradient of the joint loss function with respect to the parameters;

[0036] Total loss function It consists of two parts: joint loss function And the feedback loss function based on scene equilibrium data :

[0037]

[0038]

[0039] in, Indicates the size of the scene-balanced dataset after data augmentation; For input data, is the true label corresponding to each scene, Indicates that the model has parameters Lower pair input predictions, represents the loss function used to measure the prediction error, It is a hyperparameter that controls the proportion of feedback loss in the total loss;

[0040] Step (24): In each iteration, according to the current total loss Calculate gradients and update model parameters ; At the same time, the number of samples in the scene balanced dataset is used Calculating feedback loss , helping the large model to focus on the details in complex scenes; the iterative process is repeated continuously, so that the large model can gradually improve its ability to distinguish various complex scenes under the dual mechanisms of adaptive multi-task learning and scene balanced feedback.

[0041] Furthermore, step (3) includes the following steps:

[0042] Generate typical scenarios in the open-source scenario editor, overlay long-tail risk conditions, and cover the diverse needs of real roads;

[0043] Synchronously collect high-frame-rate video images, lidar point clouds, and vehicle control signals, automatically annotate scene text using cross-modal semantic association technology, and improve data quality using adversarial sample generation and multimodal consistency verification.

[0044] Build a digital twin verification system, reproduce real traffic scenarios in the simulation platform, combine hardware-in-the-loop testing with a progressive real-vehicle verification strategy, and optimize the model's scene understanding, trajectory planning, and safety decision-making capabilities in stages, ultimately forming a high-fidelity simulation data test set with strong semantic associations and high safety redundancy.

[0045] Furthermore, step (4) specifically includes the following steps:

[0046] Step (41): Carry out the evaluation of the decision and planning module of the visual language large model, and use the minimum average displacement error ADE and the end point displacement error FDE to evaluate the trajectory prediction accuracy, as follows:

[0047]

[0048]

[0049] in 、 represents the real vehicle position coordinates at time t, 、 Indicates the actual vehicle position coordinates at the final moment; 、 represents the predicted position coordinates at time t, 、 Indicates the predicted position coordinates at the final moment; represents the total time steps of the predicted trajectory;

[0050] Step (42): Use grid search and random search methods to process a small-scale hyperparameter space, build a proxy model based on the Bayesian optimization method, and maximize the expected improvement function:

[0051]

[0052] in represents the mathematical expectation, represents the objective function, represents the currently known optimal objective function value, Represents the hyperparameter combination to be optimized;

[0053] Step (43): Use the knowledge distillation method to transfer the knowledge of the large model to the lightweight model. The loss function of the lightweight model is designed as:

[0054]

[0055] in is the weight coefficient of cross entropy loss and feature alignment loss, is the cross entropy loss, which is used to supervise the classification output of the student model, is the feature alignment loss, used to align the teacher model With student models The feature representation of represents the classification output of the student model, Represents the true label of the sample;

[0056] Step (44): Use cosine annealing strategy to balance convergence speed and stability, dynamic learning rate scheduling:

[0057]

[0058] in 、 are the minimum and maximum values ​​of the learning rate, is the current training step number, is the total number of training steps in one cycle;

[0059] Step (45): Quantize and prune the large model, convert the floating-point weights into 8-bit integers to reduce storage and computational overhead. The formula is as follows:

[0060]

[0061] in is a floating-point weight parameter, are the maximum and minimum values ​​of the weight, is the number of bits after quantization, (*) means converting the scaled floating-point value into an integer to make it discrete;

[0062] Perform structured pruning on the large model and remove redundant channels based on L1 regularization. The objective function of the pruned model is:

[0063]

[0064] in For mission loss, is the coefficient of the L1 regularization term, which controls the pruning intensity. is the weight matrix of the first layer.

[0065] To improve the safety and generalization performance of large-scale autonomous driving models, this application designs a full-chain autonomous driving visual language model optimization system consisting of "data balancing-model training-simulation transfer-model optimization". This system collects driving data, including visual sensor data and CAN bus data, and uses a deep learning model to extract multi-dimensional features from the real-world vehicle data. A hierarchical clustering algorithm is used to stratify the real-world vehicle data into scene layers. An information entropy weighting mechanism is embedded to adjust the weights of high-risk and long-tail scene data, achieving scene enhancement and constructing a multi-level scene library. Within the visual language model architecture, an iterative training mechanism is introduced to continuously optimize model parameters. A multi-task joint training loss function is designed, combined with a task weight adaptive adjustment strategy, to achieve a dynamic balance between task objectives and enhance the model's ability to discriminate and generalize complex scenarios. Based on the mapping relationship between sensor data and real-world vehicle data, digital twin technology is used to construct fully digital driving scenarios, forming a high-fidelity simulation test set to ensure high consistency between the simulated environment and the real-world vehicle data. Knowledge distillation is performed on the autonomous driving visual language model based on the simulation test set to improve inference speed and reduce resource usage. Hyperparameter optimization of key parameters is also performed to enhance model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is the overall flow chart of the autonomous driving large model training optimization method based on multi-scenario data balance in this application;

[0067] Figure 2 This is the large model training flow chart based on data distribution optimization for this application;

[0068] Figure 3 This is the flow chart for generating high-fidelity simulation test sets and optimizing large models for this application. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below with reference to the accompanying drawings.

[0070] like Figure 1-3 As shown in FIG, a large-scale autonomous driving model training optimization method based on multi-scenario data balance includes:

[0071] S1: Uses an end-to-end deep learning model to extract multidimensional features from data, implements scene segmentation based on a hierarchical clustering algorithm, embeds an information entropy weighting mechanism into the scene recognition model, adjusts weights for high-risk and long-tail scene data, increases the proportion of edge scenes, and constructs a data-enhanced real-world high-speed driving scene library for large-scale model training.

[0072] S2: Build a large-scale autonomous driving model using visual language. Based on a real-world vehicle data scenario library, adopt an iterative training framework for training, design a multi-task joint loss function and adaptive weight adjustment strategy to achieve multi-task goal balance, and obtain a trained large-scale autonomous driving model.

[0073] S3: Dynamics analysis of autonomous vehicles in high-speed driving conditions such as straight lines, turns, emergency braking, lane change, overtaking, and ramp merging. A high-fidelity simulation data test set is constructed based on simulation data, combining physics engines and digital twin technology.

[0074] S4: Conduct large-scale autonomous driving visual language model testing, perform hyperparameter optimization and lightweight processing, and obtain a large-scale autonomous driving model with balanced scene data.

[0075] Step S1 includes:

[0076] S11 pre-processes real-vehicle driving data. It first obtains time-series data on vehicle speed, steering angle, body posture, and acceleration from on-board sensors, and collects spatial data on road topology and obstacle distance provided by on-board vision sensors. It synchronizes the data using timestamps and performs spatial calibration with the help of inertial measurement unit (IMU) information to form a multimodal data matrix in a unified format.

[0077] The S12 adopts a hybrid architecture of a deep learning neural network based on the self-attention mechanism and a text encoder. It inputs red, green, and blue channel (RGB) images and text descriptions of driving scenarios. It uses the self-attention mechanism to perform joint feature learning on the pre-processed multimodal data. After the embedding layer, position encoding and multi-head self-attention layer, it fuses temporal dynamic information and spatial relationship information to output fused features.

[0078] S13 introduces a hierarchical clustering method to classify the fused features into scenes, assigning a label to each sample scene, and realizing real-time judgment and dynamic labeling of driving status.

[0079] In the model, S14 calculates the category probability distribution for the predicted output at each moment and uses information entropy to measure model uncertainty. This includes:

[0080] Information entropy calculation:

[0081]

[0082] in, For the The input data of samples, For samples Belong to category The probability of It reflects the uncertainty within the scene. Usually, high-risk and long-tail scenes will show higher uncertainty and larger entropy values.

[0083] Weight definition and embedding: information entropy Incorporate loss function and define sample weights

[0084]

[0085] Where, To adjust the hyperparameters, the modified scene data clustering hierarchical model loss function is:

[0086]

[0087] in, The total number of tasks to optimize the model for data distribution, Input for the model The predicted output of is the true label, is the traditional cross entropy loss function, The weight is calculated dynamically.

[0088] S15 builds a risk quantification module. Based on the statistical results of the information entropy of each scenario, it dynamically adjusts the weights of high-risk and long-tail scenarios in the driving data, and feeds back to the data collection and large-scale model retraining links to achieve continuous optimization of data distribution. It generates a multi-level scenario library based on balanced real-vehicle driving data and uses it for large-scale autonomous driving model training.

[0089] In step S2, based on the multi-level scene library of real vehicle data established in S1, the multi-task joint loss function design and iterative training of the autonomous driving visual language large model are carried out:

[0090] S21 designs a joint loss function for traffic scenarios and driving decisions, expressed as:

[0091]

[0092] in, is the overall joint loss function used to guide model parameters Updates; is the total number of tasks; For the The weight coefficient of each task is used to balance the importance of each task; For the The loss function of each task depends on the current model parameters , the loss function can be mean square error, cross entropy, etc., determined according to the characteristics of the task. By summing the loss functions of each task with weights, a unified loss target is formed , providing overall guidance for large model training.

[0093] S22 designs a task weight adaptive adjustment strategy to dynamically balance the contribution of each task in training. Based on the performance of the current iteration, the following weight update formula is designed:

[0094]

[0095] in, is the total number of tasks, is the overall joint loss function, Represents a hyperparameter that controls the sensitivity of the loss value to the weight. Represents the index of each task. The exponential function ensures that larger losses receive smaller weights, making the tasks more balanced during joint training.

[0096] To address the challenges of complex road conditions and changing environments in high-speed driving scenarios and enhance the large model's ability to discern complex scenarios, the S23 introduces an iterative training mechanism and incorporates scene-balancing data as feedback in each iteration. This continuously optimizes model parameters, using feedback data to help the model promptly correct biases in recognizing complex scenarios. In each iteration, the model parameters are updated using a joint loss function and a gradient descent method, as follows:

[0097]

[0098] in, Indicates the The model parameters at the iteration, Represents the learning rate, which controls the step size of parameter updates. Represents the gradient of the joint loss function with respect to the parameters. Total loss Contains two parts: joint loss and feedback loss based on scene equalization data :

[0099]

[0100]

[0101] in, Represents the size of the scene-balanced dataset after data augmentation. The feedback loss is normalized by dividing it by the number of samples to ensure that each data sample contributes equally to the total feedback loss when calculating the average loss, so that the magnitude of the feedback loss is consistent with other loss terms; For input data, is the corresponding true label, Indicates that the model has parameters Lower pair input predictions, represents the loss function used to measure the prediction error (such as cross entropy, mean square error), is a hyperparameter that controls the proportion of feedback loss in the total loss.

[0102] In each iteration, according to the current total loss Calculate gradients and update model parameters ; At the same time, use the scene to balance the number of data sets Calculating feedback loss , helping the large model to focus on the details in complex scenes; the iterative process is repeated continuously, so that the large model can gradually improve its ability to distinguish various complex scenes under the dual mechanisms of adaptive multi-task learning and scene balanced feedback.

[0103] In step S3, the vehicle dynamics model and complex load distribution are simulated with high fidelity through the physics engine to restore the control response and stability under different loads, speeds and road conditions; typical scenarios such as continuous curves and braking on rainy and snowy roads are generated in the open source scene editor, and long-tail risk conditions such as weather and obstacles are superimposed to cover the diverse needs of real roads; high-frame rate video images, lidar point clouds and vehicle control signals are simultaneously collected, and automatic annotation of scene text is achieved through cross-modal semantic association technology, and adversarial sample generation and multi-modal consistency verification are used to improve data quality; a digital twin verification system is constructed to reproduce real traffic scenarios in the simulation platform, and combined with hardware-in-the-loop testing and progressive real-vehicle verification strategies, the model's scene understanding, trajectory planning and safety decision-making capabilities are optimized in stages, ultimately forming a test set with strong semantic association and high safety redundancy, which significantly reduces dependence on real-vehicle road testing while meeting functional safety standards.

[0104] In step S4, based on the constructed high-fidelity simulation data test set, the performance evaluation indicators of the large model are clarified, and hyperparameter optimization and lightweight processing of the large model are performed.

[0105] S41 conducts evaluation of the decision-making and planning module of the visual language large model, and uses the minimum average displacement error (ADE) and the end point displacement error (FDE) to evaluate the trajectory prediction accuracy. The formula is as follows:

[0106]

[0107]

[0108] in 、 represents the real vehicle position coordinates at time t, 、 Indicates the actual vehicle position coordinates at the final moment; 、 represents the predicted position coordinates at time t, 、 Indicates the predicted position coordinates at the final moment; Represents the total time steps of the predicted trajectory.

[0109] S42 targets the high-dimensional parameter space of large visual language models, combining traditional optimization algorithms with automation technology to improve efficiency. It uses traditional grid search and random search methods to process small-scale hyperparameter spaces such as learning rates, and builds a proxy model based on Bayesian optimization methods to maximize the expected improvement function:

[0110]

[0111] in represents the mathematical expectation, represents the objective function, represents the currently known optimal objective function value, Represents the hyperparameter combination to be optimized.

[0112] The knowledge distillation method in automated optimization technology is used to transfer the knowledge of the large model to the lightweight model. The loss function of the lightweight model is designed as follows:

[0113]

[0114] in is the weight coefficient of cross entropy loss and feature alignment loss, is the cross entropy loss, which is used to supervise the classification output of the student model, is the feature alignment loss, used to align the teacher model With student models The feature representation of represents the classification output of the student model, Represents the true label of the sample;

[0115] The cosine annealing strategy is used to balance the convergence speed and stability, and the dynamic learning rate scheduling is used:

[0116]

[0117] in 、 are the minimum and maximum values ​​of the learning rate, is the current training step number, is the total number of training steps in one cycle;

[0118] Due to the limitations of onboard computing power, S43 needs to use compression technology to reduce the complexity of large models while maintaining performance. First, the large model is quantized and pruned, and floating-point weights are converted to 8-bit integers to reduce storage and computing overhead. The formula is as follows:

[0119]

[0120] in is a floating-point weight parameter, are the maximum and minimum values ​​of the weight, is the number of bits after quantization, (*) means converting the scaled floating-point value into an integer to make it discretizable.

[0121] Then perform structured pruning on the large model and remove redundant channels based on L1 regularization. The objective function of the pruned model is:

[0122]

[0123] in For mission loss, is the coefficient of the L1 regularization term, which controls the pruning intensity. is the weight matrix of the first layer.

[0124] In response to the problems of skewed real-car high-speed driving data, poor generalization performance of large autonomous driving models, and insufficient long-tail scenario data, this application designs a multi-scenario data balancing and large-model training optimization method for autonomous driving models: the information entropy weighting mechanism and hierarchical clustering algorithm are integrated into the deep learning model to achieve long-tail scenario enhancement; a multi-task joint loss function and weight adaptive adjustment strategy are designed to achieve multi-task target balance of the visual language large model; a high-fidelity simulation data test set is constructed by combining the physical engine and digital twin technology, and a model testing and optimization process is designed on this basis; a full-chain autonomous driving visual language large model optimization system of "data balancing-model training-simulation migration-model optimization" is formed, which solves the problem of uneven distribution of real-car high-speed driving data, improves the overall generalization performance of the visual language large model, and plays an important role in enhancing the driving safety, reliability and operational efficiency of autonomous driving vehicles in complex roads and dynamic traffic scenarios.

[0125] The above are exemplary embodiments of the present application, and the scope of protection of the present application is defined by the claims and their equivalents. Those skilled in the art should understand that the present application is not limited by the above examples. The above examples and descriptions are merely illustrative of the principles of the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An optimization method for large-scale autonomous driving model training based on multi-scenario data balance, characterized by: The steps include: Step (1): Collect driving data, use a deep learning model to extract multidimensional features from the driving data, use a hierarchical clustering algorithm to stratify the real vehicle driving data into scenarios, and embed an information entropy weighting mechanism to adjust the weights of high-risk and long-tail scenario data to achieve scenario enhancement and build a data-enhanced real vehicle high-speed driving scenario library; Step (2): Build a large model of visual language autonomous driving, based on a real vehicle high-speed driving scene library, adopt an iterative training framework for training, design a multi-task joint loss function and a weight adaptive adjustment strategy to achieve a multi-task goal balance, and obtain a trained large model of visual language autonomous driving; Step (2) includes the following steps: Step (21): Design the traffic scenario and driving decision generation joint loss function, expressed as: in, is the overall joint loss function, which is used to guide the update of the model parameters θ; n is the total number of tasks; λ i′ is the weight coefficient of the i′th task, which is used to balance the importance of each task; is the loss function of the i′th task, which depends on the current model parameters θ. By summing the loss functions of each task with weights, a unified loss target is formed, providing overall guidance for large model training. Step (22): Design a task weight adaptive adjustment strategy to dynamically balance the contribution of each task in training. Based on the performance of the current iteration, design the following weight update formula: Where n is the total number of tasks, is the overall joint loss function, β represents a hyperparameter used to control the sensitivity of the loss value to the weight, h ′ Represents the index of each task; Step (23): An iterative training mechanism is introduced, and scene equilibrium data is added as feedback in each iteration cycle; the model parameters are continuously optimized, and the feedback data helps the model to correct the recognition bias of complex scenes in a timely manner; in each round of iteration, the model parameters are updated using the joint loss function and the gradient descent method. The formula is as follows: in, represents the model parameters at the tth iteration, η represents the learning rate, and controls the step size of parameter update. represents the gradient of the joint loss function with respect to the parameters; Total loss function It consists of two parts: joint loss function And the feedback loss function based on scene equilibrium data Among them, D balance represents the size of the scene balanced dataset after data enhancement; x′ is the input data, Ψ′ is the true label corresponding to each scene, Indicates that the model has parameters The prediction for the input x′ is: represents the loss function used to measure the prediction error, and γ is a hyperparameter that controls the proportion of feedback loss in the total loss; Step (24): In each iteration, according to the current total loss Calculate the gradient and update the model parameters θ; at the same time, use the scene to balance the number of data set samples D balance Calculate the feedback loss L feedback , helping the large model focus on the details in complex scenes; the iterative process is repeated continuously, so that the large model can gradually improve its ability to distinguish various complex scenes under the dual mechanism of adaptive multi-task learning and scene balanced feedback; Step (3): Perform dynamic simulation for the straight line, turning, emergency braking, lane change, overtaking, and ramp merging conditions of the autonomous vehicle during high-speed driving; based on the simulation data, combine the physics engine and digital twin to build a high-fidelity simulation data test set; Step (4): Based on the high-fidelity simulation data test set of step (3), the trained visual language autonomous driving large model obtained in step (2) is subjected to hyperparameter optimization and lightweight processing to obtain an autonomous driving large model with balanced scene data.

2. The method according to claim 1, characterized in that Step (1) includes the following steps: Step (11): Preprocess the real vehicle driving data: obtain vehicle speed, steering angle, vehicle body posture and acceleration time series data from the vehicle-mounted sensors, and collect road topology and obstacle distance spatial data provided by the vehicle-mounted visual sensor; The data is synchronized using timestamps and spatially calibrated with the help of inertial measurement unit (IMU) information to form a multimodal data matrix in a unified format. Step (12): A hybrid architecture of a deep learning neural network based on a self-attention mechanism and a text encoder is adopted. The red, green, and blue channel images and the text description of the driving scene are input. The self-attention mechanism is used to perform joint feature learning on the pre-processed multimodal data. After the embedding layer, position encoding, and multi-head self-attention layer, the temporal dynamic information and spatial relationship information are fused and the fused features are output. Step (13): Introduce a hierarchical clustering method to classify the fused features and assign a label to each sample scene to achieve real-time judgment and dynamic labeling of driving status; Step (14): For the predicted output at each moment in the model, calculate the category probability distribution and use information entropy to measure model uncertainty; including: Information entropy calculation: Among them, x i is the input data of the i-th sample, p ij For sample x i The probability of belonging to category j, information entropy H(x i ) reflects the uncertainty within the scenario; Weight definition and embedding: Information entropy H(x i ) into the loss function and define the sample weights: w(x i )=1+λ·H(x i ) (2) Where λ is the adjustment hyperparameter; The modified scene data clustering hierarchical model loss function is: Where N is the total number of tasks of the data distribution optimization model, f(x i ) is the model input x i The predicted output, Ψ i is the true label, l(f(x i ),Ψ i ) is the traditional cross entropy loss function, w(x i ) is the dynamically calculated weight; Step (15): Construct a risk quantification module, dynamically adjust the weights of high-risk and long-tail scenarios in the driving data based on the statistical results of the information entropy of each scenario, and feed it back to the data collection and large-scale model retraining links to achieve continuous optimization of data distribution, generate a multi-level scenario library based on balanced real-car driving data, and use it for autonomous driving large-scale model training.

3. The method according to claim 1, characterized in that Step (3) includes the following steps: Generate typical scenarios in the open-source scenario editor, overlay long-tail risk conditions, and cover the diverse needs of real roads; Synchronously collect high-frame-rate video images, lidar point clouds, and vehicle control signals, automatically annotate scene text using cross-modal semantic association technology, and improve data quality using adversarial sample generation and multimodal consistency verification. Build a digital twin verification system, reproduce real traffic scenarios in the simulation platform, combine hardware-in-the-loop testing with a progressive real-vehicle verification strategy, and optimize the model's scene understanding, trajectory planning, and safety decision-making capabilities in stages, ultimately forming a high-fidelity simulation data test set with strong semantic associations and high safety redundancy.

4. The method according to claim 1, wherein Step (4) specifically includes the following steps: Step (41): Carry out the evaluation of the decision and planning module of the visual language large model, and use the minimum average displacement error ADE and the end point displacement error FDE to evaluate the trajectory prediction accuracy, as follows: where x t 、y t Indicates the actual vehicle position coordinates at time t, x T 、y T Indicates the actual vehicle position coordinates at the final moment; represents the predicted position coordinates at time t, Represents the predicted position coordinates at the final moment; T represents the total time step of the predicted trajectory; Step (42): Using grid search and random search methods to process a small-scale hyperparameter space, a proxy model is constructed based on the Bayesian optimization method to maximize the expected improvement function: EI(x)=E[max(f(θ)-f(θ + )),0] (11) Where E represents the mathematical expectation, f(θ) represents the objective function, and f(θ + ) represents the currently known optimal objective function value, and θ represents the hyperparameter combination to be optimized; Step (43): Use the knowledge distillation method to transfer the knowledge of the large model to the lightweight model. The loss function of the lightweight model is designed as: Among them, α and β are the weight coefficients of cross entropy loss and feature alignment loss, L CE is the cross entropy loss, used to supervise the classification output of the student model, L MSE is the feature alignment loss, used to align the teacher model f t (x) and the student model f s (x) represents the feature representation, p represents the classification output of the student model, Represents the true label of the sample; Step (44): Use the cosine annealing strategy to balance convergence speed and stability, and dynamic learning rate scheduling: where Λ min , Λ max are the minimum and maximum values ​​of the learning rate, u is the current number of training steps, and U is the total number of training steps in one cycle; Step (45): Quantize and prune the large model, convert the floating-point weights into 8-bit integers to reduce storage and computational overhead. The formula is as follows: in is a floating-point weight parameter, are the maximum and minimum values ​​of the weight, b is the number of bits after quantization, and round(*) means converting the scaled floating-point value into an integer to make it discrete; Perform structured pruning on the large model and remove redundant channels based on L1 regularization. The objective function of the pruned model is: Among them L task is the task loss, φ is the coefficient of the L1 regularization term, which controls the pruning intensity, and W l is the weight matrix of the first layer.

Images