Automatic driving model online adaptive method, device, equipment and medium

Abstract

The invention provides an automatic driving model online self-adaption method and device, equipment and a medium, and belongs to the technical field of automatic driving, and the method comprises the steps: constructing a meta training data set, and employing a model-independent meta learning algorithm to train a lightweight neural network, and obtaining an initial model; deploying the initial model at a vehicle end, monitoring the sensing confidence coefficient in real time, triggering an adaptive process when the sensing confidence coefficient is lower than a threshold value, controlling the vehicle to enter a safe state, and collecting time sequence sensor data; executing gradient updating at a vehicle end by taking parameters of the initial model as a starting point and time sequence sensor data as a support set, and generating a fine tuning model; verifying the fine tuning model in the sandbox environment, and switching the driving control model of the vehicle end into the fine tuning model when the verification is passed; and the updated gradient information of each vehicle is encrypted and uploaded to the cloud and aggregated, and the initial model is updated. According to the invention, the adaptability and safety of the automatic driving system in an unknown scene are improved, the utilization of computing resources is optimized, and vehicle cloud co-evolution is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to an automatic driving model online self-adaption method, device, equipment and medium. BACKGROUND

[0002] In automatic driving, an automatic driving model of artificial intelligence realizes autonomous environment perception and decision control of a vehicle, and is a key direction for improving traffic safety, efficiency and comfort. The automatic driving model needs to be robust and reliable in a complex open road environment.

[0003] However, existing automatic driving systems still face severe challenges, which are embodied in the following aspects: first, a long tail problem, the automatic driving model is difficult to cover all possible driving scenes in the training stage, especially those rare but dangerous scenes (such as extreme weather, sudden obstacles, etc.). When the automatic model encounters a scene not fully covered in the training data, its perception and decision performance will sharply decrease, leading to safety hazards. Second, the automatic driving model update lags behind, and the current mainstream solution relies on retraining the automatic driving model in the cloud and updating it through OTA (over the air). This method has a long cycle from data collection, labeling, training to deployment, usually several weeks or even months, and cannot meet the real-time safety needs of sudden or local new scenes. Third, the contradiction between computing resources and self-adaption ability, large neural network models with strong online learning ability are difficult to run in real time on resource-limited vehicle embedded platforms; while lightweight models that can run on the vehicle end usually lack rapid self-adaption ability and are difficult to achieve efficient learning with a small amount of new data.

[0004] Although there are some solutions to solve the above problems at present, such as cloud OTA update solution and traditional online learning solution. The former relies on periodic data backhaul and model retraining, and its response speed is slow, which cannot meet the real-time safety needs. The latter directly uses new data to update the model on the vehicle end, but has a catastrophic forgetting problem, that is, the model quickly forgets old knowledge when learning new knowledge, leading to performance degradation in known scenes; in addition, large model update is time-consuming and laborious, which does not meet the real-time requirements of the vehicle end, and lacks effective safety verification mechanism.

[0005] Therefore, how to make lightweight models on the resource-limited vehicle end use a small amount of new scene data to achieve online self-adaption quickly, safely and without forgetting existing knowledge has become a key technical bottleneck restricting the realization of lifelong learning and global reliable operation of the automatic driving system. SUMMARY

[0006] In a first aspect, an embodiment of the present application provides an automatic driving model online self-adaption method, comprising the following steps: S1. Construct a meta-training dataset based on multi-scene driving data, and train a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial automatic driving model sensitive to driving task changes; S2. Deploy the initial automatic driving model on the vehicle end, monitor the perception confidence of the initial automatic driving model in real time, and trigger an adaptive process when the perception confidence is lower than a first threshold, control the vehicle to enter a safe state and collect time-series sensor data of the current scene; S3. Starting from the parameters of the initial automatic driving model, and using the time-series sensor data as a support set, perform gradient updates for a set number of steps on the vehicle end to generate a fine-tuned automatic driving model; S4. Verify whether the fine-tuned automatic driving model meets the following two conditions in a sandbox environment: The fine-tuned automatic driving model performs better than the initial automatic driving model in new scenes; The performance degradation of the fine-tuned automatic driving model in known scenes is lower than a second threshold; And when both conditions are met, switch the driving control model on the vehicle end to the fine-tuned automatic driving model; S5. Encrypt the gradient information of the adaptive gradient update and upload it to the cloud, and aggregate the gradient information of each vehicle to update the initial automatic driving model.

[0007] Further, step S1 includes the following steps: S11. Collect data in various driving scenarios, define corresponding automatic driving tasks for each driving scenario, and generate a meta-training dataset by treating each driving task with corresponding support set and query set data as a meta-task; S12. Use a depth separable convolutional network as the backbone network, and perform channel pruning on the backbone network to generate a lightweight neural network as an automatic driving base model; S13. Perform meta-learning training, use a model-independent meta-learning algorithm to perform meta-learning training on the automatic driving base model on the meta-training dataset, and output a set of initial parameters sensitive to driving task changes to generate an initial automatic driving model after the meta-learning training is completed.

[0008] Further, in step S13, the training process of the model-independent meta-learning algorithm includes inner loop update and outer loop update; Wherein, the inner loop update uses the corresponding support set data to perform gradient descent on the model parameters of the automatic driving base model for a preset number of steps to obtain adapted parameters for the corresponding meta-task; The outer loop update optimizes the initial parameters of the automatic driving base model based on the loss of all meta-tasks on the query set.

[0009] Furthermore, the specific steps of step S2 are as follows: S21. Deploy the initial autonomous driving model on the in-vehicle computing platform and run it during vehicle operation to perform real-time environmental perception or vehicle control tasks; S22. Calculate the confidence or entropy value of the perception results output by the initial autonomous driving model and use it as the perception confidence. S23. When the perception confidence of a set number of consecutive time steps is lower than the first threshold, it is determined that the initial autonomous driving model has insufficient understanding of the current driving scenario, and the online adaptive process is triggered. S24. Execute the online adaptive process, and bring the vehicle into a minimum risk state through takeover prompts or automatic control. At the same time, collect the timing synchronization data of the camera and lidar within a future set time period to generate timing sensor data.

[0010] Furthermore, the specific steps of step S3 are as follows: S31. Load the parameters of the initial autonomous driving model onto the vehicle's autonomous driving controller or onboard computing platform; S32. Define the acquired time-series sensor data as a new driving task, and use the time-series sensor data as the support set for the defined driving task; S33. Perform meta-learning inner loop fine-tuning, starting with the parameters of the initial autonomous driving model, and perform an inner loop update process of model-independent meta-learning for the defined driving task:

[0011] in, These are the parameters for the initial autonomous driving model. For the inner loop learning rate, For the loss function of the new driving task on the support set, These are the updated parameters of the initial autonomous driving model; S34. After iterating through the inner loop update process a set number of times, stop updating and use the final obtained model parameters as parameters for fine-tuning the autonomous driving model.

[0012] Furthermore, the specific steps of step S4 are as follows: S41. Deploy the fine-tuned autonomous driving model in a sandbox environment that isolates the vehicle from the actual vehicle control system; S42. Use the newly acquired scene time-series sensor data fragments, as well as data fragments representing known driving scenarios extracted from pre-stored data on the vehicle, as a validation dataset; S43. In a sandbox environment, forward inference is performed on the validation dataset using the initial autonomous driving model and the fine-tuned autonomous driving model, respectively. The performance metrics of the two models on new scene segments and known scene segments are compared. The performance metrics include the average accuracy of object detection, the accuracy of lane line detection, or the trajectory tracking error of vehicle control commands. S44. Determine whether the following two conditions are met: The fine-tuned autonomous driving model outperformed the initial autonomous driving model in performance metrics on new scene segments. The performance metric degradation on the known scene segment is below the second threshold; When both conditions are met, a switching command is triggered to smoothly replace the driving control model in the real vehicle control system with a fine-tuned autonomous driving model.

[0013] Furthermore, the specific steps of step S5 are as follows: S51. On the vehicle side, the gradient vector generated by the gradient update in step S3 or the parameter change of the fine-tuned autonomous driving model verified in step S4 is encapsulated into a knowledge package. S52. The knowledge package is encrypted and then uploaded to the cloud server via the vehicle network; S53. The cloud server receives the encrypted knowledge packets from each vehicle and decrypts them to obtain model update information. It then uses a federated averaging algorithm to aggregate the update information from each model to obtain global update information. S54. Use global update information to optimize the parameters of the initial autonomous driving model stored in the cloud, and generate a new initial autonomous driving model for subsequent deployment to the vehicle.

[0014] Secondly, embodiments of this application also provide an online adaptive device for an autonomous driving model, comprising: The cloud-based meta-training module is used to construct a meta-training dataset based on multi-scenario driving data, and to train a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial autonomous driving model that is sensitive to changes in driving tasks. The vehicle-side model deployment and monitoring module is used to deploy the initial autonomous driving model on the vehicle, monitor the perception confidence of the initial autonomous driving model in real time, and trigger an adaptive process when the perception confidence is lower than a first threshold. The safety data acquisition and control module executes an adaptive process to control the vehicle to enter a safe state and acquire time-series sensor data of the current scene; The vehicle-side online meta-learning module is used to perform gradient updates on the vehicle side for a set number of steps, starting from the parameters of the initial autonomous driving model and using the time-series sensor data as the support set, to generate a fine-tuned autonomous driving model. The safety verification and switching module is used to verify in a sandbox environment whether the fine-tuned autonomous driving model meets the following two conditions: The fine-tuned autonomous driving model outperforms the initial autonomous driving model in new scenarios. The performance of the fine-tuned autonomous driving model degrades below the second threshold in known scenarios; And when both conditions are met, the vehicle-side driving control model is switched to the fine-tuned autonomous driving model; The knowledge feedback and aggregation evolution module is used to encrypt and upload the gradient information of this adaptive gradient update to the cloud, and aggregate the gradient information of each vehicle to update the initial autonomous driving model.

[0015] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the online adaptive method for an autonomous driving model as described in the first aspect.

[0016] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the online adaptive method for the autonomous driving model as described in the first aspect.

[0017] As can be seen from the above technical solutions, this application has the following advantages: The online adaptive method, device, equipment, and medium for autonomous driving models provided in this application enable the autonomous driving model to quickly adapt to new scenarios through an online adaptive mechanism, improving performance in unknown or rare scenarios and enhancing the safety and reliability of autonomous driving. The adaptive results from the vehicle are fed back to the cloud for aggregation, continuously optimizing the initial cloud model to form a complete adaptive closed loop, achieving lifelong learning and evolution of the entire system. Lightweight neural networks and meta-learning are used to reduce the computational burden of the model on the vehicle, ensuring rapid and efficient adaptive updates on resource-constrained vehicle platforms. Through sandbox environment verification and safety mechanisms, it is ensured that the fine-tuned model undergoes rigorous verification before being put into actual control, avoiding the introduction of new risks due to model updates. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the online adaptive method for autonomous driving models according to the present invention.

[0020] Figure 2 This is a schematic diagram of the online adaptive device for the autonomous driving model of the present invention. Detailed Implementation

[0021] The various embodiments of this disclosure will be described more fully in the following detailed description of the specific steps of the online adaptive method for autonomous driving models. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0022] This embodiment provides an online adaptive method for autonomous driving models, which improves the adaptability and safety of autonomous driving, optimizes vehicle-cloud collaboration, and efficiently utilizes computing resources.

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figure 1 The diagram shows a flowchart of an online adaptive method for an autonomous driving model in a specific embodiment. The method includes the following steps: S1. Construct a meta-training dataset based on multi-scenario driving data, and use a model-independent meta-learning algorithm to train a lightweight neural network to obtain an initial autonomous driving model that is sensitive to changes in driving tasks; It should be noted that by constructing a meta-training dataset based on multi-scenario driving data and training a lightweight neural network using a model-independent meta-learning algorithm, an initial autonomous driving model sensitive to changes in driving tasks is obtained. This provides a foundation for rapid adaptation on the vehicle side, ensuring that the initial model has the ability to quickly adapt to new tasks. At the same time, the lightweight processing reduces the computational complexity and storage requirements of the model, making it more suitable for running on the vehicle side. S2. Deploy the initial autonomous driving model on the vehicle, monitor the perception confidence of the initial autonomous driving model in real time, and trigger the adaptive process when the perception confidence is lower than the first threshold to control the vehicle to enter a safe state and collect the time-series sensor data of the current scene. It should be noted that the initial autonomous driving model is deployed on the vehicle and its perception confidence is monitored in real time. When the perception confidence is lower than the threshold, the adaptive process is triggered to control the vehicle to enter a safe state and collect the time-series sensor data of the current scene. This allows for timely detection of the model's insufficient understanding of the current scene, ensuring that the vehicle can quickly take safety measures when encountering unknown scenarios, and providing data support for subsequent model fine-tuning. S3. Starting with the parameters of the initial autonomous driving model and using the time-series sensor data as the support set, perform a set number of gradient updates on the vehicle to generate a fine-tuned autonomous driving model. It should be noted that, starting with the parameters of the initial autonomous driving model, the collected time-series sensor data is used as the support set, and a set number of gradient updates are performed on the vehicle to generate a fine-tuned autonomous driving model. Through the meta-learning inner loop update process, the model is quickly optimized with very little data and limited computing resources, which improves the speed and effect of vehicle-side adaptation and enables the model to quickly adapt to new scenarios. S4. Verify in a sandbox environment whether the fine-tuned autonomous driving model meets the following two conditions: The fine-tuned autonomous driving model outperforms the initial autonomous driving model in new scenarios. The performance of the fine-tuned autonomous driving model degrades below the second threshold in known scenarios; And when both conditions are met, the vehicle-side driving control model is switched to the fine-tuned autonomous driving model; It should be noted that the sandbox environment is used to verify whether the fine-tuned autonomous driving model meets the conditions for performance improvement and controllable performance degradation. When the conditions are met, the driving control model on the vehicle is switched to the fine-tuned model. Through a strict verification mechanism, it is ensured that the fine-tuned model improves performance in new scenarios and has controllable performance degradation in known scenarios, thereby ensuring the safety and reliability of model updates and avoiding the introduction of new safety hazards due to model updates. S5. The gradient information of this adaptive gradient update is encrypted and uploaded to the cloud, and the gradient information of each vehicle is aggregated to update the initial autonomous driving model; It should be noted that the vehicle-side adaptive gradient update information is encrypted and uploaded to the cloud, and the gradient information of each vehicle is aggregated to update the initial autonomous driving model. This enables experience sharing among fleet members and continuous optimization of the initial cloud model, forming a complete adaptive closed loop. This improves the adaptive capability and evolutionary efficiency of the entire system, allowing the system to continuously learn and adapt to new driving scenarios.

[0025] This embodiment enhances the adaptability and safety of the autonomous driving system in unknown scenarios through cloud pre-training and rapid vehicle-side adaptation; it optimizes the utilization of computing resources by employing lightweight neural networks and meta-learning to ensure efficient adaptive updates on resource-constrained vehicles; and it guarantees the safety and reliability of the update process through sandbox verification and knowledge feedback mechanisms, forming a closed loop of vehicle-cloud collaborative evolution to improve the overall performance and evolution efficiency of the system.

[0026] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, in order to fully illustrate the specific implementation process in this embodiment, another online adaptive method for autonomous driving models is provided, taking the rare combined scenario of an autonomous vehicle encountering sudden fog and road construction obstructing traffic as an example. Cloud equipment: It adopts a distributed cloud server cluster, including 8 high-performance GPU servers (each configured with NVIDIA A100 GPU and 128GB of memory) for meta-training dataset construction, initial model training and global gradient aggregation. Vehicle-side equipment: Equipped with an onboard computing platform (computing power ≥200TOPS), integrating a multimodal sensor suite (including 4 8MP high-definition cameras, 2 128-line LiDARs, and 3 millimeter-wave radars), supporting time-series synchronous acquisition of sensor data; at the same time, an independent sandbox verification unit is deployed, physically isolated from the actual vehicle control system to ensure the safety of the verification process; (II) Software and Parameter Configuration Basic model configuration: The lightweight neural network adopts a deep separable convolutional network (based on the MobileNetV3 architecture). After channel pruning, the number of parameters is compressed to 8.6M, and the computational cost is reduced to 1.2GFLOPs. The learning rate of the model-independent meta-learning algorithm is α=0.01 for the inner loop and β=0.001 for the outer loop. The number of inner loop update steps is preset to 5, and the number of vehicle-side fine-tuning iterations is set to 8. Threshold settings: The first threshold (perceptual confidence trigger threshold) is set to 0.75, that is, when the perceptual confidence is lower than 0.75 for 5 consecutive time steps (0.1 seconds per time step), the adaptive process is triggered; the second threshold (known scene performance degradation threshold) is set to 5%, that is, the performance index of the fine-tuned model in the known scene does not decrease by more than 5%; Data configuration: The meta-training dataset contains 2000 driving scenarios, covering common scenarios such as sunny days, rainy days, nighttime, and traffic jams, as well as a small number of rare scenarios such as extreme weather and sudden obstacles. Each scenario corresponds to one meta-task, and each meta-task contains a support set (80 samples) and a query set (320 samples). 1000 known scenario data fragments are pre-stored on the vehicle side for performance evaluation of known scenarios in the sandbox validation. The method includes the following steps: S1. Construct a meta-training dataset based on multi-scenario driving data, and train a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial autonomous driving model sensitive to changes in driving tasks; the specific steps of step S1 are as follows: S11. Collect data from various driving scenarios, define corresponding autonomous driving tasks for each driving scenario, and combine each driving task with its corresponding support set and query set data as a meta-task to generate a meta-training dataset. For example, driving data of the fleet is collected in multiple regions and under multiple road conditions across the country. The data types include raw sensor data (images, point clouds, radar signals), vehicle status data (vehicle speed, steering angle, braking signals), and scene annotation information (weather, road conditions, obstacle types).

[0027] The collected data is preprocessed as follows: sensor data time-series synchronization (timestamp error ≤1ms), image distortion correction, point cloud denoising and registration; driving tasks are divided according to scene type, a total of 2000 meta-tasks are defined, each meta-task is split into a support set (for inner loop adaptation) and a query set (for outer loop optimization), and finally a meta-training dataset with a total scale of 800,000 samples is formed. S12. A depthwise separable convolutional network is used as the backbone network, and channel pruning is performed on the backbone network to generate a lightweight neural network as the base model for autonomous driving. For example, a depthwise separable convolutional network is used as the backbone network to replace the traditional standard convolutional layer: the 3×3 standard convolution is decomposed into 3×3 depthwise convolution (channel-wise convolution) and 1×1 pointwise convolution (channel fusion), reducing the computation by 75% while maintaining the feature extraction capability.

[0028] L1 regularized channel pruning is adopted: during training, the importance weight of each channel is calculated, and channels with an absolute weight value less than 0.001 are removed. Finally, 60% of the core channels are retained to generate a compact and lightweight autonomous driving base model. S13. Perform meta-learning training. Use the model-independent meta-learning algorithm to perform meta-learning training on the autonomous driving base model on the meta-training dataset. After the meta-learning training is completed, output a set of initial parameters that are sensitive to changes in driving tasks to generate the initial autonomous driving model. In step S13, the training process of the model-independent meta-learning algorithm includes inner loop update and outer loop update. For example, training is performed alternately between inner and outer loops to optimize initial parameters: Inner loop update: 16 meta-tasks are randomly sampled from the meta-training dataset, and for each meta-task, the loss function is computed using its support set data. Update model parameters using gradient descent: Each meta-task performs a 3-step inner loop update to obtain temporary parameters adapted to that task. Outer loop update: Use the query set data from each meta-task to calculate temporary parameters. The inference loss is accumulated by summing the query set loss of all meta-tasks, with respect to the initial parameters. Perform global optimization: ; Repeat the inner and outer loop process described above for a total of 100 epochs. Stop training when the average loss on the query set converges to below 0.08, and output initial parameters that are sensitive to changes in the driving task. Generate an initial autonomous driving model; The inner loop update, for each meta-task, uses the corresponding support set data to perform gradient descent on the model parameters of the autonomous driving base model for a preset number of steps to obtain the adaptation parameters for the corresponding meta-task. The outer loop update optimizes the initial parameters of the autonomous driving base model based on the loss of all meta-tasks on the query set; S2. Deploy the initial autonomous driving model on the vehicle, monitor the perception confidence level of the initial autonomous driving model in real time, and trigger the adaptive process when the perception confidence level is lower than the first threshold to control the vehicle to enter a safe state and collect the time-series sensor data of the current scene; the specific steps of step S2 are as follows: S21. Deploy the initial autonomous driving model on the in-vehicle computing platform and run it during vehicle operation to perform real-time environmental perception or vehicle control tasks; For example, the initial autonomous driving model trained in the cloud is securely transmitted to the vehicle computing platform via the Internet of Vehicles. After deployment, it establishes communication with the sensor suite and vehicle control system, and performs environmental perception (target detection, lane line recognition, scene classification) and vehicle control tasks in real time during vehicle operation. S22. Calculate the confidence or entropy value of the perception results output by the initial autonomous driving model and use it as the perception confidence. S23. When the perception confidence of a set number of consecutive time steps is lower than the first threshold, it is determined that the initial autonomous driving model has insufficient understanding of the current driving scenario, and the online adaptive process is triggered. For example, when a vehicle is driving on a highway, the initial model outputs perception results in real time (position of the vehicle ahead, lane coordinates, obstacle type) and calculates perception confidence: the maximum probability value output by Softmax is used as the confidence value for target detection results, and the matching degree between the predicted trajectory and the actual trajectory is used as the confidence value for lane detection. The combined perception confidence value is obtained by combining the two. When a vehicle enters a sudden foggy area, visibility drops sharply to 50 meters, camera image contrast decreases, and lidar point cloud is interfered with by fog scattering. The model's confidence in detecting vehicles ahead drops to 0.62, and the confidence in lane line recognition drops to 0.58. Moreover, the overall perception confidence for five consecutive time steps (0.5 seconds) is lower than the first threshold of 0.75. It is determined that the model has insufficient understanding of the current "fog + road construction occupying lanes" composite scenario, triggering the online adaptive process. S24. Execute the online adaptive process, and bring the vehicle into a minimum risk state through takeover prompts or automatic control, while collecting the time-series synchronization data of the camera and lidar within a future set time period to generate time-series sensor data; For example, the system immediately activates the safety mechanism: it issues a takeover prompt to the driver through the in-vehicle display and voice prompt, and at the same time automatically and smoothly reduces the vehicle speed from 100km / h to 60km / h, keeps the vehicle in the center of the lane, and enters a minimum risk state; Synchronous acquisition of time-series sensor data for the next 30 seconds: The camera acquires one image frame every 0.05 seconds, and the lidar acquires one point cloud frame every 0.1 seconds to ensure data synchronization. A total of 600 image frames and 300 point cloud frames are acquired to generate time-series sensor data (including key information such as fog environment characteristics, cone locations in the construction area, and road occupancy range). S3. Starting with the parameters of the initial autonomous driving model and using the time-series sensor data as the support set, perform a set number of gradient updates on the vehicle to generate a fine-tuned autonomous driving model; the specific steps of step S3 are as follows: S31. Load the parameters of the initial autonomous driving model onto the vehicle's autonomous driving controller or onboard computing platform; S32. Define the acquired time-series sensor data as a new driving task, and use the time-series sensor data as the support set for the defined driving task; For example, the parameters of the initial autonomous driving model are loaded on the on-board computing platform. Ensure parameter integrity and consistency. Define the collected time-series sensor data as the new driving task. (“Dense fog + construction road occupation” scenario task), and use this time series data as the support set for the new task (50 samples, including construction cones, road occupation areas and foggy environment samples at different angles and distances). S33. Perform meta-learning inner loop fine-tuning, starting with the parameters of the initial autonomous driving model, and perform an inner loop update process of model-independent meta-learning for the defined driving task:

[0029] in, These are the parameters for the initial autonomous driving model. For the inner loop learning rate, For the loss function of the new driving task on the support set, These are the updated parameters of the initial autonomous driving model; S34. After iterating through the inner loop update process a set number of times, stop updating and use the final model parameters as parameters for fine-tuning the autonomous driving model; For example, fine-tuning of the meta-learning inner loop is performed based on the initial parameters. The gradient is updated using the following formula, along with the new task support set: The loss function for the new task on the support set is a weighted sum of cross-entropy loss and regression loss. The inner loop update process is iterated 8 times, calculating the loss function value after each update. The update stops when the loss value drops below 0.12. The final model parameters are then obtained. This involves fine-tuning the parameters of the autonomous driving model. The entire fine-tuning process takes 28 seconds, meeting the real-time requirements of the vehicle. S4. Verify in a sandbox environment whether the fine-tuned autonomous driving model meets the following two conditions: The fine-tuned autonomous driving model outperforms the initial autonomous driving model in new scenarios. The performance of the fine-tuned autonomous driving model degrades below the second threshold in known scenarios; And when both conditions are met, the vehicle-side driving control model is switched to the fine-tuned autonomous driving model; the specific steps of step S4 are as follows: S41. Deploy the fine-tuned autonomous driving model in a sandbox environment that isolates the vehicle from the actual vehicle control system; S42. Use the newly acquired scene time-series sensor data fragments, as well as data fragments representing known driving scenarios extracted from pre-stored data on the vehicle, as a validation dataset; For example, the fine-tuned autonomous driving model is deployed to a vehicle-side sandbox environment, which is completely isolated from the real vehicle control system. This environment only receives sensor data input and does not output actual control commands, thus avoiding the verification process from affecting vehicle driving safety.

[0030] Construct a validation dataset: consisting of two parts—a new scenario data segment (100 samples extracted from collected time-series sensor data, covering changes in fog concentration and different perspectives of the construction area) and a known scenario data segment (200 samples extracted from pre-stored data on the vehicle, covering 10 known scenarios such as highways in sunny weather and urban roads in rainy weather). S43. In a sandbox environment, forward inference is performed on the validation dataset using the initial autonomous driving model and the fine-tuned autonomous driving model, respectively. The performance metrics of the two models on new scene segments and known scene segments are compared. The performance metrics include the average accuracy of object detection, the accuracy of lane line detection, or the trajectory tracking error of vehicle control commands. For example, the performance metrics for the new scenario are: mean accuracy of target detection (mAP), lane detection accuracy, and trajectory tracking error. Test results show that the fine-tuned model improved vehicle detection mAP in the new scenario from 58.2% to 82.6%, lane detection accuracy from 61.5% to 85.3%, and trajectory tracking error from 0.8m to 0.3m, all of which are superior to the initial model. Known performance metrics: In a sunny highway scene, the fine-tuned model achieved a target detection mAP of 92.1%, compared to 95.3% for the initial model, representing a performance degradation of 3.4%. In a rainy urban road scene, the fine-tuned model achieved a lane detection accuracy of 89.7%, compared to 93.5% for the initial model, representing a performance degradation of 4.1%, both below the second threshold of 5%. S44. Determine whether the following two conditions are met: The fine-tuned autonomous driving model outperformed the initial autonomous driving model in performance metrics on new scene segments. The performance metric degradation on the known scene segment is below the second threshold; When both conditions are met, a switching command is triggered to smoothly replace the driving control model in the real vehicle control system with a fine-tuned autonomous driving model. For example, since the fine-tuned model meets the two conditions of "the performance of the new scenario is better than the initial model" and "the performance degradation of the known scenario is lower than the second threshold", the sandbox verification unit triggers a switching command. The driving control model in the real vehicle control system is replaced from the initial model to the fine-tuned model through a smooth transition algorithm. The switching process takes 0.3 seconds, with no control interruption or jitter. The vehicle recovers stable perception and control capabilities in the fog + construction road occupation scenario. S5. Encrypt and upload the gradient information of this adaptive gradient update to the cloud, and aggregate the gradient information of each vehicle to update the initial autonomous driving model; the specific steps of step S5 are as follows: S51. On the vehicle side, the gradient vector generated by the gradient update in step S3 or the parameter change of the fine-tuned autonomous driving model verified in step S4 is encapsulated into a knowledge package. S52. The knowledge package is encrypted and then uploaded to the cloud server via the vehicle network; For example, the gradient vector (with the same dimensions as the model parameters, a total of 8.6M dimensions) generated by the gradient update in step S3 is extracted on the vehicle and encapsulated into a knowledge package containing gradient data, scene type labels ("fog + construction road occupation"), update timestamps and other information. The knowledge packet is encrypted using the AES-256 encryption algorithm through vehicle-to-everything (V2X) and uploaded to the cloud server. The transmission latency is controlled within 1.2 seconds to ensure data transmission security and efficiency. S53. The cloud server receives the encrypted knowledge packets from each vehicle and decrypts them to obtain model update information. It then uses a federated averaging algorithm to aggregate the update information from each model to obtain global update information. S54. Use global update information to optimize the parameters of the initial autonomous driving model stored in the cloud, and generate a new initial autonomous driving model for subsequent deployment to the vehicle. For example, the cloud server receives encrypted knowledge packets uploaded from 200 autonomous vehicles nationwide, decrypts them, extracts model update information (gradient vectors or parameter changes), and aggregates all update information using a federated averaging algorithm: Where N is the number of vehicles participating in the aggregation. Let be the gradient update amount for the i-th vehicle; using the aggregated global update information, optimize the initial autonomous driving model parameters stored in the cloud to generate a new initial model (parameters). The new model integrates the adaptive experience of multiple vehicles in different rare scenarios, improving the initial perception confidence of scenarios such as "fog patches" and "construction blocking the road" to over 0.85. Subsequently, the new initial model will be deployed to new vehicles or the models of existing vehicles will be updated regularly through the Internet of Vehicles, realizing the vehicle-cloud collaborative evolution of the entire system.

[0031] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0032] like Figure 2 As shown, the following are embodiments of the online adaptive device for autonomous driving models provided in this disclosure. This device and the online adaptive method for autonomous driving models in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the online adaptive device for autonomous driving models, please refer to the embodiments of the online adaptive method for autonomous driving models described above.

[0033] The device includes: The cloud-based meta-training module is used to construct a meta-training dataset based on multi-scenario driving data, and to train a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial autonomous driving model that is sensitive to changes in driving tasks. The vehicle-side model deployment and monitoring module is used to deploy the initial autonomous driving model on the vehicle, monitor the perception confidence of the initial autonomous driving model in real time, and trigger an adaptive process when the perception confidence is lower than a first threshold. The safety data acquisition and control module executes an adaptive process to control the vehicle to enter a safe state and acquire time-series sensor data of the current scene; The vehicle-side online meta-learning module is used to perform gradient updates on the vehicle side for a set number of steps, starting from the parameters of the initial autonomous driving model and using the time-series sensor data as the support set, to generate a fine-tuned autonomous driving model. The safety verification and switching module is used to verify in a sandbox environment whether the fine-tuned autonomous driving model meets the following two conditions: The fine-tuned autonomous driving model outperforms the initial autonomous driving model in new scenarios. The performance of the fine-tuned autonomous driving model degrades below the second threshold in known scenarios; And when both conditions are met, the vehicle-side driving control model is switched to the fine-tuned autonomous driving model; The knowledge feedback and aggregation evolution module is used to encrypt and upload the gradient information of this adaptive gradient update to the cloud, and aggregate the gradient information of each vehicle to update the initial autonomous driving model.

[0034] This embodiment improves the adaptability and safety of the autonomous driving system in unknown scenarios through the interactive collaboration of the cloud-based meta-training module, the vehicle-side model deployment and monitoring module, the safety data acquisition and control module, the vehicle-side online meta-learning module, the safety verification and switching module, and the knowledge feedback and aggregation evolution module. It adopts lightweight neural networks and meta-learning to optimize the utilization of computing resources, realize the vehicle-cloud collaborative evolution closed loop, ensure the safety and reliability of the update process, and improve the overall system performance.

[0035] The online adaptive method for autonomous driving models provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0036] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0037] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0038] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0039] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0040] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0041] The aforementioned electronic device implements the online adaptive method for autonomous driving models of this application, which constructs a meta-training dataset based on multi-scenario driving data and trains a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial autonomous driving model sensitive to changes in driving tasks. The initial autonomous driving model is deployed on the vehicle, and its perception confidence is monitored in real time. When the perception confidence falls below a first threshold, an adaptive process is triggered to control the vehicle into a safe state and collect temporal sensor data of the current scenario. Starting with the parameters of the initial autonomous driving model and using the temporal sensor data as a support set, a gradient update of a set number of steps is performed on the vehicle to generate a fine-tuned autonomous driving model. The model is then validated in a sandbox environment. The fine-tuned autonomous driving model meets the following two conditions: the fine-tuned autonomous driving model outperforms the initial autonomous driving model in the new scenario; the performance degradation of the fine-tuned autonomous driving model in the known scenario is below a second threshold; and when both conditions are met, the driving control model on the vehicle is switched to the fine-tuned autonomous driving model; the gradient information of this adaptive gradient update is encrypted and uploaded to the cloud, and the gradient information of each vehicle is aggregated to update the technical solution of the initial autonomous driving model. This achieves the beneficial effects of improving the adaptability and safety of the autonomous driving system in unknown scenarios through cloud pre-training and rapid vehicle-side adaptation, optimizing the utilization of computing resources, forming a vehicle-cloud collaborative evolution closed loop, and ensuring the safety and reliability of the update process.

[0042] The storage medium provided in this application stores a program product capable of implementing an online adaptive method for autonomous driving models.

[0043] The online adaptive method for autonomous driving models includes: constructing a meta-training dataset based on multi-scenario driving data, and training a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial autonomous driving model sensitive to changes in driving tasks; deploying the initial autonomous driving model on the vehicle, monitoring the perception confidence of the initial autonomous driving model in real time, and triggering an adaptive process when the perception confidence falls below a first threshold, controlling the vehicle to enter a safe state and collecting time-series sensor data of the current scenario; using the parameters of the initial autonomous driving model as a starting point and the time-series sensor data as a support set, performing a set number of gradient updates on the vehicle to generate a fine-tuned autonomous driving model; verifying in a sandbox environment whether the fine-tuned autonomous driving model meets the following two conditions: the performance of the fine-tuned autonomous driving model in the new scenario is better than that of the initial autonomous driving model; the performance degradation of the fine-tuned autonomous driving model in the known scenario is lower than a second threshold; and when both conditions are met, switching the driving control model on the vehicle to the fine-tuned autonomous driving model; encrypting and uploading the gradient information of this adaptive gradient update to the cloud, and aggregating the gradient information of each vehicle to update the initial autonomous driving model.

[0044] In some possible implementations, the online adaptive method for autonomous driving models disclosed herein can be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0045] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0046] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An online self-adaptive method for an automatic driving model, characterized in that, Comprising the following steps: S1. Constructing a meta-training dataset based on multi-scene driving data, and training a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial automatic driving model sensitive to driving task changes; S2. Deploying the initial automatic driving model on the vehicle end, monitoring the perception confidence of the initial automatic driving model in real time, and triggering an adaptive process when the perception confidence is lower than a first threshold, controlling the vehicle to enter a safe state and collecting time-series sensor data of the current scene; S3. Taking the parameters of the initial automatic driving model as the starting point and the time-series sensor data as the support set, performing gradient updates for a set number of steps on the vehicle end to generate a fine-tuned automatic driving model; S4. Verifying in a sandbox environment whether the fine-tuned automatic driving model meets the following two conditions: The fine-tuned automatic driving model performs better than the initial automatic driving model in new scenes; The performance degradation of the fine-tuned automatic driving model in known scenes is lower than a second threshold; And when both conditions are met, switching the driving control model on the vehicle end to the fine-tuned automatic driving model; S5. Encrypting and uploading the gradient information of this adaptive gradient update to the cloud, and aggregating the gradient information of each vehicle to update the initial automatic driving model. 2.The online adaptive method for automatic driving model according to claim 1, wherein, Step S1 is as follows: S11. Collecting data in various driving scenarios, defining corresponding automatic driving tasks for each driving scenario, and generating a meta-training dataset by taking each driving task and the corresponding support set and query set data as a meta-task; S12. Using a depth separable convolutional network as the backbone network, and performing channel pruning on the backbone network to generate a lightweight neural network as an automatic driving base model; S13. Performing meta-learning training, using a model-independent meta-learning algorithm to perform meta-learning training on the automatic driving base model on the meta-training dataset, and outputting a set of initial parameters sensitive to driving task changes to generate an initial automatic driving model after the meta-learning training is completed.

3. The online adaptive method for autonomous driving model according to claim 2, wherein, In step S13, the training process of the model-independent meta-learning algorithm includes inner loop update and outer loop update; Wherein, the inner loop update uses the corresponding support set data to perform gradient descent on the model parameters of the automatic driving base model for a preset number of steps for each meta-task to obtain adapted parameters for the corresponding meta-task; The outer loop update optimizes the initial parameters of the automatic driving base model based on the loss of all meta-tasks on the query set. 4.The online adaptive method for automatic driving model according to claim 1, characterized in that, Step S2 is as follows: S21. Deploying the initial automatic driving model on the vehicle-mounted computing platform and running it during vehicle travel to perform real-time environment perception or vehicle control tasks; S22. Calculating the confidence or entropy value of the perception result output by the initial automatic driving model as the perception confidence; S23. When the perception confidence of a continuous set number of time steps is lower than the first threshold, it is determined that the initial automatic driving model lacks understanding of the current driving scene, triggering an online adaptive process; S24. Perform an online adaptive process to make the vehicle enter a minimum risk state through takeover prompts or automatic control, while collecting time-synchronized data of cameras and lidar in a future set time period, and generating time-synchronized sensor data.

5. The online adaptive method for autonomous driving model according to claim 1, wherein, Step S3 includes the following specific steps: S31. Load the parameters of the initial automatic driving model on the automatic driving controller or vehicle-mounted computing platform at the vehicle end; S32. Define the collected time-synchronized sensor data as a new driving task, and use the time-synchronized sensor data as the support set of the defined driving task; S33. Perform an inner loop fine-tuning of meta-learning, starting from the parameters of the initial automatic driving model, and perform an inner loop update process of model-independent meta-learning for the defined driving task: wherein, is a parameter of an initial autonomous driving model, is an inner loop learning rate, is a loss function of a new driving task on a support set, is an updated parameter of the initial autonomous driving model; S34. After iterating the inner loop update process for a set number of times, stop updating, and use the finally obtained model parameters as the parameters of the fine-tuned automatic driving model.

6. The online adaptive method for autonomous driving model according to claim 1, wherein, Step S4 includes the following specific steps: S41. Deploy the fine-tuned automatic driving model in a sandbox environment isolated from the real vehicle control system at the vehicle end; S42. Use the collected time-synchronized sensor data segments of the new scene, and the data segments extracted from the pre-stored data at the vehicle end to represent known driving scenes as the validation data set; S43. In the sandbox environment, use the initial automatic driving model and the fine-tuned automatic driving model respectively to perform forward inference on the validation data set, and compare the performance indicators of the two models on the new scene segments and the known scene segments; the performance indicators include the average precision of target detection, the accuracy of lane line detection, or the trajectory tracking error of vehicle control instructions; S44. Determine whether the following two conditions are met: The performance indicators of the fine-tuned automatic driving model on the new scene segments are better than those of the initial automatic driving model; The performance indicators on the known scene segments degrade by less than the second threshold; When both conditions are met, trigger a switching instruction to smoothly replace the driving control model in the real vehicle control system with the fine-tuned automatic driving model.

7. The online adaptive method for autonomous driving model according to claim 1, wherein, Step S5 includes the following specific steps: S51. Package the gradient vector generated by performing gradient update in step S3 or the parameter change amount of the fine-tuned automatic driving model that passes the verification in step S4 as a knowledge package at the vehicle end; S52. Encrypt the knowledge package and upload it to the cloud server through the Internet of Vehicles; S53. The cloud server receives the encrypted knowledge packages of each vehicle and decrypts them to obtain model update information, aggregates the model update information using a federated average algorithm, and obtains global update information; S54. Use the global update information to optimize the parameters of the initial automatic driving model stored in the cloud to generate a new initial automatic driving model for subsequent deployment to the vehicle end.

8. An automatic driving model online self-adaption device, characterized in that, It includes: A cloud meta-training module for constructing a meta-training data set based on multi-scene driving data, and training a lightweight neural network using a model-independent meta-learning algorithm to obtain an initial automatic driving model sensitive to driving task changes; A vehicle end model deployment and monitoring module for deploying the initial automatic driving model to the vehicle end, monitoring the perception confidence of the initial automatic driving model in real time, and triggering an adaptive process when the perception confidence is lower than a first threshold; A safety data collection and control module, which executes an adaptive process to control the vehicle to enter a safety state and collect time-series sensor data of the current scene; An online meta-learning module at the vehicle end, which takes the parameters of the initial automatic driving model as a starting point and the time-series sensor data as a support set to perform gradient updates of a set number of steps at the vehicle end to generate a fine-tuned automatic driving model; A safety verification and switching module, which verifies in a sandbox environment whether the fine-tuned automatic driving model meets the following two conditions: The fine-tuned automatic driving model performs better than the initial automatic driving model in a new scene; The performance degradation of the fine-tuned automatic driving model in known scenes is lower than a second threshold; When both conditions are met, the driving control model at the vehicle end is switched to the fine-tuned automatic driving model; A knowledge back transmission and aggregation evolution module, which encrypts and uploads the gradient information of the gradient updates of this adaptation to the cloud, and aggregates the gradient information of each vehicle to update the initial automatic driving model.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the online adaptive method of the automatic driving model according to any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the online adaptive method of the automatic driving model according to any one of claims 1 to 7.

Images