Deep reinforcement learning automatic driving model optimization method and system, electronic equipment and readable medium

Abstract

The invention provides a deep reinforcement learning automatic driving model optimization method, and belongs to the field of intelligent driving. Constructing a simulation environment and training a basic IL model according to the long-tail scene; performing large-scale data collection according to the simulation environment, and performing model optimization on the basic IL model through a distributed RL algorithm; according to the optimized RL model, candidate tracks are generated, through an environment interactive learning mechanism of reinforcement learning, autonomous exploration and trial and error in a simulation environment are achieved, and dependence on real driving data is reduced; and in combination with a dynamic reward function, the model is guided to actively learn a coping strategy of a long-tail scene, and the generalization bottleneck of traditional data driving is broken through. In a closed-loop training environment combined with traffic simulation, model output can be corrected in real time through online strategy optimization, the adaptability of the system to dynamic changes is improved, and deviation caused by closed-loop control feedback is adapted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to a deep reinforcement learning automatic driving model optimization method and system, an electronic device and a computer readable medium. BACKGROUND

[0002] With the development of automatic driving technology, end-to-end architecture has gradually become mainstream. Traditional modular solutions separate the design of perception, planning, control and other links, resulting in cumulative errors and lack of flexibility and adaptability in complex scenarios48. End-to-end methods directly map sensor inputs (such as cameras, lidar) to driving decisions through deep learning models, reducing error propagation in intermediate links and improving dynamic environment adaptability.

[0003] However, existing end-to-end solutions rely heavily on imitation learning (IL) or supervised learning, which is based on training models with large amounts of human driving data to replicate expert behavior. Although this method can cover regular scenarios, it performs poorly in long-tail scenarios (such as sudden obstacles and extreme weather), and it is difficult to solve the error accumulation problem in closed-loop control. In addition, traditional end-to-end models are highly dependent on data and lack safety verification mechanisms, resulting in insufficient explainability and difficulty in meeting the needs of high safety level scenarios.

[0004] On the other hand, training an automatic driving model from scratch using reinforcement learning requires a large number of simulation scenarios, and the workload of map and scenario construction is staggering, with costs that cannot be ignored. SUMMARY

[0005] The present application aims to solve at least one of the technical problems in the prior art and proposes a deep reinforcement learning automatic driving model optimization method and system.

[0006] In a first aspect, a deep reinforcement learning automatic driving model optimization method is provided, which includes:

[0007] screening long-tail scenarios;

[0008] constructing a simulation environment and training a basic IL model according to the long-tail scenarios;

[0009] collecting large-scale data according to the simulation environment and optimizing the basic IL model through a distributed RL algorithm;

[0010] generating candidate trajectories according to the optimized RL model.

[0011] In some embodiments, in the step of screening long-tail scenarios, it includes:

[0012] Collect raw multi-modal driving data including sensor data and trajectory data, and clean and label the data;

[0013] Use a pre-trained IL model or RL model to predict the data segment, calculate the prediction entropy, and the data with high entropy value represents low model confidence, which is marked as a high-uncertainty sample and automatically filtered out as a candidate long-tail scene; or use self-supervised contrastive learning to extract a feature vector of the scene, and then cluster the feature vector using a clustering algorithm, and the sparsely distributed clusters in the clustering result are considered as long-tail scene categories;

[0014] Combine the filtered long-tail scene data with the regular scene data to form a balanced high-value training set for subsequent RL training.

[0015] In some embodiments, in the step of constructing a simulation environment according to the long-tail scene and training a basic IL model, the following steps are included:

[0016] Design a simulation environment based on the OpenAIGym library;

[0017] Use a neural network to encode multi-modal sensor data into a low-dimensional feature vector;

[0018] Use a BEVTransformer to generate a bird's eye view feature from traffic scene data, capturing the global geometric information of the scene;

[0019] Process the bird's eye view feature through a Transformer architecture, using a self-attention mechanism to capture the interaction between dynamic obstacles, map elements, and other traffic participants, and output an interpretable state vector table;

[0020] Integrate the state representation module into the simulation environment and train the basic IL model.

[0021] In some embodiments, in the step of collecting large-scale data according to the simulation environment and optimizing the basic IL model through a distributed RL algorithm, the following steps are included:

[0022] Add a KL divergence term to the standard PPO loss function, and the total loss function is where, is the behavior policy of the modified RL model, is the behavior policy output by the basic IL model, θ is the network parameter, and λ is the weighting coefficient;

[0023] Run multiple simulation environments in parallel to collect interaction data;

[0024] Buffer mechanism is used to cache the state-action-reward data generated by each environment, and batch data collection is realized;

[0025] The central learner samples data from Buffer and updates model parameters in batches;

[0026] The Learner regularly updates the policy network and value network, and synchronizes the new parameters to the Actor nodes in each environment.

[0027] In some embodiments, in the step of generating candidate trajectories according to the optimized RL model, the following steps are included:

[0028] A trajectory diffusion module is added to the decision layer of the RL model to generate a diffusion model;

[0029] The diffusion model outputs multiple candidate trajectories, each with a generation probability;

[0030] The K-Nearest Neighbor clustering algorithm is used to cluster the candidate trajectories, merge similar modal trajectories, and form a trajectory distribution;

[0031] In combination with the value function or policy network of the RL model, the optimal trajectory is selected from the clustered trajectories for execution.

[0032] In some embodiments, in the step of adding a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model, the following steps are included:

[0033] Gaussian noise is added to the initial trajectory x0 step by step to generate a noise trajectory x t , where the noise schedule follows , is a predefined noise schedule parameter, t is the label of the noise step number, q is the probability distribution of the t-step noise trajectory based on the t-1 step trajectory, N is the Gaussian distribution, and I is the identity matrix;

[0034] Based on the current state S t , the trajectory is recovered step by step through the denoising network, and the conditional probability is:

[0035] , where μ θ and Σ θ are the outputs of the denoising network, t is the label of the noise step number, q is the probability distribution of the t-1 step noise trajectory based on the t-step trajectory and the current state, N is the Gaussian distribution, and θ is the network parameter.

[0036] In some embodiments, the following steps are included:

[0037] The optimal trajectory is executed in the simulation environment to obtain environmental feedback rewards and state updates.

[0038] In a second aspect, the present application provides a deep reinforcement learning autonomous driving model optimization system, comprising:

[0039] a data collection unit configured to filter long-tail scenes;

[0040] a simulation unit configured to construct a simulation environment and train a basic IL model according to the long-tail scenes;

[0041] an optimization unit configured to collect large-scale data according to the simulation environment and optimize the basic IL model through a distributed RL algorithm;

[0042] a generation unit configured to generate a candidate trajectory according to the optimized RL model.

[0043] In a third aspect, the present application also provides an electronic device, comprising:

[0044] one or more processors;

[0045] a memory configured to store one or more programs;

[0046] when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods.

[0047] In a fourth aspect, the present application also provides a computer readable medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps in any of the methods.

[0048] The deep reinforcement learning autonomous driving model optimization method provided by the present application filters long-tail scenes, constructs a simulation environment and trains a basic IL model according to the long-tail scenes, collects large-scale data according to the simulation environment, and optimizes the basic IL model through a distributed RL algorithm, and generates a candidate trajectory according to the optimized RL model. Through the environment interactive learning mechanism of reinforcement learning, autonomous exploration and trial and error are performed in the simulation environment, and the dependence on real driving data is reduced. In combination with a dynamic reward function, the model is guided to actively learn the coping strategies for long-tail scenes, and the generalization bottleneck of traditional data-driven methods is broken. In a closed-loop training environment combined with traffic simulation, the model output can be corrected in real time through online policy optimization, the adaptability of the system to dynamic changes is improved, and the deviation caused by closed-loop control feedback is adapted. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a step schematic diagram of an embodiment of the deep reinforcement learning autonomous driving model optimization method of the present application;

[0050] Figure 2 is a principle schematic diagram of an embodiment of the deep reinforcement learning autonomous driving model optimization method of the present application;

[0051] Figure 3 This is a schematic diagram of an embodiment of the deep reinforcement learning autonomous driving model optimization system of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0054] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0055] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0057] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0058] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0059] In related technologies, existing end-to-end solutions mostly rely on imitation learning or supervised learning, the core of which is to train models using a large amount of human driving data to reproduce expert behavior. While such methods can cover conventional scenarios, they perform poorly in long-tail scenarios (such as sudden obstacles and extreme weather) and struggle to solve the error accumulation problem in closed-loop control. Furthermore, traditional end-to-end models are highly dependent on data and lack safety verification mechanisms, resulting in insufficient interpretability and making it difficult to meet the requirements of high-safety-level scenarios.

[0060] To address at least one of the technical problems existing in the aforementioned related technologies, this invention provides a deep reinforcement learning-based autonomous driving model optimization method.

[0061] Figure 1 and Figure 2 The present invention provides a flowchart and schematic diagram of a deep reinforcement learning-based autonomous driving model optimization method, comprising the following steps:

[0062] Step S10: Filter long-tail scenarios.

[0063] Specifically, this embodiment automatically identifies and extracts long-tail scenarios (such as extreme weather, sudden obstacles, and complex interactive behaviors) from massive driving data to construct a high-value training set, ensuring that the model can learn rare but critical scenarios.

[0064] The specific implementation steps are as follows:

[0065] Step 1.1: Data preprocessing.

[0066] Specifically, raw multimodal driving data (such as sensor data, trajectory data, etc.) is collected, cleaned, and labeled.

[0067] Step 1.2: Active learning and screening.

[0068] Uncertainty-based sampling: Predictions are made on data segments using a pre-trained base IL or RL model, and the prediction entropy is calculated. Data with high entropy values ​​indicate low model confidence, and these data are labeled as samples with high uncertainty and automatically selected as candidate long-tail scenarios.

[0069] Scene clustering analysis: Self-supervised contrastive learning (such as SimCLR) is used to extract feature vectors of scenes, and then clustering algorithms (such as K-means or DBSCAN) are used to cluster these feature vectors. In the clustering results, sparsely distributed clusters are regarded as long-tail scene categories, thereby identifying rare scenes.

[0070] Step 1.3: Construct the training set.

[0071] Specifically, the selected long-tail scenario data is combined with regular scenario data to form a balanced high-value training set for subsequent RL training.

[0072] It is understandable that this embodiment, through active learning and screening, ensures that the training set contains diverse and challenging scenarios, providing high-quality input for subsequent RL model training and avoiding overfitting of the model in common scenarios. The selected long-tail scenarios are directly used for simulation environment construction and model training, improving the generalization ability of the overall system.

[0073] Step S20: Construct a simulation environment and train a basic IL model based on the long-tail scenario.

[0074] It is understood that this embodiment constructs a high-fidelity, scalable virtual-real fusion simulation environment, supports closed-loop training and verification of autonomous driving models, and encodes multimodal sensor data into low-dimensional state vectors.

[0075] The technical implementation steps are as follows:

[0076] Step 2.1: Design a simulation environment based on the OpenAIGym library.

[0077] Specifically, a simulation environment is designed based on the OpenAIGym library, integrating a physics engine (such as CARLA or SUMO) to support vehicle dynamics modeling, environment interaction, and reward calculation. The environment allows parallel instance execution to support distributed training.

[0078] Step 2.2: Encode the multimodal sensor data into low-dimensional feature vectors using a neural network.

[0079] Specifically, neural networks (such as CNN or Transformer) are used to encode multimodal sensor data from cameras, LiDAR, and other sources into low-dimensional feature vectors.

[0080] Step 2.2: Use BEVTransformer to generate bird's-eye view (BEV) features from traffic scene data to capture global geometric information of the scene.

[0081] Step 2.3: Process BEV features using the Transformer architecture, and use a self-attention mechanism to capture the interaction relationships between dynamic obstacles, map elements and other traffic participants, and output an interpretable state vector.

[0082] It is understood that this embodiment uses BEVFormer to generate bird's-eye view features from traffic scene data, and combines spatiotemporal attention with the Transformer architecture to capture the interaction between dynamic obstacles, map information and other obstacles.

[0083] Step 2.4: Integrate the state representation module into the simulation environment and train the basic IL model.

[0084] Specifically, the state representation module is integrated into the simulation environment to achieve real-time state updates and reward feedback, forming a closed-loop training process.

[0085] State representation, as we understand it, transforms raw data into states that the RL model can understand, enabling the model to perceive the context and interactions of complex scenarios. This step relies on long-tail scene data to construct realistic simulation scenarios and provides an interaction interface for distributed RL algorithms. The quality of state representation directly affects the convergence and performance of the RL model.

[0086] Step S30: Collect large-scale data based on the simulation environment, and optimize the basic IL model using a distributed RL algorithm.

[0087] It is understood that this embodiment designs an efficient distributed RL algorithm based on the PPO (Proximal Policy Optimization) optimization strategy and uses the basic IL model for constraints to prevent the model from deviating from reasonable behavior.

[0088] The technical implementation steps are as follows:

[0089] Step 3.1: Add the KL divergence term from the output of the base IL model to the standard PPO loss function. The total loss function is: ,in, This is the behavior strategy of the modified RL model. θ represents the behavioral strategy output by the basic IL model, λ represents the network parameters, and λ represents the tradeoff coefficients. The KL divergence term ensures that the RL model does not deviate significantly from the reasonable behavior of the basic IL model during optimization, avoiding getting trapped in local optima. The PPO algorithm is used as the core of RL, and its advantages lie in stability and sample efficiency.

[0090] Step 3.2: Run in parallel across multiple simulation environments to collect interactive data.

[0091] Specifically, multiple simulation environment instances are deployed and run in parallel to collect interactive data.

[0092] Step 3.3: Use a buffer mechanism to cache the state-action-reward data generated by each environment, to achieve...

[0093] Batch data collection.

[0094] Step 3.4: The central learner samples data from the buffer and updates the model parameters in batches.

[0095] It is understandable that optimizing parallel efficiency and accelerating computation can be achieved through distributed computing frameworks (such as Ray or Horovod).

[0096] train.

[0097] Step 3.4: The Learner periodically updates the policy network and value network, and synchronizes the new parameters to each ring.

[0098] Actor nodes in the environment.

[0099] It is understandable that the distributed architecture utilizes a simulation environment for large-scale data collection, improving training efficiency. The KL divergence term in the loss function uses the base IL model (trained from the collected data) as a priori to constrain the behavior of the RL model, ensuring safety and rationality. The trained RL model will be used for trajectory generation in step 4, forming a policy optimization closed loop.

[0100] Step S40: Generate candidate trajectories based on the optimized RL model.

[0101] Specifically, by leveraging the probabilistic modeling capabilities of the diffusion model, high-coverage and physically feasible candidate trajectories are generated, enhancing the model's decision robustness in long-tail scenarios.

[0102] The specific implementation steps are as follows:

[0103] Technical implementation steps:

[0104] Step 4.1: Add a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model.

[0105] It is understandable to add a trajectory diffusion module to the decision layer of the RL model as a trajectory generator.

[0106] Step 4.2: The diffusion model outputs multiple candidate trajectories, each with a generation probability.

[0107] Specifically, the step of adding a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model also includes:

[0108] Gaussian noise is gradually added to the initial trajectory x0 to generate a noisy trajectory x.t Among them: noise scheduling follows , Here are the predefined noise scheduling parameters, t is the label of the number of noise addition steps, q is the probability distribution of the noise trajectory at step t based on the trajectory at step t-1, N is a Gaussian distribution, and I is the identity matrix.

[0109] Based on the current state S t The trajectory is gradually recovered through a denoising network, with the conditional probability being:

[0110] , where μ θ and Σ θ is the output of the denoising network, t is the label of the number of noise addition steps, q is the probability distribution of the noise trajectory of step t-1 based on the trajectory of step t and the current state, N is a Gaussian distribution, and θ is the network parameter.

[0111] Step 4.3: Use the K-nearest neighbor clustering algorithm to cluster the candidate trajectories, merge trajectories with similar modes, and form a trajectory distribution.

[0112] Step 4.4: Combine the value function or policy network of the RL model to select the optimal trajectory from the clustered trajectories for execution.

[0113] It is understood that the diffusion model in this embodiment utilizes its probabilistic generation capabilities to provide diverse trajectory candidates for RL strategies, compensating for the shortcomings of RL when exploration is insufficient, especially in long-tail scenarios. Trajectory generation relies on the state representations of the above steps as conditions, and ensures the feasibility and optimality of the trajectory through clustering and RL selection.

[0114] Furthermore, the deep reinforcement learning-based autonomous driving model optimization method also includes the following steps:

[0115] The optimal trajectory is executed in the simulation environment for environmental feedback rewards and state updates.

[0116] It is understandable that the selected trajectory is executed in the simulation environment, the environment provides feedback rewards and state updates, and the data is stored in the RolloutBuffer for subsequent model updates, forming a closed loop of training data to further optimize the RL model.

[0117] This invention provides a deep reinforcement learning-based autonomous driving model optimization method that filters long-tail scenarios; constructs a simulation environment and trains a basic IL model based on the long-tail scenarios; collects large-scale data based on the simulation environment and optimizes the basic IL model using a distributed RL algorithm; and generates candidate trajectories based on the optimized RL model. This invention's technical solution utilizes an interactive learning mechanism within the reinforcement learning environment to autonomously explore and experiment in the simulation environment, reducing reliance on real driving data. Combined with a dynamic reward function, it guides the model to actively learn coping strategies for long-tail scenarios (such as emergency obstacle avoidance and adaptation to abnormal weather), breaking through the generalization bottleneck of traditional data-driven approaches. In a closed-loop training environment combined with traffic simulation, this solution can improve the system's adaptability to dynamic changes and compensate for deviations caused by closed-loop control feedback by optimizing the model output in real time through online strategy optimization.

[0118] Furthermore, based on a multi-objective reinforcement learning framework, perception fusion and subsequent prediction, decision-making, and planning are decoupled, and independent reward functions are designed for prediction, planning, and control tasks. A dynamic weight allocation mechanism coordinates the optimization direction of multiple tasks (e.g., focusing on trajectory smoothness during normal driving and prioritizing response speed in emergency scenarios), thereby achieving task collaboration and efficient resource allocation.

[0119] Please see Figure 3 The present invention also provides a deep reinforcement learning autonomous driving model optimization system. Applied to the deep reinforcement learning autonomous driving model optimization method provided in the above embodiments, it specifically includes:

[0120] The data acquisition unit is used to filter long-tail scenarios.

[0121] Specifically, this embodiment automatically identifies and extracts long-tail scenarios (such as extreme weather, sudden obstacles, and complex interactive behaviors) from massive driving data to construct a high-value training set, ensuring that the model can learn rare but critical scenarios.

[0122] The specific implementation steps are as follows:

[0123] Step 1.1: Data preprocessing.

[0124] Specifically, raw multimodal driving data (such as sensor data, trajectory data, etc.) is collected, cleaned, and labeled.

[0125] Step 1.2: Active learning and screening.

[0126] Uncertainty-based sampling: Predictions are made on data segments using a pre-trained base IL or RL model, and the prediction entropy is calculated. Data with high entropy values ​​indicate low model confidence, and these data are labeled as samples with high uncertainty and automatically selected as candidate long-tail scenarios.

[0127] Scene clustering analysis: Self-supervised contrastive learning (such as SimCLR) is used to extract feature vectors of scenes, and then clustering algorithms (such as K-means or DBSCAN) are used to cluster these feature vectors. In the clustering results, sparsely distributed clusters are regarded as long-tail scene categories, thereby identifying rare scenes.

[0128] Step 1.3: Construct the training set.

[0129] Specifically, the selected long-tail scenario data is combined with regular scenario data to form a balanced high-value training set for subsequent RL training.

[0130] It is understandable that this embodiment, through active learning and screening, ensures that the training set contains diverse and challenging scenarios, providing high-quality input for subsequent RL model training and avoiding overfitting of the model in common scenarios. The selected long-tail scenarios are directly used for simulation environment construction and model training, improving the generalization ability of the overall system.

[0131] The simulation unit is used to construct a simulation environment and train a basic IL model based on the long-tail scenario.

[0132] It is understood that this embodiment constructs a high-fidelity, scalable virtual-real fusion simulation environment, supports closed-loop training and verification of autonomous driving models, and encodes multimodal sensor data into low-dimensional state vectors.

[0133] The technical implementation steps are as follows:

[0134] Step 2.1: Design a simulation environment based on the OpenAIGym library.

[0135] Specifically, a simulation environment is designed based on the OpenAIGym library, integrating a physics engine (such as CARLA or SUMO) to support vehicle dynamics modeling, environment interaction, and reward calculation. The environment allows parallel instance execution to support distributed training.

[0136] Step 2.2: Encode the multimodal sensor data into low-dimensional feature vectors using a neural network.

[0137] Specifically, neural networks (such as CNN or Transformer) are used to encode multimodal sensor data from cameras, LiDAR, and other sources into low-dimensional feature vectors.

[0138] Step 2.2: Use BEVTransformer to generate bird's-eye view (BEV) features from traffic scene data to capture global geometric information of the scene.

[0139] Step 2.3: Process BEV features using the Transformer architecture, and use a self-attention mechanism to capture the interaction relationships between dynamic obstacles, map elements and other traffic participants, and output an interpretable state vector.

[0140] It is understood that this embodiment uses BEVFormer to generate bird's-eye view features from traffic scene data, and combines spatiotemporal attention with the Transformer architecture to capture the interaction between dynamic obstacles, map information and other obstacles.

[0141] Step 2.4: Integrate the state representation module into the simulation environment and train the basic IL model.

[0142] Specifically, the state representation module is integrated into the simulation environment to achieve real-time state updates and reward feedback, forming a closed-loop training process.

[0143] State representation, as we understand it, transforms raw data into states that the RL model can understand, enabling the model to perceive the context and interactions of complex scenarios. This step relies on long-tail scene data to construct realistic simulation scenarios and provides an interaction interface for distributed RL algorithms. The quality of state representation directly affects the convergence and performance of the RL model.

[0144] The optimization unit is used to collect large-scale data based on the simulation environment and optimize the basic IL model using a distributed RL algorithm.

[0145] It is understood that this embodiment designs an efficient distributed RL algorithm based on the PPO (Proximal Policy Optimization) optimization strategy and uses the basic IL model for constraints to prevent the model from deviating from reasonable behavior.

[0146] The technical implementation steps are as follows:

[0147] Step 3.1: Add the KL divergence term from the output of the base IL model to the standard PPO loss function. The total loss function is: ,in, This is the behavior strategy of the modified RL model. θ represents the behavioral strategy output by the basic IL model, λ represents the network parameters, and λ represents the tradeoff coefficients. The KL divergence term ensures that the RL model does not deviate significantly from the reasonable behavior of the basic IL model during optimization, avoiding getting trapped in local optima. The PPO algorithm is used as the core of RL, and its advantages lie in stability and sample efficiency.

[0148] Step 3.2: Run in parallel across multiple simulation environments to collect interactive data.

[0149] Specifically, multiple simulation environment instances are deployed and run in parallel to collect interactive data.

[0150] Step 3.3: Use a buffer mechanism to cache the state-action-reward data generated by each environment, to achieve...

[0151] Batch data collection.

[0152] Step 3.4: The central learner samples data from the buffer and updates the model parameters in batches.

[0153] It is understandable that optimizing parallel efficiency and accelerating computation can be achieved through distributed computing frameworks (such as Ray or Horovod).

[0154] train.

[0155] Step 3.4: The Learner periodically updates the policy network and value network, and synchronizes the new parameters to each ring.

[0156] Actor nodes in the environment.

[0157] It is understandable that the distributed architecture utilizes a simulation environment for large-scale data collection, improving training efficiency. The KL divergence term in the loss function uses the base IL model (trained from the collected data) as a priori to constrain the behavior of the RL model, ensuring safety and rationality. The trained RL model will be used for trajectory generation in step 4, forming a policy optimization closed loop.

[0158] The generation unit is used to generate candidate trajectories based on the optimized RL model.

[0159] Specifically, by leveraging the probabilistic modeling capabilities of the diffusion model, high-coverage and physically feasible candidate trajectories are generated, enhancing the model's decision robustness in long-tail scenarios.

[0160] The specific implementation steps are as follows:

[0161] Technical implementation steps:

[0162] Step 4.1: Add a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model.

[0163] It is understandable to add a trajectory diffusion module to the decision layer of the RL model as a trajectory generator.

[0164] Step 4.2: The diffusion model outputs multiple candidate trajectories, each with a generation probability.

[0165] Specifically, the step of adding a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model also includes:

[0166] Gaussian noise is gradually added to the initial trajectory x0 to generate a noisy trajectory x.t Among them: noise scheduling follows , Here are the predefined noise scheduling parameters, t is the label of the number of noise addition steps, q is the probability distribution of the noise trajectory at step t based on the trajectory at step t-1, N is a Gaussian distribution, and I is the identity matrix.

[0167] Based on the current state S t The trajectory is gradually recovered through a denoising network, with the conditional probability being:

[0168] , where μ θ and Σ θ is the output of the denoising network, t is the label of the number of noise addition steps, q is the probability distribution of the noise trajectory of step t-1 based on the trajectory of step t and the current state, N is a Gaussian distribution, and θ is the network parameter.

[0169] Step 4.3: Use the K-nearest neighbor clustering algorithm to cluster the candidate trajectories, merge trajectories with similar modes, and form a trajectory distribution.

[0170] Step 4.4: Combine the value function or policy network of the RL model to select the optimal trajectory from the clustered trajectories for execution.

[0171] It is understood that the diffusion model in this embodiment utilizes its probabilistic generation capabilities to provide diverse trajectory candidates for RL strategies, compensating for the shortcomings of RL when exploration is insufficient, especially in long-tail scenarios. Trajectory generation relies on the state representations of the above steps as conditions, and ensures the feasibility and optimality of the trajectory through clustering and RL selection.

[0172] Furthermore, the deep reinforcement learning-based autonomous driving model optimization system also includes:

[0173] An iterative unit is used to execute the optimal trajectory in the simulation environment to provide environmental feedback rewards and state updates.

[0174] It is understandable that the selected trajectory is executed in the simulation environment, the environment provides feedback rewards and state updates, and the data is stored in the RolloutBuffer for subsequent model updates, forming a closed loop of training data to further optimize the RL model.

[0175] This invention provides a deep reinforcement learning-based autonomous driving model optimization system that filters long-tail scenarios; constructs a simulation environment and trains a basic IL model based on the long-tail scenarios; collects large-scale data based on the simulation environment and optimizes the basic IL model using a distributed RL algorithm; and generates candidate trajectories based on the optimized RL model. This invention's technical solution utilizes an interactive learning mechanism within the reinforcement learning environment to autonomously explore and experiment in the simulation environment, reducing reliance on real driving data. Combined with a dynamic reward function, it guides the model to actively learn coping strategies for long-tail scenarios (such as emergency obstacle avoidance and adaptation to abnormal weather), overcoming the generalization bottleneck of traditional data-driven approaches. In a closed-loop training environment combined with traffic simulation, this solution can real-time correct the model output through online strategy optimization, improving the system's adaptability to dynamic changes and mitigating deviations caused by closed-loop control feedback.

[0176] Furthermore, based on a multi-objective reinforcement learning framework, perception fusion and subsequent prediction, decision-making, and planning are decoupled, and independent reward functions are designed for prediction, planning, and control tasks. A dynamic weight allocation mechanism coordinates the optimization direction of multiple tasks (e.g., focusing on trajectory smoothness during normal driving and prioritizing response speed in emergency scenarios), thereby achieving task collaboration and efficient resource allocation.

[0177] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the deep reinforcement learning autonomous driving model optimization methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0178] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0179] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0180] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0181] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the deep reinforcement learning autonomous driving model optimization methods described in the above embodiments. The computer-readable storage medium can be volatile or non-volatile.

[0182] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described deep reinforcement learning autonomous driving model optimization method.

[0183] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0184] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0185] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0186] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0187] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0188] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0189] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0190] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0192] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A deep reinforcement learning autonomous driving model optimization method, characterized in that, It comprises: Screening long tail scenes; Building a simulation environment and training a basic IL model according to the long tail scenes; Collecting large-scale data according to the simulation environment, and optimizing the basic IL model through a distributed RL algorithm; Generating candidate trajectories according to the optimized RL model. 2.The deep reinforcement learning autonomous driving model optimization method of claim 1, wherein, In the step of screening long tail scenes, it comprises: Collecting original multi-modal driving data, and cleaning and labeling, the multi-modal driving data comprising sensor data and trajectory data; Using a pre-trained basic IL model or RL model to predict the data segment, calculating the prediction entropy, and the data with high entropy value representing low model confidence, which is marked as high uncertainty sample and automatically screened out as candidate long tail scenes; or using self-supervised contrast learning to extract feature vectors of the scenes, and then clustering the feature vectors through clustering algorithm, and the sparse clusters in the clustering result are regarded as long tail scene categories; Combining the screened long tail scene data with the conventional scene data to form a balanced high-value training set for subsequent RL training. 3.The deep reinforcement learning autonomous driving model optimization method of claim 1, wherein, In the step of building a simulation environment and training a basic IL model according to the long tail scenes, it comprises: Designing a simulation environment based on OpenAIGym library; Using a neural network to encode multi-modal sensor data into a low-dimensional feature vector; Using BEVTransformer to generate bird's eye view features from traffic scene data to capture global geometric information of the scene; Processing the bird's eye view features through a Transformer architecture, using a self-attention mechanism to capture the interaction between dynamic obstacles, map elements and other road users, and outputting an interpretable state vector table; Integrating the state representation module into the simulation environment to train the basic IL model. 4.The method of claim 1, wherein, In the step of collecting large-scale data according to the simulation environment, and optimizing the basic IL model through a distributed RL algorithm, it comprises: In the standard PPO loss function, a KL divergence term with the output of the base IL model is added, and the total loss function is where, is the behavior policy of the modified RL model, is the behavior policy output by the base IL model, θ is the network parameter, and λ is the weighting coefficient. Parallelly running multiple simulation environments to collect interaction data; Using a buffer mechanism to cache state-action-reward data generated by each environment to realize batch data collection; The central learner samples data from the Buffer and updates model parameters in batches; The Learner regularly updates the policy network and value network, and synchronizes the new parameters to the Actor nodes in each environment. 5.The deep reinforcement learning autonomous driving model optimization method of claim 1, wherein, In the step of generating candidate trajectories according to the optimized RL model, it comprises: Adding a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model; The diffusion model outputs multiple candidate trajectories, each with a generation probability; Using K-Nearest Neighbor clustering algorithm to cluster the candidate trajectories, merging similar modal trajectories to form a trajectory distribution; Combining the value function or policy network of the RL model to select the optimal trajectory from the clustered trajectories for execution. 6.The method of claim 5, wherein, In the step of adding a trajectory diffusion module to the decision layer of the RL model to generate a diffusion model, it further comprises: Gaussian noise is added step by step to the initial trajectory x0 to generate a noisy trajectory x t wherein: the noise schedule follows , is a predefined noise schedule parameter, t is a label of the number of noise adding steps, q is a probability distribution based on which the noisy trajectory at step t is generated from the trajectory at step t-1, N is a Gaussian distribution, and I is an identity matrix. Based on the current state S t , the trajectory is gradually recovered by the denoising network, and the conditional probability is: where μ θ and Σ θ are the output of the denoising network, t is the label of the number of noisy steps, q is the probability distribution of the noise trajectory at step t-1 based on the trajectory at step t and the current state, N is a Gaussian distribution, and θ are the network parameters.

7. The deep reinforcement learning autonomous driving model optimization method of claim 5, wherein, Further comprising the following steps: The optimal trajectory is executed in the simulation environment to perform environment feedback reward and state update. 8.A deep reinforcement learning autonomous driving model optimization system, characterized in that, It comprises: A data acquisition unit for screening long tail scenes; A simulation unit for building a simulation environment and training a basic IL model according to the long tail scenes; An optimization unit is configured to perform large-scale data collection according to the simulation environment, and perform model optimization on the basic IL model through a distributed RL algorithm. A generation unit is configured to generate a candidate trajectory according to the optimized RL model.

9. An electronic device, comprising: The method comprises the following steps: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 7.

10. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method as claimed in any one of claims 1 to 7.

Images