Intelligent driving skill training method and system based on behavior cloning and reinforcement learning

Abstract

The present invention provides an intelligent driving skill training method and system based on behavior cloning and reinforcement learning, which relates to the technical field, including collecting driving behavior data of human drivers, and dividing a training set and a validation set to train a behavior cloning model to obtain an initial driving model. According to the performance of the initial model, driving skill indicators such as safety, efficiency and comfort to be optimized are determined, and diversified training scenarios are constructed in a virtual training environment. Using a reward function containing the above indicators, the initial model is reinforced through a deep reinforcement learning algorithm to obtain an optimized driving model. The optimized model is tested in a test scene set, and when the safety, efficiency and comfort indicators all reach a preset threshold, the training is completed. By combining behavior cloning and reinforcement learning, the present invention can effectively improve the comprehensive performance of the intelligent driving model in terms of safety, efficiency and comfort.

Description

Technical Field

[0001] The present invention relates to intelligent driving technology, and in particular to an intelligent driving skill training method and system based on behavior cloning and reinforcement learning. Background Art

[0002] Intelligent driving technology aims to give vehicles the ability to autonomously perceive the environment, make decisions and control vehicle driving, ultimately achieving the goal of unmanned driving. In recent years, with the rapid development of artificial intelligence technology, especially the rise of deep learning, intelligent driving technology has made significant progress. At present, the mainstream intelligent driving technologies mainly include rule-based methods, behavior cloning-based methods and reinforcement learning-based methods.

[0003] The rule-based approach relies on pre-defined rules to control vehicle behavior, requiring a large number of rules to be manually formulated to cover various driving scenarios, and is difficult to adapt to complex and dynamically changing traffic environments. In addition, the rule-based approach is difficult to achieve highly autonomous driving and lacks the ability to cope with scenarios that are not pre-defined.

[0004] The behavior cloning-based method trains the driving model by imitating the behavior of human drivers. It can use a large amount of human driving data for learning, thereby quickly obtaining the initial driving strategy. However, this method is limited by the coverage of the training data and is difficult to generalize to unseen scenarios. In addition, pure imitation learning is difficult to surpass the level of human drivers and is easily affected by data bias.

[0005] Reinforcement learning-based methods learn the optimal driving strategy through trial and error interaction with the environment, and have strong autonomous learning and generalization capabilities. However, reinforcement learning algorithms usually require a large amount of training data and computing resources, and the training process is time-consuming and difficult to converge. In addition, there are safety risks in conducting reinforcement learning training in a real traffic environment, so it is usually necessary to train in a virtual environment. However, there are differences between the virtual environment and the real environment, which makes it difficult to directly apply the trained model to actual driving. Summary of the invention

[0006] The embodiments of the present invention provide an intelligent driving skill training method and system based on behavior cloning and reinforcement learning, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides intelligent driving skills training methods based on behavior cloning and reinforcement learning, including:

[0009] Collecting driving behavior data of a human driver in a preset driving scenario, the driving behavior data including environmental perception information and control instruction information; dividing the driving behavior data into a training data subset and a verification data subset; using the training data subset to train a behavior cloning model, using the environmental perception information as an input feature and the control instruction information as an output label, using a supervised learning method to obtain an initial driving model, and using the verification data subset to evaluate the performance of the initial driving model to obtain an initial performance evaluation result;

[0010] Determine the driving skill index to be optimized according to the initial performance evaluation result, wherein the driving skill index includes a safety index, an efficiency index and a comfort index; construct a virtual training environment, and set a variety of training scenarios corresponding to the driving skill index in the virtual training environment; design a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index and the comfort index; use the initial driving model as a policy network of a reinforcement learning agent, train it using a deep reinforcement learning algorithm in the virtual training environment, and optimize the parameters of the policy network based on the feedback of the reward function to obtain an optimized driving model;

[0011] A test scenario set is constructed in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; the optimized driving model is deployed in the test scenario set for testing to obtain test performance data; when the comparison results of the safety index, efficiency index and comfort index in the test performance data with the preset test thresholds all meet the preset results, it is confirmed that the training of the optimized driving model is completed; when any indicator in the test performance data is less than the preset test threshold, the test performance data is added to the training data set, and reinforcement learning training based on the deep reinforcement learning algorithm is continued until the test performance data meets the preset conditions.

[0012] The driving behavior data is divided into a training data subset and a verification data subset; the behavior cloning model is trained using the training data subset, the environmental perception information is used as an input feature, the control instruction information is used as an output label, an initial driving model is obtained using a supervised learning method, and the performance of the initial driving model is evaluated using the verification data subset, and the initial performance evaluation results obtained include:

[0013] The driving behavior data is stratified according to scene types, wherein the scene types include straight-line driving scenes, curve turning scenes, lane change scenes, vehicle following scenes, and intersection passing scenes, and random sampling is performed in a ratio of eight to two in each of the scene types to obtain a training data subset and a validation data subset;

[0014] Performing feature extraction preprocessing on the environmental perception information in the training data subset to obtain input features, wherein the feature extraction preprocessing includes: converting the laser radar point cloud data into a three-dimensional grid feature based on the vehicle body coordinate system after downsampling through voxel filtering, converting the forward visual image into a feature vector through key area extraction and spatial pyramid pooling, and normalizing the vehicle state information to a range of negative one to positive one to obtain normalized features; performing normalization processing on the control instruction information in the training data subset to obtain an output label;

[0015] An initial driving model is constructed based on a supervised learning method, wherein the initial driving model includes a feature extraction module, a time series modeling module and a control output module. The feature extraction module uses a three-dimensional convolutional neural network to process the three-dimensional grid features, uses a two-dimensional convolutional neural network to process the feature vectors, and uses a fully connected layer to process the normalized features, and fuses the processed features to obtain fused features; the time series modeling module uses a bidirectional long short-term memory network to process the fused features, and obtains time series features based on the time series information of five seconds before and after; the control output module processes the time series features through a multi-head attention mechanism, and maps the time series features to the corresponding output labels;

[0016] The input features are used as inputs, the output labels are used as training targets, the initial driving model is trained using a multi-task learning framework, different loss weights are set for the steering control task, the longitudinal control task and the behavior prediction task, the steering control task uses a smoothed one-norm loss function, the longitudinal control task uses a mean square error loss function, and the behavior prediction task uses a cross entropy loss function; a two-norm regularization constraint and a random inactivation mechanism are introduced during the training process, and an adaptive moment estimation optimizer is used to optimize the parameters of the initial driving model;

[0017] The verification data subset is input into the trained initial driving model for performance evaluation, and a control accuracy index, a timing consistency index and a behavior similarity index are calculated, wherein the control accuracy index includes a steering angle error mean and an acceleration error root mean square, the timing consistency index includes a control smoothness and an action continuity score, and the behavior similarity index includes a trajectory matching degree and a driving style consistency score, to obtain an initial performance evaluation result.

[0018] Constructing a virtual training environment, setting a variety of training scenarios corresponding to the driving skill index in the virtual training environment; designing a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index, and the comfort index, including:

[0019] The driving skill index includes a safety index, an efficiency index and a comfort index. The safety index includes the minimum headway and lane departure distance. The efficiency index includes the average driving speed, the travel time and the task completion time. The comfort index includes the lateral acceleration change rate, the longitudinal acceleration change rate and the steering angle change rate.

[0020] A virtual training environment is constructed based on the driving skill index, and in the virtual training environment, an emergency avoidance scenario, an emergency braking scenario, and a vehicle cut-in scenario are set for the safety index; a multi-lane driving scenario, a roundabout passing scenario, and a traffic light intersection scenario are set for the efficiency index; and a lane keeping scenario, a vehicle following scenario, and a lane changing scenario are set for the comfort index;

[0021] Collecting sensor data in the virtual training environment, the sensor data including laser radar data, camera data and millimeter wave radar data, and calculating the value of the driving skill indicator in real time based on the sensor data;

[0022] A hierarchical reward function is designed according to the driving skill index, wherein the hierarchical reward function uses an exponential penalty function to calculate the deviation between the minimum time distance and the safety threshold to obtain a safety score for the safety index, converts the deviation between the average driving speed and the target speed and the travel time into an efficiency score for the efficiency index, and uses a quadratic penalty function to calculate the lateral acceleration change rate, the longitudinal acceleration change rate and the steering angle change rate to obtain a comfort score for the comfort index;

[0023] Corresponding adaptive weight coefficients are set for the safety score, the efficiency score and the comfort score, the sum of the adaptive weight coefficients is one, and the safety score, the efficiency score and the comfort score are respectively multiplied by the corresponding adaptive weight coefficients and the sum is obtained to obtain a total score.

[0024] Constructing a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploying the optimized driving model to the test scenario set for testing to obtain test performance data; when the safety index, efficiency index, and comfort index in the test performance data are all greater than a preset test threshold, confirming that the training of the optimized driving model is completed includes:

[0025] Constructing a test scenario set in the virtual training environment, the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios, and the scenarios that are not involved in the training are constructed by changing the road curvature parameters, lane width parameters, road friction coefficient parameters, traffic flow density parameters and weather environment parameters in the diversified training scenarios; deploying the optimized driving model to the test scenario set for testing, and repeatedly executing the test on each test scenario using the Monte Carlo method to obtain test performance data;

[0026] Calculating the safety index, recording the minimum headway of the safety index being less than two seconds as a dangerous event, counting the frequency of occurrence of the dangerous event during the test, and obtaining a maximum value of the lane departure distance in the safety index based on the frequency of occurrence;

[0027] Calculate the theoretical shortest travel time of each test path, and use the ratio of the task completion time in the efficiency index to the theoretical shortest travel time as the task completion time ratio;

[0028] Calculating the comfort index, performing statistical distribution characteristic analysis on the longitudinal acceleration change rate and the lateral acceleration change rate in the comfort index, and calculating the ninety-fifth percentile values ​​of the longitudinal acceleration change rate and the lateral acceleration change rate;

[0029] The preset test thresholds include a dangerous event frequency threshold, a lane departure distance threshold, an average driving speed threshold, a task completion time ratio threshold, a longitudinal acceleration change rate threshold, and a lateral acceleration change rate threshold;

[0030] When the safety index, the efficiency index and the comfort index all meet the judgment conditions in fifty consecutive tests, it is confirmed that the training of the optimized driving model is completed.

[0031] When any indicator in the test performance data is less than the preset test threshold, the test performance data is added to the training data set, and reinforcement learning training is continued based on the deep reinforcement learning algorithm until the test performance data meets the preset conditions including:

[0032] When any indicator of any level in the test performance data is less than the corresponding preset test threshold, extract the input data, intermediate features and output results of the level to which the performance non-compliant indicator belongs, and construct a scene data sequence containing time series association; perform spatiotemporal dimension reduction and feature extraction on the scene data sequence to generate a scene feature vector;

[0033] Based on the scene feature vector, similarity calculation is performed on different scenes, and scenes with similarity higher than a preset similarity threshold are clustered and stored to form scene group data; a hierarchical adaptive reward function is constructed, wherein the hierarchical adaptive reward function includes a perception layer reward function, a decision layer reward function, and a control layer reward function, and the weight of each layer reward function is dynamically adjusted based on the difference between the test performance data and the preset test threshold;

[0034] Construct a multi-level experience replay buffer pool, wherein the multi-level experience replay buffer pool includes a perception layer buffer pool, a decision layer buffer pool, and a control layer buffer pool, and assign intra-layer priority and inter-layer priority to training samples according to the difference between the performance index of each layer and the corresponding threshold;

[0035] The scene group data are stored in the corresponding multi-level experience replay buffer pool according to the level to which the performance non-compliance indicator belongs; the performance improvement degree of each scene group in the multi-level experience replay buffer pool is regularly evaluated, and when the test performance data corresponding to a scene group is higher than the preset test threshold for multiple consecutive times, the scene group is removed from the corresponding buffer pool;

[0036] Sampling training samples from the multi-level experience replay buffer pool based on the intra-layer priority and the inter-layer priority; adopting a curriculum learning strategy to preferentially train network layers with performance gaps, wherein the curriculum learning strategy dynamically adjusts the training ratio of different layers based on the difference between the performance index of each layer and the corresponding threshold;

[0037] Construct a hierarchical reinforcement learning architecture, and perform targeted training on the networks of the substandard layers in the hierarchical reinforcement learning architecture based on the training samples; repeatedly execute the test performance data generation, scenario data extraction, scenario cluster construction, reward function adjustment, priority update and targeted training steps until all indicators of all layers in the test performance data meet the preset test thresholds.

[0038] According to a second aspect of the embodiments of the present invention,

[0039] Provides an intelligent driving skills training system based on behavior cloning and reinforcement learning, including:

[0040] The first unit is used to collect driving behavior data of a human driver in a preset driving scenario, wherein the driving behavior data includes environmental perception information and control instruction information; divide the driving behavior data into a training data subset and a verification data subset; use the training data subset to train a behavior cloning model, use the environmental perception information as an input feature, use the control instruction information as an output label, use a supervised learning method to obtain an initial driving model, and use the verification data subset to evaluate the performance of the initial driving model to obtain an initial performance evaluation result;

[0041] The second unit is used to determine the driving skill index to be optimized according to the initial performance evaluation result, wherein the driving skill index includes a safety index, an efficiency index and a comfort index; construct a virtual training environment, and set a variety of training scenarios corresponding to the driving skill index in the virtual training environment; design a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index and the comfort index; use the initial driving model as a policy network of a reinforcement learning agent, use a deep reinforcement learning algorithm for training in the virtual training environment, and optimize the parameters of the policy network based on the feedback of the reward function to obtain an optimized driving model;

[0042] The third unit is used to construct a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploy the optimized driving model to the test scenario set for testing to obtain test performance data; when the comparison results of the safety index, efficiency index and comfort index in the test performance data with the preset test thresholds meet the preset conditions, confirm that the training of the optimized driving model is completed; when any indicator in the test performance data is less than the preset test threshold, add the test performance data to the training data set, and continue to perform reinforcement learning training based on the deep reinforcement learning algorithm until the test performance data meets the preset conditions.

[0043] According to a third aspect of the embodiments of the present invention,

[0044] An electronic device is provided, comprising:

[0045] processor;

[0046] a memory for storing processor-executable instructions;

[0047] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0048] According to a fourth aspect of the embodiments of the present invention,

[0049] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0050] The beneficial effects of this application are as follows:

[0051] 1. Improve the performance of the driving model: This method combines the advantages of behavioral cloning and reinforcement learning. First, the driving behavior of human drivers is quickly learned through behavioral cloning to obtain the initial driving model. Then, reinforcement learning is used to optimize specific driving skill indicators in a virtual environment, and finally a driving model with better performance is obtained.

[0052] 2. Support multi-dimensional driving skill optimization: This method designs a reward function that includes safety, efficiency and comfort indicators, and conducts reinforcement learning training based on the reward function. It can achieve multi-dimensional optimization of the driving model, so that the driving model reaches a high level in terms of safety, efficiency and comfort.

[0053] 3. Improve training efficiency and generalization ability: This method uses a virtual training environment for reinforcement learning training, which can construct a variety of training scenarios and avoid the risks and costs of real-world training. At the same time, by testing and iterative training in a set of test scenarios that are not involved in training, the generalization ability of the driving model can be improved, so that it can maintain good performance when facing new driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of an intelligent driving skill training method based on behavior cloning and reinforcement learning according to an embodiment of the present invention;

[0055] Figure 2 It is a structural schematic diagram of an intelligent driving skill training system based on behavior cloning and reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0058] Figure 1 FIG. 1 is a flow chart of an intelligent driving skill training method based on behavior cloning and reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0059] S11. collecting driving behavior data of a human driver in a preset driving scenario, wherein the driving behavior data includes environmental perception information and control instruction information; dividing the driving behavior data into a training data subset and a verification data subset; using the training data subset to train a behavior cloning model, using the environmental perception information as an input feature and the control instruction information as an output label, using a supervised learning method to obtain an initial driving model, and using the verification data subset to evaluate the performance of the initial driving model to obtain an initial performance evaluation result;

[0060] S12. Determine the driving skill index to be optimized according to the initial performance evaluation result, wherein the driving skill index includes a safety index, an efficiency index and a comfort index; construct a virtual training environment, and set a variety of training scenarios corresponding to the driving skill index in the virtual training environment; design a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index and the comfort index; use the initial driving model as the policy network of the reinforcement learning agent, and use a deep reinforcement learning algorithm to train in the virtual training environment, and optimize the parameters of the policy network based on the feedback of the reward function to obtain an optimized driving model;

[0061] S13. Construct a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploy the optimized driving model to the test scenario set for testing to obtain test performance data; when the safety index, efficiency index and comfort index in the test performance data are compared with the preset test threshold and the results meet the preset conditions, confirm that the training of the optimized driving model is completed; when any indicator in the test performance data is less than the preset test threshold, add the test performance data to the training data set, and continue to perform reinforcement learning training based on the deep reinforcement learning algorithm until the test performance data meets the preset conditions.

[0062] In an optional implementation, the driving behavior data is divided into a training data subset and a verification data subset; the behavior cloning model is trained using the training data subset, the environmental perception information is used as an input feature, the control instruction information is used as an output label, an initial driving model is obtained using a supervised learning method, and the performance of the initial driving model is evaluated using the verification data subset, and the initial performance evaluation result obtained includes:

[0063] The driving behavior data is stratified according to scene types, wherein the scene types include straight-line driving scenes, curve turning scenes, lane change scenes, vehicle following scenes, and intersection passing scenes, and random sampling is performed in a ratio of eight to two in each of the scene types to obtain a training data subset and a validation data subset;

[0064] Performing feature extraction preprocessing on the environmental perception information in the training data subset to obtain input features, wherein the feature extraction preprocessing includes: converting the laser radar point cloud data into a three-dimensional grid feature based on the vehicle body coordinate system after downsampling through voxel filtering, converting the forward visual image into a feature vector through key area extraction and spatial pyramid pooling, and normalizing the vehicle state information to a range of negative one to positive one to obtain normalized features; performing normalization processing on the control instruction information in the training data subset to obtain an output label;

[0065] An initial driving model is constructed based on a supervised learning method, wherein the initial driving model includes a feature extraction module, a time series modeling module and a control output module. The feature extraction module uses a three-dimensional convolutional neural network to process the three-dimensional grid features, uses a two-dimensional convolutional neural network to process the feature vectors, and uses a fully connected layer to process the normalized features, and fuses the processed features to obtain fused features; the time series modeling module uses a bidirectional long short-term memory network to process the fused features, and obtains time series features based on the time series information of five seconds before and after; the control output module processes the time series features through a multi-head attention mechanism, and maps the time series features to the corresponding output labels;

[0066] The input features are used as inputs, the output labels are used as training targets, the initial driving model is trained using a multi-task learning framework, different loss weights are set for the steering control task, the longitudinal control task and the behavior prediction task, the steering control task uses a smoothed one-norm loss function, the longitudinal control task uses a mean square error loss function, and the behavior prediction task uses a cross entropy loss function; a two-norm regularization constraint and a random inactivation mechanism are introduced during the training process, and an adaptive moment estimation optimizer is used to optimize the parameters of the initial driving model;

[0067] The verification data subset is input into the trained initial driving model for performance evaluation, and a control accuracy index, a timing consistency index and a behavior similarity index are calculated, wherein the control accuracy index includes a steering angle error mean and an acceleration error root mean square, the timing consistency index includes a control smoothness and an action continuity score, and the behavior similarity index includes a trajectory matching degree and a driving style consistency score, to obtain an initial performance evaluation result.

[0068] An autonomous driving behavior cloning method based on multi-sensor fusion and deep learning aims to train the autonomous driving model by learning human driving behavior data so that it can drive safely and efficiently in various scenarios.

[0069] First, the large amount of collected driving behavior data is preprocessed and divided. These data contain the vehicle's environmental perception information and the corresponding driver control command information in different scenarios. The scene types include straight-line driving, curve turning, lane change, following a car, and intersection passing. For example, the straight-line driving scene data includes the laser radar point cloud data, camera images, vehicle speed, steering wheel angle, etc. when the vehicle is driving in a straight line; the curve turning scene data includes the laser radar point cloud data, camera images, vehicle speed, steering wheel angle, acceleration, etc. when the vehicle is driving in a curve. In order to ensure the generalization ability of the model, the data of each scene type is randomly divided into a training data subset and a verification data subset in a ratio of 8:2. For example, assuming that 10,000 straight-line driving scene data are collected, 8,000 are randomly selected as training data, and the remaining 2,000 are used as verification data.

[0070] Next, the training data subset is preprocessed for feature extraction. The LiDAR point cloud data is downsampled by voxel filtering and converted into a three-dimensional grid feature based on the vehicle body coordinate system. For example, the point cloud space can be divided into several small cubes (voxels), and the features of each voxel can be represented by the number of points contained in it, the average reflectivity, etc. The forward visual image is converted into a feature vector through key area extraction and spatial pyramid pooling. For example, the target detection algorithm can be used to extract key areas such as vehicles, pedestrians, and traffic signs in the image, and then spatial pyramid pooling can be used to integrate regional features of different sizes into a fixed-length vector. Vehicle status information, such as speed, acceleration, etc., is normalized to the range of -1 to 1. For example, if the vehicle speed ranges from 0 to 100km / h, 50km / h is normalized to 0, 100km / h is normalized to 1, and 0km / h is normalized to -1. Control command information, such as steering wheel angle, accelerator pedal opening, brake pedal opening, etc., is also similarly normalized to obtain the output label.

[0071] Then, an initial driving model based on supervised learning is constructed. The model consists of a feature extraction module, a timing modeling module, and a control output module. The feature extraction module uses a three-dimensional convolutional neural network to process three-dimensional grid features, a two-dimensional convolutional neural network to process image feature vectors, and a fully connected layer to process normalized vehicle state features. These processed features are fused to obtain fused features. The timing modeling module uses a bidirectional long short-term memory network to process fused features, and uses the timing information of the five seconds before and after to extract timing features. The control output module uses a multi-head attention mechanism to process timing features and map them to corresponding control command output labels, such as steering wheel angle, acceleration, etc.

[0072] Next, the initial driving model is trained using a subset of the training data. A multi-task learning framework is used to simultaneously train three tasks: steering control, longitudinal control, and behavior prediction. For example, the output of the steering control task is the steering wheel angle, the output of the longitudinal control task is the acceleration, and the output of the behavior prediction task is the vehicle trajectory or driving intention in the next few seconds. Different loss weights are set for each task. The steering control task uses a smoothed one-norm loss function to calculate the difference between the predicted value and the true value, the longitudinal control task uses a mean square error loss function, and the behavior prediction task uses a cross entropy loss function. During the training process, two-norm regularization constraints and random dropout mechanisms are introduced to prevent the model from overfitting. The model parameters are optimized using an adaptive moment estimation optimizer.

[0073] Finally, the performance of the trained initial driving model is evaluated using the validation data subset. The control accuracy index, timing consistency index, and behavior similarity index are calculated. The control accuracy index includes the steering angle error mean and the acceleration error root mean square. The timing consistency index includes the control smoothness and action continuity score. The behavior similarity index includes the trajectory matching and driving style consistency score. These indicators together reflect the driving performance of the model.

[0074] The solution of this application can:

[0075] Improve driving safety: By learning and imitating the behavior of human drivers in various scenarios, the model can better adapt to complex traffic environments, make safer and more reliable driving decisions, and reduce the occurrence of traffic accidents. Improve driving efficiency: The model can make reasonable plans and controls based on road conditions and traffic conditions, optimize the vehicle's driving trajectory, improve fuel economy, and reduce traffic congestion. Enhance driving comfort: The model can learn the smooth driving style of human drivers, avoid uncomfortable operations such as sudden acceleration and sudden braking, and provide a more comfortable riding experience.

[0076] In an optional implementation, a virtual training environment is constructed, and diversified training scenarios corresponding to the driving skill indicators are set in the virtual training environment; a reward function is designed according to the driving skill indicators, and the reward function performs a weighted summation of the scores of the safety indicator, the efficiency indicator, and the comfort indicator, including:

[0077] The driving skill index includes a safety index, an efficiency index and a comfort index. The safety index includes the minimum headway and lane departure distance. The efficiency index includes the average driving speed, the travel time and the task completion time. The comfort index includes the lateral acceleration change rate, the longitudinal acceleration change rate and the steering angle change rate.

[0078] A virtual training environment is constructed based on the driving skill index, and in the virtual training environment, an emergency avoidance scenario, an emergency braking scenario, and a vehicle cut-in scenario are set for the safety index; a multi-lane driving scenario, a roundabout passing scenario, and a traffic light intersection scenario are set for the efficiency index; and a lane keeping scenario, a vehicle following scenario, and a lane changing scenario are set for the comfort index;

[0079] Collecting sensor data in the virtual training environment, the sensor data including laser radar data, camera data and millimeter wave radar data, and calculating the value of the driving skill indicator in real time based on the sensor data;

[0080] A hierarchical reward function is designed according to the driving skill index, wherein the hierarchical reward function uses an exponential penalty function to calculate the deviation between the minimum time distance and the safety threshold to obtain a safety score for the safety index, converts the deviation between the average driving speed and the target speed and the travel time into an efficiency score for the efficiency index, and uses a quadratic penalty function to calculate the lateral acceleration change rate, the longitudinal acceleration change rate and the steering angle change rate to obtain a comfort score for the comfort index;

[0081] Corresponding adaptive weight coefficients are set for the safety score, the efficiency score and the comfort score, the sum of the adaptive weight coefficients is one, and the safety score, the efficiency score and the comfort score are respectively multiplied by the corresponding adaptive weight coefficients and the sum is obtained to obtain a total score.

[0082] Build a virtual training environment for training autonomous driving models, which can simulate various real-world driving scenarios and provide a comprehensive evaluation system based on multiple indicators.

[0083] First, define driving skill indicators, which cover three aspects: safety, efficiency, and comfort. Safety indicators include minimum headway (the minimum distance between the vehicle and the obstacle in front) and lane departure distance. Efficiency indicators include average driving speed, time to complete a specific section of road, and total time to complete a preset task. Comfort indicators include lateral acceleration change rate, longitudinal acceleration change rate, and steering angle change rate.

[0084] Next, according to the defined driving skill indicators, a variety of training scenarios are constructed in the virtual training environment. For safety indicators, emergency avoidance scenarios (for example, pedestrians or vehicles suddenly appear in front), emergency braking scenarios, and vehicle cut-in scenarios are set. For efficiency indicators, multi-lane driving scenarios, roundabout traffic scenarios, and traffic light intersection scenarios are set. For comfort indicators, lane keeping scenarios, following vehicle scenarios, and lane changing scenarios are set. For example, in an emergency avoidance scenario, an autonomous driving vehicle is set to travel on a straight road at a speed of 50 kilometers per hour, and a stationary vehicle suddenly appears 20 meters ahead. Test whether the autonomous driving vehicle can avoid it safely.

[0085] In a virtual training environment, various sensors such as LiDAR, cameras, and millimeter-wave radar are simulated and data from these sensors is collected. This data will be used to calculate the value of driving skill indicators in real time. For example, the lane line is identified using camera data, and the distance between the vehicle and the lane line is calculated to obtain the value of the lane deviation distance. Assume that in a scenario, the LiDAR data shows that the distance between the vehicle and the vehicle in front is 10 meters, the camera data shows that the vehicle deviates from the center line of the lane by 0.5 meters, and the millimeter-wave radar data shows that the lateral acceleration change rate of the vehicle is 2m / s³.

[0086] Then, a hierarchical reward function is designed to evaluate the performance of the autonomous driving model. The reward function weights the scores of safety, efficiency, and comfort indicators. For the safety indicator, an exponential penalty function is used to calculate the deviation of the minimum headway from the safety threshold (for example, 2 meters) to obtain the safety score. The closer the minimum headway is to the safety threshold, the smaller the penalty and the higher the score. For the efficiency indicator, the deviation of the average driving speed from the target speed (for example, the speed limit) and the travel time are converted into an efficiency score. The closer the average driving speed is to the target speed and the shorter the travel time, the higher the score. For the comfort indicator, a quadratic penalty function is used to calculate the lateral acceleration change rate, the longitudinal acceleration change rate, and the steering angle change rate to obtain the comfort score. The smaller the acceleration change rate and the steering angle change rate, the more comfortable the ride experience and the higher the score. For example, suppose that in a scenario, the safety score is 80 points, the efficiency score is 90 points, and the comfort score is 70 points.

[0087] Finally, set the corresponding adaptive weight coefficients for the safety score, efficiency score, and comfort score. The sum of these weight coefficients is 1, and they will be dynamically adjusted according to the progress of training and the performance of the model. The safety score, efficiency score, and comfort score are multiplied by the corresponding adaptive weight coefficients and summed to get the total score.

[0088] The solution of this application can:

[0089] Improve training efficiency: By building a variety of virtual training scenarios and hierarchical reward functions, autonomous driving models can be trained more efficiently, shortening training time and reducing training costs. Virtual environments can quickly simulate various complex traffic scenarios and provide a large amount of training data, thereby accelerating the learning speed of the model. Enhance model robustness: Virtual training environments can simulate various extreme and dangerous scenarios, such as emergency avoidance and emergency braking, which are difficult or costly to reproduce in the real world. By training models in a virtual environment to deal with these scenarios, the robustness and safety of the model can be enhanced, enabling it to make safer decisions when faced with emergencies. Improve safety: The virtual training environment provides a safe testing platform for comprehensive testing and evaluation of autonomous driving models without endangering real-world safety. This can effectively reduce the risk of accidents caused by model defects and improve the safety of autonomous driving technology.

[0090] In an optional implementation, a test scenario set is constructed in the virtual training environment, the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; the optimized driving model is deployed to the test scenario set for testing to obtain test performance data; when the safety index, efficiency index and comfort index in the test performance data are all greater than a preset test threshold, confirming that the training of the optimized driving model is completed includes:

[0091] Constructing a test scenario set in the virtual training environment, the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios, and the scenarios that are not involved in the training are constructed by changing the road curvature parameters, lane width parameters, road friction coefficient parameters, traffic flow density parameters and weather environment parameters in the diversified training scenarios; deploying the optimized driving model to the test scenario set for testing, and repeatedly executing the test on each test scenario using the Monte Carlo method to obtain test performance data;

[0092] Calculating the safety index, recording the minimum headway of the safety index being less than two seconds as a dangerous event, counting the frequency of occurrence of the dangerous event during the test, and obtaining a maximum value of the lane departure distance in the safety index based on the frequency of occurrence;

[0093] Calculate the theoretical shortest travel time of each test path, and use the ratio of the task completion time in the efficiency index to the theoretical shortest travel time as the task completion time ratio;

[0094] Calculating the comfort index, performing statistical distribution characteristic analysis on the longitudinal acceleration change rate and the lateral acceleration change rate in the comfort index, and calculating the ninety-fifth percentile values ​​of the longitudinal acceleration change rate and the lateral acceleration change rate;

[0095] The preset test thresholds include a dangerous event frequency threshold, a lane departure distance threshold, an average driving speed threshold, a task completion time ratio threshold, a longitudinal acceleration change rate threshold, and a lateral acceleration change rate threshold;

[0096] When the safety index, the efficiency index and the comfort index all meet the judgment conditions in fifty consecutive tests, it is confirmed that the training of the optimized driving model is completed.

[0097] Autonomous driving model training and verification methods

[0098] First, build a variety of training scenarios in a virtual training environment. These scenarios cover various road types, traffic conditions, and weather conditions, such as urban roads, highways, rural roads, as well as sunny, rainy, foggy, etc. Set parameters such as road curvature, lane width, road friction coefficient, traffic flow density, and combine different parameter values ​​to generate a large number of training scenarios for training autonomous driving models. For example, you can set an urban road scene, with a curvature radius randomly selected between 50 meters and 200 meters, a lane width randomly selected between 3 meters and 4 meters, a road friction coefficient randomly selected between 0.7 and 0.9, and a traffic flow density randomly selected between 1,000 and 3,000 vehicles per hour. The weather is set to sunny or rainy.

[0099] Then, based on the existing diverse training scenarios, a test scenario set is constructed. The test scenario set contains not only training scenarios, but also new scenarios that have not participated in the training. These new scenarios are generated by changing the parameters in the training scenarios. For example, the road curvature parameter in the training scenario is increased by 50%, or the lane width parameter is reduced by 10%, or the road friction coefficient is reduced by 20%, or the traffic flow density is doubled, or the weather environment is changed to foggy, etc., so as to generate new scenarios different from the training scenarios.

[0100] Next, the trained autonomous driving model is deployed to a set of test scenarios for testing. The Monte Carlo method is used to repeat the test multiple times for each test scenario, for example, 50 times, and the test performance data is collected, including vehicle trajectory, speed, acceleration, distance to other vehicles, etc.

[0101] Based on the collected test performance data, safety indicators, efficiency indicators and comfort indicators are calculated.

[0102] The calculation method of the safety index is as follows: the frequency of dangerous events during the test is counted. Dangerous events are defined as the minimum headway time less than two seconds or lane departure warning. For example, in 50 tests, there are 2 events with the minimum headway time less than two seconds and 5 lane departure warnings, then the frequency of dangerous events is (2+5) / 50=14%. At the same time, the maximum lane departure distance is recorded.

[0103] The efficiency index is calculated as follows: Calculate the theoretical shortest travel time for each test path, and use the ratio of the actual task completion time to the theoretical shortest travel time as the task completion time ratio. For example, if the theoretical shortest travel time for a test path is 100 seconds and the actual task completion time is 120 seconds, then the task completion time ratio is 120 / 100=1.2.

[0104] The calculation method of the comfort index is as follows: Analyze the statistical distribution characteristics of the longitudinal acceleration change rate and the lateral acceleration change rate, and calculate their 95th percentile values. For example, during the test, the 95th percentile value of the longitudinal acceleration change rate is 2m / s³, and the 95th percentile value of the lateral acceleration change rate is 1.5m / s³.

[0105] The test thresholds are pre-set, including the frequency threshold of dangerous events, the lane departure distance threshold, the average driving speed threshold, the task completion time ratio threshold, the longitudinal acceleration change rate threshold, and the lateral acceleration change rate threshold. For example, the frequency threshold of dangerous events is set to 5%, the lane departure distance threshold is set to 0.5 meters, the average driving speed threshold is set to 40 kilometers per hour, the task completion time ratio threshold is set to 1.5, the longitudinal acceleration change rate threshold is set to 3m / s³, and the lateral acceleration change rate threshold is set to 2m / s³.

[0106] Determine whether the test performance data meets the preset test threshold. If the safety index, efficiency index and comfort index all meet the judgment conditions in 50 consecutive tests, that is, the frequency of dangerous events is less than the threshold of dangerous event frequency, the maximum lane departure distance is less than the threshold of lane departure distance, the average driving speed is greater than the average driving speed threshold, the task completion time ratio is less than the task completion time ratio threshold, the 95th percentile value of the longitudinal acceleration change rate is less than the longitudinal acceleration change rate threshold, and the 95th percentile value of the lateral acceleration change rate is less than the lateral acceleration change rate threshold, then it is confirmed that the autonomous driving model training is completed.

[0107] The solution of this application can:

[0108] Improve safety: By testing in a test set that includes scenarios that were not included in the training, and focusing on safety indicators such as the frequency of dangerous events and lane departure distance, the safety of the autonomous driving model in complex and unknown environments can be effectively improved. Improve efficiency: By evaluating efficiency indicators such as the task completion time ratio, the driving strategy of the autonomous driving model can be optimized so that it can complete the driving task as quickly as possible while ensuring safety. Improve comfort: By analyzing comfort indicators such as the longitudinal acceleration change rate and the lateral acceleration change rate, the control strategy of the autonomous driving model can be adjusted to make the driving process smoother and more comfortable, improving the passenger experience.

[0109] In an optional implementation, when any indicator in the test performance data is less than the preset test threshold, the test performance data is added to the training data set, and reinforcement learning training is continued based on the deep reinforcement learning algorithm until the test performance data meets the preset conditions including:

[0110] When any indicator of any level in the test performance data is less than the corresponding preset test threshold, extract the input data, intermediate features and output results of the level to which the performance non-compliant indicator belongs, and construct a scene data sequence containing time series association; perform spatiotemporal dimension reduction and feature extraction on the scene data sequence to generate a scene feature vector;

[0111] Based on the scene feature vector, similarity calculation is performed on different scenes, and scenes with similarity higher than a preset similarity threshold are clustered and stored to form scene group data; a hierarchical adaptive reward function is constructed, wherein the hierarchical adaptive reward function includes a perception layer reward function, a decision layer reward function, and a control layer reward function, and the weight of each layer reward function is dynamically adjusted based on the difference between the test performance data and the preset test threshold;

[0112] Construct a multi-level experience replay buffer pool, wherein the multi-level experience replay buffer pool includes a perception layer buffer pool, a decision layer buffer pool, and a control layer buffer pool, and assign intra-layer priority and inter-layer priority to training samples according to the difference between the performance index of each layer and the corresponding threshold;

[0113] The scene group data are stored in the corresponding multi-level experience replay buffer pool according to the level to which the performance non-compliance indicator belongs; the performance improvement degree of each scene group in the multi-level experience replay buffer pool is regularly evaluated, and when the test performance data corresponding to a scene group is higher than the preset test threshold for multiple consecutive times, the scene group is removed from the corresponding buffer pool;

[0114] Sampling training samples from the multi-level experience replay buffer pool based on the intra-layer priority and the inter-layer priority; adopting a curriculum learning strategy to preferentially train network layers with performance gaps, wherein the curriculum learning strategy dynamically adjusts the training ratio of different layers based on the difference between the performance index of each layer and the corresponding threshold;

[0115] Construct a hierarchical reinforcement learning architecture, and perform targeted training on the networks of the substandard layers in the hierarchical reinforcement learning architecture based on the training samples; repeatedly execute the test performance data generation, scenario data extraction, scenario cluster construction, reward function adjustment, priority update and targeted training steps until all indicators of all layers in the test performance data meet the preset test thresholds.

[0116] A hierarchical adaptive training method based on deep reinforcement learning is used to improve the performance of complex tasks, especially when the task can be decomposed into multiple levels such as perception, decision-making and control. The core of this method is to dynamically adjust the training strategy according to the test performance data, and improve the level that does not meet the performance requirements until all levels meet the preset performance requirements.

[0117] First, the system is initialized, including defining the reinforcement learning environment, building a hierarchical reinforcement learning architecture (e.g., dividing the deep neural network into a perception layer, a decision layer, and a control layer), setting preset test thresholds (e.g., setting a threshold for each performance indicator at each layer), and initializing the hierarchical adaptive reward function and the multi-level experience replay buffer pool. For example, the performance indicators of the perception layer can be the accuracy and recall rate of target detection, the performance indicators of the decision layer can be the efficiency and safety of path planning, and the performance indicators of the control layer can be the accuracy and speed of action execution.

[0118] Next, the training process begins. First, the current reinforcement learning model is used to interact with the environment to generate test performance data. For example, the trained model is run in an autonomous driving simulation environment, and the performance indicators of each layer are recorded.

[0119] Then, determine whether the test performance data meets the preset conditions. If all indicators at all levels meet or exceed the preset threshold, the training ends. Otherwise, extract the input data, intermediate features, and output results of the level to which the performance indicator does not meet the standard. For example, if the path planning efficiency of the decision layer is lower than the preset threshold, record the input data (e.g., sensor data), intermediate features (e.g., map information), and output results (e.g., planned path) that cause the inefficiency.

[0120] Next, construct a scene data sequence containing time series associations. For example, concatenate the input data, intermediate features, and output results of multiple consecutive time steps to form a scene data sequence that reflects the behavior process of the performance substandard level in the scene. Perform spatiotemporal dimensionality reduction and feature extraction on the scene data sequence to generate a scene feature vector. For example, use principal component analysis (PCA) to reduce the dimensionality of the scene data sequence, extract key features, and form a compact scene feature vector.

[0121] Based on the scene feature vector, the similarity of different scenes is calculated, and the scenes with similarity higher than the preset similarity threshold are clustered and stored to form scene group data. For example, the similarity between different scene feature vectors is calculated using cosine similarity, and scenes with similarity higher than 0.8 are classified into a scene group.

[0122] Then, dynamically adjust the hierarchical adaptive reward function. The reward function includes the perception layer reward function, the decision layer reward function, and the control layer reward function. The weight of each layer reward function is dynamically adjusted according to the difference between the test performance data and the preset test threshold. For example, if the performance index of the decision layer is far away from the threshold, the weight of the decision layer reward function is increased to guide the model to pay more attention to the performance improvement of the decision layer. Assuming that the preset threshold of the decision layer path planning efficiency is 90%, and the current test result is 70%, the weight of the decision layer reward function is increased, for example, from 0.5 to 0.7.

[0123] Next, the scene group data is stored in the corresponding multi-level experience playback buffer pool according to the level of the performance non-compliance indicator. The buffer pool includes the perception layer buffer pool, the decision layer buffer pool and the control layer buffer pool. For example, the scene group data related to the non-compliance of the decision layer performance is stored in the decision layer buffer pool.

[0124] Assign intra-layer priority and inter-layer priority to training samples based on the difference between each layer's performance indicator and the corresponding threshold. For example, the larger the difference, the higher the priority. Assuming that the current value of the perception layer's accuracy is 80% and the threshold is 95%, and the current value of the decision layer's efficiency is 70% and the threshold is 90%, the inter-layer priority of the decision layer is higher than that of the perception layer. Within the decision layer, less efficient scene groups have higher intra-layer priorities.

[0125] Regularly evaluate the performance improvement of each scenario group in the multi-level experience playback buffer pool. When the test performance data corresponding to a scenario group exceeds the preset test threshold for multiple consecutive times, remove the scenario group from the corresponding buffer pool. For example, if the decision-making efficiency of a scenario group exceeds 90% in three consecutive tests, remove the scenario group from the decision-making buffer pool.

[0126] Training samples are sampled from the multi-level experience replay buffer pool based on intra-layer priority and inter-layer priority. A curriculum learning strategy is adopted to prioritize the training of network layers with performance gaps. For example, if the performance gap of the decision layer is the largest, training samples are sampled from the decision layer buffer pool first, and the training ratio of the decision layer is increased.

[0127] Based on the training samples, the networks of the substandard layers in the hierarchical reinforcement learning architecture are trained in a targeted manner. The above steps are repeated until all indicators of all layers meet the preset test thresholds.

[0128] The solution of this application can:

[0129] Improve training efficiency: By training the layers that do not meet the performance requirements, unnecessary training of the layers that have already met the performance requirements is avoided, thereby improving training efficiency. Improve overall performance: Through layered training and adaptive reward functions, the performance of each layer can be effectively improved, ultimately improving the overall performance of the system. Enhance robustness: By clustering and prioritizing similar scenarios, the model can be better learned and generalized to a variety of different scenarios, thereby enhancing the robustness of the model.

[0130] Figure 2 FIG. 1 is a schematic diagram of the structure of an intelligent driving skill training system based on behavior cloning and reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0131] The first unit is used to collect driving behavior data of a human driver in a preset driving scenario, wherein the driving behavior data includes environmental perception information and control instruction information; divide the driving behavior data into a training data subset and a verification data subset; use the training data subset to train a behavior cloning model, use the environmental perception information as an input feature, use the control instruction information as an output label, use a supervised learning method to obtain an initial driving model, and use the verification data subset to evaluate the performance of the initial driving model to obtain an initial performance evaluation result;

[0132] The second unit is used to determine the driving skill index to be optimized according to the initial performance evaluation result, wherein the driving skill index includes a safety index, an efficiency index and a comfort index; construct a virtual training environment, and set a variety of training scenarios corresponding to the driving skill index in the virtual training environment; design a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index and the comfort index; use the initial driving model as a policy network of a reinforcement learning agent, use a deep reinforcement learning algorithm for training in the virtual training environment, and optimize the parameters of the policy network based on the feedback of the reward function to obtain an optimized driving model;

[0133] The third unit is used to construct a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploy the optimized driving model to the test scenario set for testing to obtain test performance data; when the comparison results of the safety index, efficiency index and comfort index in the test performance data with the preset test thresholds meet the preset conditions, confirm that the training of the optimized driving model is completed; when any indicator in the test performance data is less than the preset test threshold, add the test performance data to the training data set, and continue to perform reinforcement learning training based on the deep reinforcement learning algorithm until the test performance data meets the preset conditions.

[0134] According to a third aspect of the embodiments of the present invention,

[0135] An electronic device is provided, comprising:

[0136] processor;

[0137] a memory for storing processor-executable instructions;

[0138] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0139] According to a fourth aspect of the embodiments of the present invention,

[0140] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0141] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent driving skill training method based on behavior cloning and reinforcement learning, characterized in that: include: Collecting driving behavior data of human drivers in preset driving scenarios, wherein the driving behavior data includes environmental perception information and control command information; The driving behavior data is divided into a training data subset and a verification data subset; the behavior cloning model is trained using the training data subset, the environmental perception information is used as an input feature, the control instruction information is used as an output label, an initial driving model is obtained by a supervised learning method, and the performance of the initial driving model is evaluated using the verification data subset to obtain an initial performance evaluation result; Determining a driving skill index to be optimized according to the initial performance evaluation result, wherein the driving skill index includes a safety index, an efficiency index, and a comfort index; constructing a virtual training environment, and setting a variety of training scenarios corresponding to the driving skill index in the virtual training environment; Designing a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index, and the comfort index; Using the initial driving model as a policy network of a reinforcement learning agent, training with a deep reinforcement learning algorithm in the virtual training environment, optimizing the parameters of the policy network based on feedback from the reward function, and obtaining an optimized driving model; Constructing a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploying the optimized driving model to the test scenario set for testing, and obtaining test performance data; When the comparison results of the safety index, efficiency index and comfort index in the test performance data with the preset test thresholds meet the preset conditions, it is confirmed that the training of the optimized driving model is completed; when any indicator in the test performance data is less than the preset test threshold, the test performance data is added to the training data set, and reinforcement learning training based on the deep reinforcement learning algorithm is continued until the test performance data meets the preset conditions.

2. The method according to claim 1, characterized in that: The driving behavior data is divided into a training data subset and a verification data subset; the behavior cloning model is trained using the training data subset, the environmental perception information is used as an input feature, the control instruction information is used as an output label, an initial driving model is obtained using a supervised learning method, and the performance of the initial driving model is evaluated using the verification data subset, and the initial performance evaluation results obtained include: The driving behavior data is stratified according to scene types, wherein the scene types include straight-line driving scenes, curve turning scenes, lane change scenes, vehicle following scenes, and intersection passing scenes, and random sampling is performed in a ratio of eight to two in each of the scene types to obtain a training data subset and a validation data subset; Performing feature extraction preprocessing on the environmental perception information in the training data subset to obtain input features, wherein the feature extraction preprocessing includes: converting the laser radar point cloud data into a three-dimensional grid feature based on the vehicle body coordinate system after downsampling through voxel filtering, converting the forward visual image into a feature vector through key area extraction and spatial pyramid pooling, and normalizing the vehicle state information to a range of negative one to positive one to obtain normalized features; performing normalization processing on the control instruction information in the training data subset to obtain an output label; An initial driving model is constructed based on a supervised learning method, wherein the initial driving model includes a feature extraction module, a time series modeling module and a control output module. The feature extraction module uses a three-dimensional convolutional neural network to process the three-dimensional grid features, uses a two-dimensional convolutional neural network to process the feature vectors, and uses a fully connected layer to process the normalized features, and fuses the processed features to obtain fused features; the time series modeling module uses a bidirectional long short-term memory network to process the fused features, and obtains time series features based on the time series information of five seconds before and after; the control output module processes the time series features through a multi-head attention mechanism, and maps the time series features to the corresponding output labels; The input features are used as inputs, the output labels are used as training targets, the initial driving model is trained using a multi-task learning framework, different loss weights are set for the steering control task, the longitudinal control task and the behavior prediction task, the steering control task uses a smoothed one-norm loss function, the longitudinal control task uses a mean square error loss function, and the behavior prediction task uses a cross entropy loss function; a two-norm regularization constraint and a random inactivation mechanism are introduced during the training process, and an adaptive moment estimation optimizer is used to optimize the parameters of the initial driving model; The verification data subset is input into the trained initial driving model for performance evaluation, and a control accuracy index, a timing consistency index and a behavior similarity index are calculated, wherein the control accuracy index includes a steering angle error mean and an acceleration error root mean square, the timing consistency index includes a control smoothness and an action continuity score, and the behavior similarity index includes a trajectory matching degree and a driving style consistency score, to obtain an initial performance evaluation result.

3. The method according to claim 1, characterized in that Constructing a virtual training environment, and setting various training scenarios corresponding to the driving skill indicators in the virtual training environment; A reward function is designed according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index and the comfort index, including: The driving skill index includes a safety index, an efficiency index and a comfort index. The safety index includes the minimum headway and lane departure distance. The efficiency index includes the average driving speed, the travel time and the task completion time. The comfort index includes the lateral acceleration change rate, the longitudinal acceleration change rate and the steering angle change rate. A virtual training environment is constructed based on the driving skill index, and in the virtual training environment, an emergency avoidance scenario, an emergency braking scenario, and a vehicle cut-in scenario are set for the safety index; a multi-lane driving scenario, a roundabout passing scenario, and a traffic light intersection scenario are set for the efficiency index; and a lane keeping scenario, a vehicle following scenario, and a lane changing scenario are set for the comfort index; Collecting sensor data in the virtual training environment, the sensor data including laser radar data, camera data and millimeter wave radar data, and calculating the value of the driving skill indicator in real time based on the sensor data; A hierarchical reward function is designed according to the driving skill index, wherein the hierarchical reward function uses an exponential penalty function to calculate the deviation between the minimum time distance and the safety threshold to obtain a safety score for the safety index, converts the deviation between the average driving speed and the target speed and the travel time into an efficiency score for the efficiency index, and uses a quadratic penalty function to calculate the lateral acceleration change rate, the longitudinal acceleration change rate and the steering angle change rate to obtain a comfort score for the comfort index; Corresponding adaptive weight coefficients are set for the safety score, the efficiency score and the comfort score, the sum of the adaptive weight coefficients is one, and the safety score, the efficiency score and the comfort score are respectively multiplied by the corresponding adaptive weight coefficients and the sum is obtained to obtain a total score.

4. The method according to claim 3, characterized in that The initial driving model is used as a policy network of a reinforcement learning agent, a deep reinforcement learning algorithm is used for training in the virtual training environment, and the parameters of the policy network are optimized based on the feedback of the reward function, so as to obtain an optimized driving model including: The initial driving model is used as a policy network of a reinforcement learning agent, wherein the policy network includes an input layer, a hidden layer, and an output layer, wherein the input layer receives vehicle state information in the virtual training environment, wherein the vehicle state information includes current speed, current acceleration, current heading angle, and relative positions and relative speeds of surrounding vehicles, wherein the hidden layer includes a three-layer fully connected layer structure, and wherein the output layer generates control instructions for a continuous action space; In the virtual training environment, a deep reinforcement learning algorithm is used to train the policy network, wherein the deep reinforcement learning algorithm is designed based on an Actor-Critic architecture, the policy network is used as an Actor network to generate actions, and a Critic network is set to evaluate a state-action value function; An experience replay pool is constructed based on the immediate rewards, reward training data is sampled from the experience replay pool, and the parameters of the policy network are iteratively updated using a proximal policy optimization algorithm, wherein the proximal policy optimization algorithm ensures training stability by limiting the degree of difference between the new and old policies; During the training process, exploratory noise is added to the actions output by the policy network, wherein the variance of the exploratory noise decays as the number of training rounds increases, thereby realizing a policy transition from exploration to utilization; The performance of the policy network is evaluated in a test scenario. When the safety score, the efficiency score, and the comfort score exceed preset policy thresholds for multiple consecutive rounds, the parameters of the current policy network are optimized to obtain an optimized driving model.

5. The method according to claim 3, characterized in that: Constructing a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploying the optimized driving model to the test scenario set for testing, and obtaining test performance data; When the safety index, efficiency index and comfort index in the test performance data are all greater than the preset test threshold, confirming that the optimized driving model training is completed includes: Constructing a test scenario set in the virtual training environment, the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios, and the scenarios that are not involved in the training are constructed by changing the road curvature parameters, lane width parameters, road friction coefficient parameters, traffic flow density parameters and weather environment parameters in the diversified training scenarios; deploying the optimized driving model to the test scenario set for testing, and repeatedly executing the test on each test scenario using the Monte Carlo method to obtain test performance data; Calculating the safety index, recording the minimum headway of the safety index being less than two seconds as a dangerous event, counting the frequency of occurrence of the dangerous event during the test, and obtaining a maximum value of the lane departure distance in the safety index based on the frequency of occurrence; Calculate the theoretical shortest travel time of each test path, and use the ratio of the task completion time in the efficiency index to the theoretical shortest travel time as the task completion time ratio; Calculating the comfort index, performing statistical distribution characteristic analysis on the longitudinal acceleration change rate and the lateral acceleration change rate in the comfort index, and calculating the ninety-fifth percentile values ​​of the longitudinal acceleration change rate and the lateral acceleration change rate; The preset test thresholds include a dangerous event frequency threshold, a lane departure distance threshold, an average driving speed threshold, a task completion time ratio threshold, a longitudinal acceleration change rate threshold, and a lateral acceleration change rate threshold; When the safety index, the efficiency index and the comfort index all meet the judgment conditions in fifty consecutive tests, it is confirmed that the training of the optimized driving model is completed.

6. The method according to claim 1, characterized in that When any indicator in the test performance data is less than the preset test threshold, the test performance data is added to the training data set, and reinforcement learning training is continued based on the deep reinforcement learning algorithm until the test performance data meets the preset conditions including: When any indicator of any level in the test performance data is less than the corresponding preset test threshold, extract the input data, intermediate features and output results of the level to which the performance non-compliant indicator belongs, and construct a scene data sequence containing time series association; perform spatiotemporal dimension reduction and feature extraction on the scene data sequence to generate a scene feature vector; Based on the scene feature vector, similarity calculation is performed on different scenes, and scenes with similarity higher than a preset similarity threshold are clustered and stored to form scene group data; a hierarchical adaptive reward function is constructed, wherein the hierarchical adaptive reward function includes a perception layer reward function, a decision layer reward function, and a control layer reward function, and the weight of each layer reward function is dynamically adjusted based on the difference between the test performance data and the preset test threshold; Construct a multi-level experience replay buffer pool, wherein the multi-level experience replay buffer pool includes a perception layer buffer pool, a decision layer buffer pool, and a control layer buffer pool, and assign intra-layer priority and inter-layer priority to training samples according to the difference between the performance index of each layer and the corresponding threshold; The scene group data are stored in the corresponding multi-level experience replay buffer pool according to the level to which the performance non-compliance indicator belongs; the performance improvement degree of each scene group in the multi-level experience replay buffer pool is regularly evaluated, and when the test performance data corresponding to a scene group is higher than the preset test threshold for multiple consecutive times, the scene group is removed from the corresponding buffer pool; Sampling training samples from the multi-level experience replay buffer pool based on the intra-layer priority and the inter-layer priority; adopting a curriculum learning strategy to preferentially train network layers with performance gaps, wherein the curriculum learning strategy dynamically adjusts the training ratio of different layers based on the difference between the performance index of each layer and the corresponding threshold; Construct a hierarchical reinforcement learning architecture, and perform targeted training on the networks of the substandard layers in the hierarchical reinforcement learning architecture based on the training samples; repeatedly execute the test performance data generation, scenario data extraction, scenario cluster construction, reward function adjustment, priority update and targeted training steps until all indicators of all layers in the test performance data meet the preset test thresholds.

7. An intelligent driving skill training system based on behavior cloning and reinforcement learning, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to collect driving behavior data of a human driver in a preset driving scenario, wherein the driving behavior data includes environmental perception information and control command information; The driving behavior data is divided into a training data subset and a verification data subset; the behavior cloning model is trained using the training data subset, the environmental perception information is used as an input feature, the control instruction information is used as an output label, an initial driving model is obtained by a supervised learning method, and the performance of the initial driving model is evaluated using the verification data subset to obtain an initial performance evaluation result; The second unit is used to determine the driving skill index to be optimized according to the initial performance evaluation result, wherein the driving skill index includes a safety index, an efficiency index and a comfort index; construct a virtual training environment, and set a variety of training scenarios corresponding to the driving skill index in the virtual training environment; Designing a reward function according to the driving skill index, wherein the reward function performs a weighted summation of the scores of the safety index, the efficiency index, and the comfort index; Using the initial driving model as a policy network of a reinforcement learning agent, training with a deep reinforcement learning algorithm in the virtual training environment, optimizing the parameters of the policy network based on feedback from the reward function, and obtaining an optimized driving model; A third unit is used to construct a test scenario set in the virtual training environment, wherein the test scenario set includes scenarios that are not involved in the training in the diversified training scenarios; deploy the optimized driving model to the test scenario set for testing, and obtain test performance data; When the comparison results of the safety index, efficiency index and comfort index in the test performance data with the preset test thresholds meet the preset conditions, it is confirmed that the training of the optimized driving model is completed; when any indicator in the test performance data is less than the preset test threshold, the test performance data is added to the training data set, and reinforcement learning training based on the deep reinforcement learning algorithm is continued until the test performance data meets the preset conditions.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images