An end-to-end automatic driving strategy evolution method combining supervision and reinforcement fine-tuning

Abstract

The present application relates to the field of automatic driving strategy evolution, and in particular to an end-to-end automatic driving strategy evolution method based on supervised-reinforcement joint fine-tuning, wherein the data samples of the user using the automatic driving system are divided into performance defect samples, physiological and psychological feedback samples, partial intervention samples and complete takeover samples according to feedback levels, the end-to-end automatic driving strategy evolution is realized by using a reinforcement learning fine-tuning method for the performance defect samples and the physiological and psychological feedback samples, the end-to-end automatic driving strategy evolution is realized by using a supervised learning fine-tuning method for the partial intervention samples and the complete takeover samples, a dynamic sample library is established, the memory samples are dynamically updated according to the difference degree of the samples, the memory sample loss is weighted and calculated according to the difference degree between the memory samples and the samples currently used for fine-tuning training, and finally the memory sample loss is used to perform soft constraint on the reinforcement learning fine-tuning and the supervised learning fine-tuning. The present application makes full use of multi-scene data sample flow and can continuously learn.

Description

Technical Field

[0001] This invention relates to a method for the evolution of autonomous driving strategies, specifically to an end-to-end method for the evolution of autonomous driving strategies that combines supervised fine-tuning and reinforcement fine-tuning. Background Technology

[0002] With the rapid development of artificial intelligence technology, autonomous driving systems are moving from a hierarchical architecture to an end-to-end architecture. However, existing end-to-end autonomous driving systems are generally trained in a single open-loop manner, resulting in fixed policy structures and parameters. Unlike human drivers, they cannot continuously learn and improve during driving, leading to weak environmental adaptability and low user acceptance in current autonomous vehicles. Therefore, how to enable autonomous driving systems to continuously evolve during actual operation, truly adapt to complex and ever-changing real-world traffic environments, and continuously improve performance has become a key technical challenge that urgently needs to be overcome in the field of autonomous driving. In recent years, several self-evolutionary methods for autonomous driving strategies have been proposed. The main idea is to fine-tune driving strategies using user feedback data from autonomous vehicles. Depending on the fine-tuning method used, self-evolutionary methods for autonomous driving strategies can be divided into two types: supervised fine-tuning methods and reinforcement fine-tuning methods.

[0003] (1) The supervised fine-tuning method is a supervised training method based on human driver intervention or takeover behavior. In this method, the autonomous driving system simulates the actual driving process in the background but does not actually control the vehicle. At the same time, it continuously monitors the operation of the human driver and finally fine-tunes the driving strategy using human driving samples as labeled data under offline conditions.

[0004] (2) The enhanced fine-tuning method is an unsupervised training method based on the evaluation benchmark of intelligent driving strategy. This method uses a simulated environment to fine-tune the pre-trained autonomous driving strategy, improves the comprehensiveness of the training samples of autonomous driving strategy, and effectively solves the distribution offset problem that may be encountered when deploying autonomous driving strategies trained by open-loop supervised learning.

[0005] Existing fine-tuning methods still have the following problems:

[0006] (1) User feedback data samples from autonomous driving systems are diverse, and existing fine-tuning methods do not adequately collect and utilize the data. During user interaction with autonomous driving systems, four scenarios can be categorized based on the level of human feedback: normal operation of the autonomous driving system, user discomfort caused by the autonomous driving system, partial user intervention in the autonomous driving system, and complete user takeover of the driving system. Existing supervisory fine-tuning methods only retrieve driving data from scenarios where the user takes complete control of the driving system. However, this scenario has a limited proportion in the overall operation of the autonomous driving function and cannot comprehensively and accurately reflect the driver's satisfaction with the vehicle's autonomous driving system.

[0007] (2) Existing supervised fine-tuning methods collect user driving behavior data and then use offline methods to supervise learning and update the policy; existing reinforcement fine-tuning methods allow the agent to explore freely in an offline virtual environment and then update the policy based on the exploration data. It can be seen that most of the existing fine-tuning methods are offline updates, which limits the flexibility of the self-evolution of autonomous driving policies. The reason for this is that they lack the constraint of remembering old data.

[0008] (3) Existing supervised fine-tuning methods aim to improve the similarity between autonomous driving strategies and human driving behavior mechanisms, but suboptimal driver behaviors in the dataset limit the performance ceiling of autonomous driving strategies.

[0009] (4) The existing reward model of the reinforcement fine-tuning method is too crude and cannot reflect the real experience and needs of the driver, making it difficult to adapt to the real operating environment of the autonomous driving system.

[0010] Therefore, how to develop an online evolution method for autonomous driving strategies that can fully and efficiently utilize multi-scenario data sample streams and continuously learn has become a core technical challenge that urgently needs to be overcome in the field of intelligent vehicle technology. Summary of the Invention

[0011] To address the aforementioned technical problems, this invention provides an end-to-end autonomous driving strategy evolution method based on supervised-reinforcement joint fine-tuning, comprising the following steps:

[0012] First, regarding performance defect samples and physiological and psychological feedback samples The unlabeled sample library was obtained by merging the samples.

[0013] A human-like reward function is constructed through inverse reinforcement learning, and an action value estimation network model is pre-trained. The human-like reward function is dynamically generated by the human-like reward model constructed through inverse reinforcement learning, ensuring that the policy optimization direction aligns with the driver's cognition and habits. The inverse reinforcement learning process includes an alternating iterative process of inverse reinforcement learning and forward reinforcement learning, which takes place in a virtual environment. The action value estimation network model is used to determine the degree of defect in autonomous driving samples and filter out performance defective samples, and also serves as the initial action value estimation network for reinforcement learning. The autonomous driving policy and the action value estimation network model are then fine-tuned using a forward reinforcement learning fine-tuning method.

[0014] Secondly, for some intervention samples and complete takeover of samples The merging process yields a labeled sample library. ;

[0015] Supervised learning fine-tuning is viewed as a regression task, with the raw information collected in partial intervention and full takeover scenarios serving as input. The driver's behavior at the corresponding moment is used as the real label. ,in Supervised learning fine-tunes the end-to-end autonomous driving strategy. After the labeled sample set is collected, the autonomous driving strategy parameters are copied. Then, it is fine-tuned, and the original policy parameters are replaced after training. The goal of fine-tuning is to minimize the predicted labels. With real labels Differences;

[0016] Finally, during the online fine-tuning evolution process, a dynamic sample library is established, and the memory samples are dynamically updated according to the degree of difference between the samples. The memory sample loss is then calculated by weighting the difference between the memory samples and the samples currently used for fine-tuning training. The memory sample loss is used to apply soft constraints to reinforcement learning fine-tuning and supervised learning fine-tuning.

[0017] Furthermore, the performance defect samples are extracted from the normal operation scenarios of the autonomous driving system. The value of the samples is estimated using a human-like value model. When the action value of the driving data is lower than a threshold, it is identified as having a performance defect, and the autonomous driving system records the vehicle state information. and action information As a component of the sample, it is represented as:

[0018]

[0019] in, This is a performance defect sample library. For the value of the action, The action value threshold for determining whether a sample is a performance defect sample;

[0020] The physiological and psychological feedback samples are extracted from scenarios where the autonomous driving system causes physiological and psychological discomfort to the user. Data collection and analysis of human physiological states are performed using data acquisition devices, with the driver's physiological state used as an additional attribute of the sample, represented as follows:

[0021]

[0022] in, For physiological and psychological feedback sample database, For physiological and psychological feedback information, The physiological and psychological feedback levels, The threshold for determining the physiological and psychological feedback level of a sample;

[0023] The partial intervention samples and complete takeover samples were collected in scenarios where the user partially intervened in the autonomous driving system and scenarios where the user completely took over driving, respectively. When the human driver was dissatisfied with the driving performance of the intelligent driving system in a specific scenario, the human driver actively acquired partial or full control of the vehicle, and the autonomous driving system recorded the vehicle state information during this process. and user demonstration data , represented as:

[0024]

[0025]

[0026] in, , These are a partial intervention sample library and a complete takeover sample library, respectively. , These represent the flags triggered by the driver's intervention and takeover actions, respectively.

[0027] Furthermore, in the aforementioned reinforcement learning fine-tuning method, from the unlabeled sample library... Extracting a single transfer tuple , as the sample data structure required for reinforcement learning updates, in , These represent the current state and the next state, respectively. The actions that the autonomous driving strategy takes based on the current state. The immediate reward provided by the environment during this state transition;

[0028] Time step status The raw information collected for the autonomous driving system includes image information from cameras, point cloud information from LiDAR, positioning information from the positioning system, and vehicle kinematics information from the inertial measurement unit; time step status Writing in key-value pair form:

[0029]

[0030] in, , These represent the number of cameras and LiDAR sensors, respectively.

[0031] Time step Action output The direct control quantities in the lateral and longitudinal directions of intelligent vehicles, namely the accelerator pedal opening. Brake pedal opening and steering wheel angle Time step status writing:

[0032]

[0033] Rewards for time step t The weighted average of several sub-reward items is expressed as follows:

[0034]

[0035] in, For sub-reward vectors, This is the weight matrix of the sub-reward vector.

[0036] The established rewards include goal-oriented rewards and evaluative rewards. Goal-oriented rewards aim to guide the vehicle according to navigation and rules, while evaluative rewards aim to improve the safety and comfort of the vehicle and other road users. Sub-rewards... Specifically defined as:

[0037]

[0038] in, As a displacement reward, To maintain the reward for speed, As a security reward, As an aggressive reward, For comfort rewards.

[0039] The displacement reward The reward for driving in the center line of the lane is set to be a positive value, which is a targeted reward and is defined as follows:

[0040]

[0041] in, This represents the longitudinal displacement of the vehicle at a single sampling time step. The lateral keeping coefficient is used to penalize driving behavior after the vehicle deviates from the center of the lane. This is the directional coefficient, which is penalized when a vehicle travels in the opposite direction.

[0042] The speed maintenance reward The system is designed to reward vehicles for maintaining a relatively high speed in the correct direction; this is a targeted reward and is defined as...

[0043]

[0044] in, The current speed of the vehicle. The maximum speed allowed in the scenario;

[0045] The security reward This study characterizes the impact of traffic participant behavior on vehicle safety, assesses vehicle driving safety using a risk field, and measures the relative risk field strength of surrounding traffic participants to the vehicle. Defined as:

[0046]

[0047] in, This refers to traffic participants surrounding the vehicle, specifically those in front of, behind, to the left, to the right, to the left front, to the left rear, to the right front, and to the right rear. ; For traffic participants Vehicles and traffic participants in the Frenet coordinate system Distance vector between them; This is the distance scaling factor. and For traffic participants Acceleration along its direction of travel and normal acceleration; The peak field strength of the risk field occurs among traffic participants. The location of the center of mass; and They are respectively With traffic participants The angle between the direction of travel and the normal direction; and For the forward and normal acceleration coefficients; m j For traffic participants The quality;

[0048] The driving safety reward function is constructed as follows:

[0049]

[0050] in, The threshold field strength at which a driver will not respond to a risk.

[0051] The aggressive reward This characterizes the impact of a vehicle's behavior on the safety of surrounding traffic participants, and defines aggressive rewards as:

[0052]

[0053] in, The risk level posed by a vehicle to other road users is high.

[0054] The comfort reward The comfort reward, representing the driver's need for driving comfort, is defined as:

[0055]

[0056] in, The longitudinal abruptness of the vehicle.

[0057] Weight matrix of sub-reward vector The solution is obtained using inverse reinforcement learning. Specifically, the method involves first extracting state-action pairs from the human driving process and then concatenating them to form a teaching trajectory. A teaching sample set is formed by summarizing several teaching trajectories. Secondly, the trajectory generated by the autonomous driving strategy is aligned with the expert teaching trajectory to maximize the cumulative reward of the expert teaching trajectory, and the weight matrix of the sub-reward vector is updated. Define the loss function for this process. for:

[0058]

[0059] in, The number of trajectories in a single teaching training set; These are the parameters of the agent network; Sub-reward vector weight matrix The trajectory of the trained agent and the trajectory shown by the expert The probability of consistency.

[0060] Finally, based on the loss function established above, the weight matrix of the sub-reward vector is updated using the gradient descent method. The positive reinforcement learning process is completed based on the new reward function, and the agent network parameters are updated; this positive reinforcement learning process is repeated continuously until... After convergence, the construction of the human-like reward function and the human-like action value estimation network is completed.

[0061] Furthermore, the aforementioned positive reinforcement learning fine-tuning method, after determining the composition of the transition tuples, employs a deep deterministic policy gradient reinforcement learning algorithm, designing a real policy network and a real action value estimation network, each with a corresponding target network, totaling four networks. During training, the parameters of each neural network are first initialized, with the parameters of the real policy network and the target policy network corresponding to... , The parameters of the real action value estimation network and the target action value estimation network correspond to respectively , The policy network parameters follow those of the pre-trained end-to-end autonomous driving policy network, while the action value estimation network parameters follow those of the action value estimation network trained through inverse reinforcement learning. Secondly, the real-world policy network... Based on the current state Output the action at the current moment. Real-world action value estimation network Based on the current state and actions Output action value estimate Target action value estimation The network depends on the state in the next moment. and target policy network Estimated next moment action Output the estimated value of the action at the next moment. Based on the estimated value of the action at the next moment. and the immediate reward at the current moment Calculate the value of the target action :

[0062]

[0063] in, This represents the discount rate.

[0064] The action value estimation network used in the fine-tuning process adopts the same network established during the inverse reinforcement learning process. The update process of the real-world action value estimation network minimizes the temporal difference error, which represents the gap between the action value calculated by the real-world network and the target action value. The parameter optimization method is gradient descent, and the final loss function is... The calculation formula is as follows:

[0065]

[0066] in, The number of unlabeled samples used for fine-tuning in reinforcement learning; This is the estimated value of the current action.

[0067] The policy network used in the fine-tuning process follows the autonomous driving policy. After sampling the unlabeled sample set, the autonomous driving policy is copied and fine-tuned, and then replaced with the original policy after training. During training, the policy network selects the action with the highest action value at the corresponding time and uses the target action value estimated by the target action value estimation network to guide the policy network update. The parameter optimization method uses gradient ascent, and the objective function is... as follows:

[0068]

[0069] in, For real-world policy networks, based on state The re-output action. To reduce fluctuations in the target action value output during training, it is necessary to limit the update amplitude and frequency of the target network; for the update of the target network, a soft update method is adopted, introducing a soft update coefficient. During each update, the new parameters are not used entirely; instead, a weighted average of the old and new parameters is used, known as the soft update coefficient. The magnitude of the soft update is determined by the following formula:

[0070]

[0071] Furthermore, in the supervised learning fine-tuning method, the predicted label... Generated by the policy network:

[0072]

[0073] in, The policy network is based on vehicle status information The generated predicted action distribution.

[0074] To reduce the predicted label With real labels Differences, design supervised learning to fine-tune the loss function for:

[0075]

[0076] in, This represents the number of unlabeled samples used for supervised fine-tuning.

[0077] Furthermore, to ensure memory stability during strategy evolution, a memory constraint method is established, the process of which includes the following three parts:

[0078] (1) Structuring of sample stream data;

[0079] The sample data stored in the memory bank is divided into two categories: unlabeled samples and labeled samples. Unlabeled samples include normal autonomous driving samples, performance defect samples, and physiological and psychological feedback samples. Labeled samples include partially intervened samples and completely taken over samples. For unlabeled samples, the raw data is recorded. and the actions generated by the strategy For labeled samples, record the raw data. and human actions ;

[0080] (2) Establishment of dynamic memory;

[0081] To ensure the stability of the autonomous driving strategy with old data, a dynamic memory is established, and the old data in the memory is used to constrain the update of the strategy. During the training process, the old data is classified based on the differences in state distribution, and different spaces are allocated to different types of old data, thus forming the dynamic memory. The specific process is as follows:

[0082] The conditional Kullback-Leibler divergence is used to represent the difference between distributions. The conditional Kullback-Leibler divergence between two sample flow distributions... Defined as:

[0083]

[0084] in, This is the original input; For strategy output; , This represents the different sample flow distributions fitted using a Gaussian mixture model; For the first sample stream, the original input The probability distribution; , This represents the conditional probability of the autonomous driving strategy or the human action output based on the original input conditions in different types of sample streams. and These are the integral infinitesimals representing the original input and the policy output, respectively.

[0085] Monte Carlo sampling was used to calculate the conditional KL divergence and the conditional Kullback-Leibler divergence. writing:

[0086]

[0087] in, and These represent the number of samples for the original input and the policy output in the sample stream, respectively.

[0088] In the early stages of the dynamic memory process, when all conditional Kullback-Leibler divergences exceed the threshold, a new sample stream will be stored as a new type of memory sample. In the later stages of the dynamic memory process, the number of memory sample types stored in the memory bank is limited to a certain limit. When the number of sample types exceeds the limit, a new sample stream will replace the set of memory samples with the smallest difference.

[0089] (3) Soft constraints on memory samples;

[0090] Update using different types of memory samples within the dynamic memory bank; when fine-tuning the strategy using the current sample stream, define the loss function using the dynamic memory samples; the loss function of the memory samples. The definition is as follows:

[0091]

[0092] in, The number of sample types in the dynamic memory; These are the parameters for the autonomous driving policy network; The loss function for the current batch of samples used for training, in reinforcement learning fine-tuning. In supervised learning fine-tuning , The action value of the sample The difference between strategic actions and expert actions. This represents the number of samples in a training batch. For the first The weights for calculating the loss for each type of sample set are defined as follows:

[0093]

[0094] in, For the current sample stream and the first Conditional Kullback-Leibler divergence of samples of each class.

[0095] The loss function of the memorized samples is used as a soft constraint during fine-tuning, and a comprehensive loss function is constructed using the loss functions of the current samples and the memorized samples. :

[0096]

[0097] in, Different weight values ​​are used for the weighting of memory sample loss in the two evolutionary approaches of reinforcement fine-tuning and supervised fine-tuning.

[0098] The beneficial effects of this invention are:

[0099] 1. Compared with traditional strategy evolution methods, this invention focuses on analyzing the diversity of user feedback data when using autonomous driving systems. Based on driving scenarios and driver feedback levels, vehicle data samples are divided into performance defect samples, physiological and psychological feedback samples, partial intervention samples, and complete takeover samples, and targeted fine-tuning is carried out to effectively improve the efficiency of data utilization.

[0100] 2. Compared with offline fine-tuning methods, this invention establishes a dynamic memory bank and uses old sample constraint strategies to ensure the stability of the strategy memory, thereby realizing online updates of the strategy and greatly improving the flexibility of the self-evolution of autonomous driving strategies.

[0101] 3. Compared with the supervised fine-tuning method, the present invention can flexibly adjust between the reinforcement fine-tuning method and the supervised fine-tuning method based on whether there are labels, which can further improve the performance ceiling of the strategy based on suboptimal driving data;

[0102] 4. Compared with reinforcement fine-tuning methods, this invention uses inverse reinforcement learning to establish a reward model and action value evaluation mechanism, effectively identifying defective samples and accurately establishing the evolutionary direction of the strategy based on the driver's real needs. Attached Figure Description

[0103] Figure 1 This is a schematic diagram of the overall process of the autonomous driving strategy evolution method of the present invention, which combines supervised fine-tuning and enhanced fine-tuning.

[0104] Figure 2 This is a schematic diagram illustrating the hierarchical structure of feedback data samples from users of the autonomous driving system according to the present invention.

[0105] Figure 3 This is a schematic diagram showing the locations of traffic participants in this invention;

[0106] Figure 4 This is a schematic diagram of the reverse reinforcement learning process of the present invention;

[0107] Figure 5 This is a schematic diagram of the reinforcement learning fine-tuning process of the present invention;

[0108] Figure 6 This is a schematic diagram of the memory sample constraint process of the present invention. Detailed Implementation

[0109] like Figure 1 As shown, this invention provides an end-to-end autonomous driving policy evolution method based on supervised-reinforcement joint fine-tuning. It employs two policy evolution methods—reinforcement learning fine-tuning and supervised learning fine-tuning—based on the characteristics of samples at different feedback levels, and applies soft constraints to both methods. The method includes the following steps:

[0110] First, regarding performance defect samples and physiological and psychological feedback samples The driver's dissatisfaction with the autonomous driving strategy output is low and they believe there is no need to take over the vehicle, therefore no explicit takeover behavior is generated as a truth value label. Thus, this invention identifies performance defect samples. and physiological and psychological feedback samples Merging to obtain an unlabeled sample library The autonomous driving strategy still has room for optimization at this point. To further improve the kinematic performance of the autonomous driving strategy, a reinforcement learning fine-tuning method is used to achieve end-to-end evolution of the autonomous driving strategy. The reinforcement learning fine-tuning method includes the following steps:

[0111] A human-like reward function is constructed through inverse reinforcement learning, and an action value estimation network model is pre-trained. The human-like reward function is dynamically generated by the human-like reward model constructed through inverse reinforcement learning, ensuring that the policy optimization direction aligns with the driver's cognition and habits. The inverse reinforcement learning process includes an alternating iterative process of inverse reinforcement learning and forward reinforcement learning, which takes place in a virtual environment. The action value estimation network model is used to determine the degree of defect in autonomous driving samples and filter out performance defective samples, and also serves as the initial action value estimation network for reinforcement learning. The autonomous driving policy and the action value estimation network model are then fine-tuned using a forward reinforcement learning fine-tuning method.

[0112] Secondly, for some intervention samples and complete takeover of samples When drivers are highly dissatisfied with the output of the autonomous driving strategy and take over actions, user demonstration data is generated as ground truth labels. Therefore, this invention uses some intervention samples... and complete takeover of samples The merging results in a labeled sample library. At this point, the autonomous driving strategy deviates significantly from the driver's cognition. To improve the human-likeness of the autonomous driving strategy, a supervised learning fine-tuning method is used to achieve the evolution of the end-to-end autonomous driving strategy.

[0113] Supervised learning fine-tuning is viewed as a regression task, with the raw information collected in partial intervention and full takeover scenarios serving as input. The driver's actions at the corresponding moment (accelerator / brake pedal opening and steering wheel angle) are the real labels. ,in Supervised learning fine-tunes the end-to-end autonomous driving strategy. After the labeled sample set is collected, the autonomous driving strategy parameters are copied. Then, it is fine-tuned, and the original policy parameters are replaced after training. The goal of fine-tuning is to minimize the predicted labels. With real labels Differences;

[0114] Among them, the design of the loss function focuses on the difference between the true value of human demonstration and the output of autonomous driving policy, so as to ensure that the output of autonomous driving policy can be closer to human driving behavior.

[0115] Finally, in order to ensure that fine-tuning training for new samples does not lead to performance degradation of the policy in other scenarios during the online fine-tuning evolution process, this invention establishes a dynamic sample library. The dynamic sample library dynamically updates the memory samples according to the degree of difference between the samples. Then, the memory sample loss is calculated by weighting the degree of difference between the memory samples and the samples currently used for fine-tuning training. The memory sample loss is used to apply soft constraints to reinforcement learning fine-tuning and supervised learning fine-tuning.

[0116] Furthermore, this invention categorizes user data samples from the autonomous driving system into performance defect samples, physiological and psychological feedback samples, partial intervention samples, and complete takeover samples based on feedback levels. Performance defect samples and physiological and psychological feedback samples are collected when the vehicle is in control of the driving, while partial intervention samples and complete takeover samples are collected when the human partially and completely takes over driving, respectively. Figure 2 As shown, the details are as follows:

[0117] The performance defect samples are extracted from normal operating scenarios of the autonomous driving system. A human-like value model is used to estimate the value of the samples. When the action value of the driving data is below a threshold, it is considered to have a performance defect. The autonomous driving system records vehicle status information. and action information As a component of the sample, it is represented as:

[0118]

[0119] in, This is a performance defect sample library. For the value of the action, The action value threshold for determining whether a sample is a performance defect sample;

[0120] The physiological and psychological feedback samples are extracted from scenarios where the autonomous driving system causes physiological and psychological discomfort to the user. Human physiological states are collected and analyzed using devices such as eye trackers, 3D motion analyzers, electrocardiogram and electromyography analyzers, and facial expression analyzers. The driver's physiological state is then used as an additional attribute of the sample, represented as follows:

[0121]

[0122] in, For physiological and psychological feedback sample database, For physiological and psychological feedback information, The physiological and psychological feedback levels, The threshold for determining the physiological and psychological feedback level of a sample;

[0123] The partial intervention samples and complete takeover samples were collected in scenarios where the user partially intervened in the autonomous driving system and scenarios where the user completely took over driving, respectively. When the human driver was dissatisfied with the driving performance of the intelligent driving system in a specific scenario, the human driver actively acquired partial or full control of the vehicle, and the autonomous driving system recorded the vehicle state information during this process. and user demonstration data , represented as:

[0124]

[0125]

[0126] in, , These are a partial intervention sample library and a complete takeover sample library, respectively. , These represent the flags triggered by the driver's intervention and takeover actions, respectively.

[0127] Furthermore, in the aforementioned reinforcement learning fine-tuning method, from the unlabeled sample library... Extracting a single transfer tuple , as the sample data structure required for reinforcement learning updates, in , These represent the current state and the next state, respectively. The actions that the autonomous driving strategy takes based on the current state. The immediate reward provided by the environment during this state transition;

[0128] Time step status The raw information collected by the autonomous driving system includes image information from cameras, point cloud information from LiDAR, positioning information from the Global Positioning System (GPS), and vehicle kinematics information from the Inertial Measurement Unit (IMU), written in key-value pair format:

[0129]

[0130] in, , These represent the number of cameras and LiDAR sensors, respectively.

[0131] Time step Action output The direct control quantities in the lateral and longitudinal directions of intelligent vehicles, namely the accelerator pedal opening. Brake pedal opening and steering wheel angle Time step status writing:

[0132]

[0133] Rewards for time step t The weighted average of several sub-reward items is expressed as follows:

[0134]

[0135] Among them, among them, For sub-reward vectors, This is the weight matrix of the sub-reward vector. The intelligent driving task can be described as ensuring the safety and comfort of the vehicle and surrounding vehicles while driving according to navigation instructions and traffic rules. Therefore, the rewards set in this invention include goal-oriented rewards and evaluative rewards. The purpose of establishing goal-oriented rewards is to ensure the vehicle drives according to navigation and rules, while the purpose of establishing evaluative rewards is to improve the safety and comfort of the vehicle and other traffic participants. Sub-rewards Specifically defined as:

[0136]

[0137] in, As a displacement reward, To maintain the reward for speed, As a security reward, As an aggressive reward, For comfort rewards.

[0138] The displacement reward The reward for driving in the center line of the lane is set to be a positive value, which is a targeted reward and is defined as follows:

[0139]

[0140] in, This represents the longitudinal displacement of the vehicle at a single sampling time step. The lateral keeping coefficient is used to penalize driving behavior after the vehicle deviates from the center of the lane. This is the directional coefficient, which is penalized when a vehicle travels in the opposite direction.

[0141] The speed maintenance reward The system is designed to reward a vehicle for maintaining a relatively high speed in the correct direction; this is a targeted reward and is defined as follows:

[0142]

[0143] in, The current speed of the vehicle. The maximum speed allowed in the scenario;

[0144] The security reward This study characterizes the impact of traffic participant behavior on vehicle safety, assesses vehicle driving safety using a risk field, and measures the relative risk field strength of surrounding traffic participants to the vehicle. Defined as:

[0145]

[0146] in, This refers to traffic participants surrounding the vehicle, specifically those in front of, behind, to the left, to the right, to the left front, to the left rear, to the right front, and to the right rear. ; For traffic participants Vehicles and traffic participants in the Frenet coordinate system Distance vector between them; This is the distance scaling factor. and For traffic participants Acceleration along its direction of travel and normal acceleration; The peak field strength of the risk field occurs among traffic participants. The location of the center of mass; and They are respectively With traffic participants The angle between the direction of travel and the normal direction; and For the forward and normal acceleration coefficients; m j For traffic participants The quality;

[0147] The driving safety reward function is constructed as follows:

[0148]

[0149] in, The threshold field strength at which a driver will not respond to a risk.

[0150] The aggressive reward This characterizes the impact of a vehicle's behavior on the safety of surrounding traffic participants. Aggressive rewards are defined as:

[0151]

[0152] in, The risk level posed by a vehicle to other road users is high.

[0153] The comfort reward The comfort reward, representing the driver's need for driving comfort, is defined as:

[0154]

[0155] in, The longitudinal abruptness of the vehicle.

[0156] Weight matrix of sub-reward vector The solution is obtained using inverse reinforcement learning. Specifically, the method involves first extracting state-action pairs from the human driving process and then concatenating them to form a teaching trajectory. A teaching sample set is formed by summarizing several teaching trajectories. Secondly, the trajectory generated by the autonomous driving strategy is aligned with the expert teaching trajectory to maximize the cumulative reward of the expert teaching trajectory, and the weight matrix of the sub-reward vector is updated. Define the loss function for this process. for:

[0157]

[0158] in, The number of trajectories in a single teaching training set; These are the parameters of the agent network; Sub-reward vector weight matrix The trajectory of the trained agent and the trajectory shown by the expert The probability of consistency.

[0159] Finally, based on the loss function established above, the weight matrix of the sub-reward vector is updated using the gradient descent method. The agent network parameters are updated by completing a positive reinforcement learning process based on the new reward function; this process is repeated continuously until... After convergence, the construction of the human-like reward function and the human-like action value estimation network is completed.

[0160] Furthermore, the aforementioned positive reinforcement learning fine-tuning method is as follows: Figure 5 As shown, after determining the composition of the transition tuple, the Deep Deterministic Policy Gradient (DDPG) reinforcement learning algorithm is adopted. DDPG is designed with one real policy network and one real action value estimation network, each of which has a target network, for a total of four networks. During training, the parameters of each neural network are first initialized. The parameters of the real policy network and the target policy network correspond to... , The parameters of the actual action value (Q-value) estimation network and the target action value (Q-value) estimation network correspond to respectively , The policy network parameters follow those of the pre-trained end-to-end autonomous driving policy network, while the action value estimation network parameters follow those of the action value estimation network trained through inverse reinforcement learning. Secondly, the real-world policy network... Based on the current state Output the action at the current moment. Real-world action value estimation network Based on the current state and actions Output action value estimate Target action value estimation The network depends on the state in the next moment. and target policy network Estimated next moment action Output the estimated value of the action at the next moment. Based on the estimated value of the action at the next moment. and the immediate reward at the current moment Calculate the value of the target action :

[0161]

[0162] in, This represents the discount rate.

[0163] The action value estimation network used in the fine-tuning process adopts the network established during inverse reinforcement learning. Through training on real-world performance defect samples and physiological / psychological feedback samples, this network becomes more applicable to the real world and better meets the needs of real drivers. The update process of the real-world action value estimation network minimizes the temporal difference error, which represents the difference between the Q-value calculated by the real network and the target Q-value. The parameter optimization method is gradient descent, and the final loss function... The calculation formula is as follows:

[0164]

[0165] in, The number of unlabeled samples used for fine-tuning in reinforcement learning; This is the estimated value of the current action.

[0166] The policy network used in the fine-tuning process follows the autonomous driving policy. After sampling the unlabeled sample set, the autonomous driving policy is copied and fine-tuned, and then replaced with the original policy after training. During training, the policy network selects the action with the highest Q-value at the corresponding time and uses the target Q-value estimated by the target action value estimation network to guide the policy network update. The parameter optimization method uses gradient ascent, and the objective function is... as follows:

[0167]

[0168] in, For real-world policy networks, based on state The re-output action. To reduce fluctuations in the target action value output during training, it is necessary to limit the update amplitude and frequency of the target network; for the update of the target network, a soft update method is adopted, introducing a soft update coefficient. During each update, the new parameters are not used entirely; instead, a weighted average of the old and new parameters is used, known as the soft update coefficient. The magnitude of the soft update is determined by the following formula:

[0169]

[0170] Furthermore, in the supervised learning fine-tuning method, the predicted label... Generated by the policy network:

[0171]

[0172] in, The policy network is based on vehicle status information The generated predicted action distribution.

[0173] To reduce the predicted label With real labels Differences, design supervised learning to fine-tune the loss function for:

[0174]

[0175] in, This represents the number of unlabeled samples used for supervised fine-tuning.

[0176] Furthermore, to ensure memory stability during strategy evolution, this invention establishes, as follows: Figure 6 The memory constraint module is shown. The workflow of the memory constraint module consists of the following three parts:

[0177] (1) Structuring of sample stream data;

[0178] The sample data stored in the memory bank is divided into two categories: unlabeled samples and labeled samples. Unlabeled samples include normal autonomous driving samples, performance defect samples, and physiological and psychological feedback samples. Labeled samples include partially intervened samples and completely taken over samples. For unlabeled samples, the raw data is recorded. and the actions generated by the strategy For labeled samples, record the raw data. and human actions ;

[0179] (2) Establishment of dynamic memory;

[0180] To ensure the stability of the strategy with respect to old data, a dynamic memory is established, using the old data in the memory to constrain the policy updates. During training, computer memory resources limit the capacity of the memory. To ensure that comprehensive operating condition information is contained within the limited memory space, this invention classifies the old data based on differences in state distribution and allocates different spaces to different types of old data, thus forming a dynamic memory. The specific process is as follows:

[0181] The conditional Kullback-Leibler (KL) divergence is used to represent the difference between distributions. The conditional KL divergence between two sample flow distributions... Defined as:

[0182]

[0183] in, This is the original input; For strategy output; , This represents the different sample flow distributions fitted using a Gaussian mixture model; For the first sample stream, the original input The probability distribution; , This represents the conditional probability of the autonomous driving strategy or the human action output based on the original input conditions in different types of sample streams. and These are the integral infinitesimals representing the original input and the policy output, respectively.

[0184] because , Since the KL divergence is a Gaussian mixture model, the calculation may not have an analytical solution. Therefore, Monte Carlo sampling is used to approximate the calculation of the conditional KL divergence. writing:

[0185]

[0186] in, and These represent the number of samples for the original input and the policy output in the sample stream, respectively.

[0187] In the early stages of the dynamic memory process, the dynamic memory bank has sufficient capacity. When a new sample stream differs significantly from all the memory sample types in the bank, i.e., when all conditional KL divergences exceed the threshold, the new sample stream will be stored as a new memory sample type. In the later stages of the dynamic memory process, the dynamic memory bank tends to be saturated. To ensure that each type of sample has a sufficient number of samples in a fixed sample storage space, the number of memory sample types stored in the bank is limited to a certain limit. When the number of sample types exceeds the limit, a new sample stream will replace the set of memory samples with the smallest difference.

[0188] (3) Soft constraints on memory samples;

[0189] This invention utilizes different types of memory samples within a dynamic memory bank to constrain the update strategy. When fine-tuning the strategy using the current sample stream, a loss function is defined using dynamic memory samples. Considering the varying similarities between different types of memory samples and the currently used training samples, higher weights are assigned to memory samples with significant differences. The loss function for memory samples... The definition is as follows:

[0190]

[0191] in, The number of sample types in the dynamic memory; These are the parameters for the autonomous driving policy network; The loss function for the current batch of samples used for training, in reinforcement learning fine-tuning. In supervised learning fine-tuning , The action value of the sample The difference between strategic actions and expert actions. This represents the number of samples in a training batch. For the first The weights for calculating the loss for each type of sample set are defined as follows:

[0192]

[0193] The loss function of the memorized samples is used as a soft constraint during fine-tuning, and a comprehensive loss function is constructed using the loss functions of the current samples and the memorized samples. :

[0194]

[0195] in, Different weight values ​​are used for the weighting of memory sample loss in the two evolutionary approaches of reinforcement fine-tuning and supervised fine-tuning.

[0196] To fully utilize multi-scenario data samples, this invention categorizes samples into performance defect samples, physiological and psychological feedback samples, partial intervention samples, and complete takeover samples based on the scenario. Specifically, to obtain performance defect samples, this invention utilizes a human-like value network to calculate the cumulative reward of the sample state under an end-to-end strategy, filtering data samples with low rewards in normal driving scenarios. To obtain physiological and psychological feedback samples, this invention introduces a series of driver behavior and physiological state collection and analysis devices to additionally collect driver physiological and psychological signals as additional rewards. Partial intervention samples and complete takeover samples are collected during human intervention and takeover of driving rights, respectively.

[0197] To establish a memory constraint on old data during policy fine-tuning, this invention establishes a dynamic memory bank, calculates the loss of the memory samples and uses it as an additional term in the policy network loss during reinforcement fine-tuning and supervised fine-tuning, thereby softly constraining policy evolution and maintaining the stability of memory of old data.

[0198] To enable the fine-tuning method to improve the upper limit of the autonomous driving strategy, this invention classifies performance defect samples and physiological and psychological feedback samples into unlabeled samples, and partially intervened samples and completely taken over samples into labeled samples. Based on whether they are labeled or not, the supervised fine-tuning method and the reinforcement fine-tuning method are called in a targeted manner. The reinforcement learning fine-tuning is used to get rid of the limitations of the quality of driver samples and further improve the upper limit of the strategy.

[0199] To make rewards more closely aligned with driver needs, this invention employs inverse reinforcement learning to establish a human-like reward model for evaluating unlabeled samples. Specifically, the physiological and psychological feedback samples use human physiological and psychological feedback levels as additional rewards.

Claims

1. An end-to-end autonomous driving policy evolution method with supervised-reinforcement joint fine-tuning, characterized in that: Includes the following steps: First, regarding performance defect samples and physiological and psychological feedback samples The unlabeled sample library was obtained by merging the samples. ; A human-like reward function is constructed through inverse reinforcement learning, and an action value estimation network model is pre-trained. The human-like reward function is dynamically generated by the human-like reward model constructed through inverse reinforcement learning, ensuring that the policy optimization direction aligns with the driver's cognition and habits. The inverse reinforcement learning process includes an alternating iterative process of inverse reinforcement learning and forward reinforcement learning, which takes place in a virtual environment. The action value estimation network model is used to determine the degree of defect in autonomous driving samples and filter out performance defective samples, and also serves as the initial action value estimation network for reinforcement learning. The autonomous driving policy and the action value estimation network model are then fine-tuned using a forward reinforcement learning fine-tuning method. Secondly, for some intervention samples and complete takeover of samples The merging process yields a labeled sample library. ; Supervised learning fine-tuning is viewed as a regression task, with the raw information collected in partial intervention and full takeover scenarios serving as input. The driver's behavior at the corresponding moment is used as the real label. ,in Supervised learning fine-tunes the end-to-end autonomous driving strategy. After the labeled sample set is collected, the autonomous driving strategy parameters are copied. Then, it is fine-tuned, and the original policy parameters are replaced after training. The goal of fine-tuning is to minimize the predicted labels. With real labels Differences; Finally, during the online fine-tuning evolution process, a dynamic sample library is established, and the memory samples are dynamically updated according to the degree of difference between the samples. The memory sample loss is then calculated by weighting the degree of difference between the memory samples and the samples currently used for fine-tuning training. The memory sample loss is used to apply soft constraints to reinforcement learning fine-tuning and supervised learning fine-tuning. To ensure the stability of memory during strategy evolution, a memory constraint method is established, and the process includes the following three parts: (1) Structuring of sample stream data; The sample data stored in the memory bank is divided into two categories: unlabeled samples and labeled samples. Unlabeled samples include normal autonomous driving samples, performance defect samples, and physiological and psychological feedback samples. Labeled samples include partially intervened samples and completely taken over samples. For unlabeled samples, record the raw data. and the actions generated by the strategy For labeled samples, record the raw data. and human actions ; (2) Establishment of dynamic memory; To ensure the stability of the autonomous driving strategy with old data, a dynamic memory is established, and the old data in the memory is used to constrain the update of the strategy. During the training process, the old data is classified based on the differences in state distribution, and different spaces are allocated to different types of old data, thus forming the dynamic memory. The specific process is as follows: The conditional Kullback-Leibler divergence is used to represent the difference between distributions. The conditional Kullback-Leibler divergence between two sample flow distributions... Defined as: ； in, This is the original input; For strategy output; , This represents the different sample flow distributions fitted using a Gaussian mixture model; For the first sample stream, the original input The probability distribution; , This represents the conditional probability of the autonomous driving strategy or the human action output based on the original input conditions in different types of sample streams. and These are the integral infinitesimals of the original input and the policy output, respectively; Monte Carlo sampling was used to calculate the conditional KL divergence and the conditional Kullback-Leibler divergence. writing: ； in, and These represent the number of samples for the original input and the policy output in the sample stream, respectively. In the early stages of the dynamic memory process, when all conditional Kullback-Leibler divergences exceed the threshold, a new sample stream will be stored as a new type of memory sample. In the later stages of the dynamic memory process, the number of memory sample types stored in the memory bank is limited to a certain limit. When the number of sample types exceeds the limit, a new sample stream will replace the set of memory samples with the smallest difference. (3) Soft constraints on memory samples; Update using different types of memory samples within the dynamic memory bank; when fine-tuning the strategy using the current sample stream, define the loss function using the dynamic memory samples; the loss function of the memory samples. The definition is as follows: ； in, The number of sample types in the dynamic memory; These are the parameters for the autonomous driving policy network; The loss function for the current batch of samples used for training, in reinforcement learning fine-tuning. In supervised learning fine-tuning , The action value of the sample The difference between strategic actions and expert actions. The number of samples in a training batch; For the first The weights for calculating the loss for each type of sample set are defined as follows: ； in, For the current sample stream and the first Conditional Kullback-Leibler divergence of samples of each class; The loss function of the memorized samples is used as a soft constraint during fine-tuning, and a comprehensive loss function is constructed using the loss functions of the current samples and the memorized samples. : ； in, Different weight values ​​are used for the weighting of memory sample loss in the two evolutionary approaches of reinforcement fine-tuning and supervised fine-tuning.

2. The end-to-end autonomous driving strategy evolution method with supervised-reinforcement joint fine-tuning according to claim 1, characterized in that: The performance defect samples are extracted from normal operating scenarios of the autonomous driving system. A human-like value model is used to estimate the value of the samples. When the action value of the driving data is below a threshold, it is considered to have a performance defect. The autonomous driving system records vehicle status information. and action information As a component of the sample, it is represented as: ； in, This is a performance defect sample library. For the value of the action, The action value threshold for determining whether a sample is a performance defect sample; The physiological and psychological feedback samples are extracted from scenarios where the autonomous driving system causes physiological and psychological discomfort to the user. Data collection and analysis of human physiological states are performed using data acquisition devices, with the driver's physiological state used as an additional attribute of the sample, represented as follows: ； in, For physiological and psychological feedback sample database, For physiological and psychological feedback information, The physiological and psychological feedback levels, The threshold for determining the physiological and psychological feedback level of a sample; The partial intervention samples and complete takeover samples were collected in scenarios where the user partially intervened in the autonomous driving system and scenarios where the user completely took over driving, respectively. When the human driver was dissatisfied with the driving performance of the intelligent driving system in a specific scenario, the human driver actively acquired partial or full control of the vehicle, and the autonomous driving system recorded the vehicle state information during this process. and user demonstration data , represented as: ； ； in, , These are a partial intervention sample library and a complete takeover sample library, respectively. , These represent the flags triggered by the driver's intervention and takeover actions, respectively.

3. The end-to-end autonomous driving strategy evolution method with supervised-reinforcement joint fine-tuning according to claim 1, characterized in that: In the aforementioned reinforcement learning fine-tuning method, from the unlabeled sample library... Extracting a single transfer tuple As the sample data structure required for reinforcement learning updates, , These represent the current state and the next state, respectively. The actions that the autonomous driving strategy takes based on the current state. The immediate reward provided by the environment during this state transition; Time step status The raw information collected for the autonomous driving system includes image information from cameras, point cloud information from LiDAR, positioning information from the positioning system, and vehicle kinematics information from the inertial measurement unit; time step status Writing in key-value pair form: ； in, , These represent the number of cameras and LiDAR sensors, respectively. Time step Action output For intelligent vehicles, the direct control quantities in the lateral and longitudinal directions are the accelerator pedal opening. Brake pedal opening and steering wheel angle Time step status writing: ； Rewards for time step t The weighted average of several sub-reward items is expressed as follows: ； in, For sub-reward vectors, This is the weight matrix of the sub-reward vector; The established rewards include goal-oriented rewards and evaluative rewards. Goal-oriented rewards aim to guide the vehicle according to navigation and rules, while evaluative rewards aim to improve the safety and comfort of the vehicle and other road users. Sub-rewards... Specifically defined as: ； in, As a displacement reward, To maintain the reward for speed, As a security reward, As an aggressive reward, For comfort rewards.

4. The end-to-end autonomous driving strategy evolution method of supervised-reinforcement joint fine-tuning according to claim 3, characterized in that: The displacement reward The reward for driving in the center line of the lane is set to be a positive value, which is a targeted reward and is defined as follows: ； in, This represents the longitudinal displacement of the vehicle at a single sampling time step. The lateral keeping coefficient is used to penalize the driver's behavior after the vehicle deviates from the center of the lane. This is the directional coefficient, which is penalized when a vehicle travels in the opposite direction. The speed maintenance reward The system is designed to reward a vehicle for maintaining a relatively high speed in the correct direction; this is a targeted reward and is defined as follows: ； in, The current speed of the vehicle. The maximum speed allowed in the scenario; The security reward This study characterizes the impact of traffic participant behavior on vehicle safety, assesses vehicle driving safety using a risk field, and measures the relative risk field strength of surrounding traffic participants to the vehicle. Defined as: ； in, For traffic participants Vehicles and traffic participants in the Frenet coordinate system Distance vector between them; This is the distance scaling factor. and For traffic participants Acceleration along its direction of travel and normal acceleration; The peak field strength of the risk field occurs among traffic participants. The location of the center of mass; and They are respectively With traffic participants The angle between the direction of travel and the normal direction; and For the forward and normal acceleration coefficients; m j For traffic participants The quality; The driving safety reward function is constructed as follows: ； in, The threshold field strength at which a driver will not respond to a risk. This refers to traffic participants surrounding the vehicle, specifically those in front of, behind, to the left, to the right, to the left front, to the left rear, to the right front, and to the right rear. ; The aggressive reward This characterizes the impact of a vehicle's behavior on the safety of surrounding traffic participants; the aggressive reward is defined as: ； in, The risk level posed by a vehicle to other road users is high. The comfort reward The comfort reward, representing the driver's need for driving comfort, is defined as: ； in, The longitudinal abruptness of the vehicle.

5. The end-to-end autonomous driving strategy evolution method of supervised-reinforcement joint fine-tuning according to claim 3 or 4, characterized in that: Weight matrix of sub-reward vector The solution is obtained using inverse reinforcement learning. Specifically, the method involves first extracting state-action pairs from the human driving process and then concatenating them to form a teaching trajectory. A teaching sample set is formed by summarizing several teaching trajectories. Secondly, the trajectory generated by the autonomous driving strategy is aligned with the expert teaching trajectory to maximize the cumulative reward of the expert teaching trajectory, and the weight matrix of the sub-reward vector is updated. Define the loss function for this process. for: ； in, The number of trajectories in a single teaching training set; These are the parameters of the agent network; Sub-reward vector weight matrix The trajectory of the trained agent and the trajectory shown by the expert The probability of consistency; Finally, based on the established loss function, the weight matrix of the sub-reward vector is updated using the gradient descent method. The positive reinforcement learning process is completed based on the new reward function, and the agent network parameters are updated; the positive reinforcement learning process is repeated continuously until... After convergence, the construction of the human-like reward function and the human-like action value estimation network is completed.

6. The end-to-end autonomous driving strategy evolution method of supervised-reinforcement joint fine-tuning according to claim 1, characterized in that: The aforementioned positive reinforcement learning fine-tuning method, after determining the composition of the transition tuples, employs a deep deterministic policy gradient reinforcement learning algorithm. It designs a real policy network and a real action value estimation network, each with a corresponding target network, totaling four networks. During training, the parameters of each neural network are first initialized; the parameters of the real policy network and the target policy network correspond to... , The parameters of the real action value estimation network and the target action value estimation network correspond to respectively , The policy network parameters follow those of the pre-trained end-to-end autonomous driving policy network, while the action value estimation network parameters follow those of the action value estimation network trained through inverse reinforcement learning. Secondly, the real-world policy network... Based on the current state Output the action at the current moment. Real-world action value estimation network Based on the current state and actions Output action value estimate Target action value estimation The network depends on the state in the next moment. and target policy network Estimated next moment action Output the estimated value of the action at the next moment. Based on the estimated value of the action at the next moment. and the immediate reward at the current moment Calculate the value of the target action : ； in, Discount rate; The action value estimation network used in the fine-tuning process adopts the same network established during the inverse reinforcement learning process. The update process of the real-world action value estimation network minimizes the temporal difference error, which represents the gap between the action value calculated by the real-world network and the target action value. The parameter optimization method is gradient descent, and the final loss function is... The calculation formula is as follows: ； in, The number of unlabeled samples used for fine-tuning in reinforcement learning; This is the estimated value of the current action. The policy network used in the fine-tuning process follows the autonomous driving policy. After sampling the unlabeled sample set, the autonomous driving policy is copied and fine-tuned, and then replaced with the original policy after training. During training, the policy network selects the action with the highest action value at the corresponding time and uses the target action value estimated by the target action value estimation network to guide the policy network update. The parameter optimization method uses gradient ascent, and the objective function is... as follows: ； in, For real-world policy networks, based on state The action of re-outputting; To reduce fluctuations in the target action value output during training, it is necessary to limit the update amplitude and frequency of the target network. For the update of the target network, a soft update approach is adopted, introducing a soft update coefficient. During each update, the new parameters are not used entirely; instead, a weighted average of the old and new parameters is used, known as the soft update coefficient. The magnitude of the soft update is determined by the following formula: 。 7. The end-to-end autonomous driving strategy evolution method of supervised-reinforcement joint fine-tuning according to claim 1, characterized in that: In the supervised learning fine-tuning method described above, the predicted label is... Generated by the policy network: ； in, The policy network is based on vehicle status information The generated predicted action distribution; To reduce the predicted label With real labels Differences, design supervised learning to fine-tune the loss function for: ； in, This represents the number of unlabeled samples used for supervised fine-tuning.

Images