End-to-end automatic driving control system and device based on human preference reinforcement learning
By introducing human preference reinforcement learning and comprehensive reward functions, the problem of rigidity and reward design difficulties in traditional autonomous driving systems in complex environments is solved, and efficient and safe autonomous driving decisions consistent with human behavior are achieved.
Patent Information
- Application Number
- CN202411114237.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-14
AI Technical Summary
Traditional autonomous driving systems are rigid in complex and dynamically changing traffic environments and are difficult to adapt. In reinforcement learning, reward function design is difficult and inefficient, making it easy to have strategy failure problems.
Introduce human preference reinforcement learning, through pre-training reward function models and real-time adjustments, combined with PPO algorithms, use semantic segmentation multi-channel BEV images and yaw angular velocity to design comprehensive reward functions to ensure that decisions are consistent with human behavior and avoid reward hacking.
It significantly improves the adaptability and safety of the autonomous driving system, ensures that the behavioral consistency in complex environments is consistent with the driving preferences of human passengers, and enhances the generalization and decision-making capabilities of the model.
Smart Images

Figure CN119018181A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of transportation and relates to an end-to-end autonomous driving control system and equipment based on human preference reinforcement learning. Background Art
[0002] In the field of autonomous driving, achieving the vision of vehicles driving safely without human intervention is the focus of today's technological pursuit. Autonomous driving technology aims to enable vehicles to navigate safely and efficiently without human intervention through advanced perception, decision-making and control systems. Traditional autonomous driving systems usually rely on rule-based approaches that require complex program coding to preset rules for various traffic situations. However, these rule-based systems are often rigid and limited when dealing with unknown and complex scenarios, and have difficulty adapting to dynamically changing environments. In order to provide a more flexible and adaptable solution, reinforcement learning has been introduced to the field of autonomous driving. Reinforcement learning provides a more flexible and intelligent autonomous driving solution by interactively learning with the environment without the need for preset rules. However, traditional reinforcement learning usually requires manual design of state space and reward functions, which is extremely difficult and inefficient in complex autonomous driving scenarios.
[0003] End-to-end reinforcement learning has been introduced into the field of autonomous driving as a way to provide more flexible and adaptable solutions. End-to-end reinforcement learning integrates environmental perception, decision-making, and control execution by learning driving behavior directly from raw sensor input, avoiding complex manual feature selection and state design, and reducing error transmission in intermediate links such as path planning and trajectory tracking. In this way, end-to-end reinforcement learning improves the robustness and adaptability of the system and can more effectively cope with dynamically changing environments.
[0004] Therefore, in this context, the introduction of human preferences to train the reward function model abandons the over-reliance on abstract and theoretical reward function settings, and instead directly introduces human intuitive judgment and feedback. By providing humans with a sequence of two states and actions, human feedback on their preferences is obtained, and the reward function model is trained with the help of the preference prediction model and the cross-entropy loss function, making the learning process more consistent with the behavior and decision-making of human drivers. This strategy that integrates human intelligence not only promotes the development of reinforcement learning algorithms in the right direction, but also significantly reduces the possible erroneous driving modes that may occur during the solution process, such as strategies that exploit loopholes in the reward system to obtain high rewards. Therefore, this provides a higher level of intelligent decision-making and safety assurance for autonomous driving systems.
[0005] In summary, the present invention proposes a new control strategy of end-to-end reinforcement learning, which realizes autonomous driving under complex urban road conditions by combining end-to-end reinforcement learning with human preferences, providing a more efficient, safe and generalizable autonomous driving solution. Summary of the invention
[0006] The present invention provides an end-to-end autonomous driving control system and device based on human preference reinforcement learning. The reward function model with human preference is adopted to solve the reward hacking problem that is easy to occur in traditional reinforcement learning, and realize end-to-end autonomous driving aligned with human's real goals and values. By introducing human preference, the model can not only learn behaviors consistent with human decision-making, but also effectively avoid the problem of strategy failure caused by reward design defects. The introduction of human preference is a major innovation of the present invention, which ensures that the decisions made by the model in the actual driving process are consistent with the expectations and values of human drivers, and enhances the practical applicability and safety of the model. Based on the pre-collected data set as the input of the pre-trained reward function model, through the advantage of pre-training, the reward function model can obtain certain prior knowledge about the task before the learning starts, helping the model to converge faster; using human preference, the learning behavior is closer to human decision-making, avoiding reward hacking behavior; based on the real-time collected vehicle-side and road-side information, as the input of the main neural network of reinforcement learning, in the process of training, the neural network parameters of the reward function model are adjusted and optimized according to human preference at fixed time periods, which helps to deal with those special situations and edge cases that may not be fully considered in the pre-training process. While ensuring the efficiency of model training and the performance of the final model are more in line with human expectations and preferences, driving safety is ensured.
[0007] The end-to-end autonomous driving control system based on human preference reinforcement learning of the present invention includes three main contents: a pre-training part, a reward function learning part and an agent learning part.
[0008] For the pre-training part, the data collected in advance in the CARLA simulator is used as input, and the true value of the output of the existing reward function plus the yaw rate is used as the response ratio as the target value to pre-train the neural network model of the reward function. By using the pre-collected high-quality data for pre-training, the reward function model obtains certain prior knowledge about the task before learning begins. This strategy helps the model converge faster, improves the initial performance of the model, and reduces training time.
[0009] For the reward function learning part, the reward prediction value is used to obtain the cross entropy between the preference predicted by the reward model and the actual preference, and the reward distribution of different actions in this state is obtained, and the L2 regularization term is added, which can help reduce the complexity of the model and prevent overfitting by adding the sum of squares of weights as a penalty term, thereby improving the generalization ability of the model. Reward function learning introduces human preferences, and with the help of preference prediction models and cross entropy loss functions, the model's learning behavior is closer to human decision-making, thereby avoiding reward hacking. This step helps the model make decisions that are more in line with human expectations in complex environments and enhances the generalization ability of the model.
[0010] For the agent learning part, the PPO algorithm is used. In the CARLA simulator, the multi-channel BEV after semantic segmentation is used as input, and the throttle opening and angle in vector form are used as output for real-time training. The agent learning uses the PPO algorithm and is trained through real-time collected vehicle-side and road-side information. This part ensures that the performance of the model in actual operation is more in line with human expectations and preferences, while ensuring driving safety. Through continuous training and optimization, the agent learning part can effectively improve the decision-making ability and adaptability of the autonomous driving system.
[0011] Through the close integration of these three parts, the end-to-end autonomous driving control system of the present invention not only solves the reward hacking problem that is prone to occur in traditional reinforcement learning, but also enables the autonomous driving system to be closely aligned with human's real goals and values. This strategy significantly enhances the generalization ability of the autonomous driving model and ensures behavioral consistency and safety in complex traffic environments, which is highly consistent with the driving preferences of human passengers.
[0012] Part 1: Pre-training
[0013] For the neural network model of the reward function, the ROACH model is used to pre-collect the data set in the CARLA simulator and save it as a .h5 file, which provides the semantic information of perception and the state of the vehicle. Then, in order to help the neural network understand the input information, the features extracted by the convolution layer and the fully connected layer are used as the input of the neural network model. The original information of the input items includes: 1) multi-channel BEV after semantic segmentation; 2) throttle, steering, brake, and speed of the vehicle, where the value range of the steering control is defined as [-1,1], indicating the degree of left and right steering of the vehicle; and the value range of the throttle control and brake control are both set to [0,1], representing the force applied by the throttle and brake pedals, 0 represents no force, and 1 represents full force. ; 3) The true value of the reward r of the ROACH model b ; 4) The yaw rate of the vehicle, in degrees / s. The output is the predicted value of the reinforcement learning reward value V(s t ), which is used in the model to update the weights of the neural network.
[0014] Multi-channel BEV image after semantic segmentation bev It includes six channels, namely, the drivable area, the expected route, the lane boundary, the remaining vehicles in multiple time series, the pedestrians in multiple time series, the traffic lights and stop signs in multiple time series and their triggering areas. The measurement vector m re The ego vehicle states not represented in the BEV are included, including ground truth measurements of steering, throttle, brake, lateral and horizontal speed. For the BEV, six convolutional layers are used for encoding; for the measurement vector, two fully connected layers are used for encoding. The outputs of the two encoders are combined and processed through two more fully connected layers to produce the latent feature j re , and finally input it into the neural network model of the reward function with two fully connected hidden layers for training. The neural network model of the reward function consists of an input layer, two hidden layers, and an output layer. The input and output layers are both configured with a single unit, and the hidden layer is set to have 128 units. Among them, each hidden layer is followed by a layer of LeakyReLU activation function to increase the nonlinear expression ability; a linear activation function is used in the output layer. In order to take stability into account, a comprehensive reward function is proposed with reference to the response ratio function. The equation is as follows:
[0015]
[0016] Among them, α represents the weight factor, α∈(0,1); R t is the actual reward observed at time t, which is replaced by the real value of the existing ROACH model in this invention; is the dimensionless yaw rate of the ego vehicle at time t; ξ is a constant to avoid the denominator being equal to zero.
[0017] For the loss function, a strategy using temporal difference learning is proposed to achieve sequence sensitivity in the reward prediction model. This strategy not only takes into account an action and its immediate reward, but also integrates the predicted reward value of the subsequent state, allowing the model to understand the long-term impact of actions in the sequence. A sequence decay coefficient λ is introduced into the loss function, which is applicable to the loss calculation of the reward prediction model. Specifically, the reward prediction V(s) for each action taken at each step is t ) is updated as follows:
[0018] V(s t )←V(s t )+β[R' t +γV(s t+1 )-V(s t )]
[0019] Where V(s t ) is in state s at time tt The reward value prediction; β is the learning rate parameter used to adjust the step size; R′ t is the actual comprehensive reward observed at time t; γ is the discount factor for future rewards, a value between 0 and 1 that measures the importance of future rewards relative to immediate rewards; s t+1 Indicates the state at the next moment.
[0020] In order to better consider the order effect in the action sequence, the following generalized order dependency loss function is defined
[0021]
[0022] Here, represents the overall sequential dependency loss function, which considers future rewards at each moment and weights them by the coefficient λ, which means that the strategy gives higher weight to predictions closer to the present moment, reflecting sequential sensitivity; T is the total number of model training steps. The square term strengthens the accuracy of the prediction, ensuring that the model is more severely punished for large deviation predictions, and promoting more accurate long-term reward predictions.
[0023] For the weight update of the neural network, the back-propagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. In the neural network, the gradient measures the rate of change of each parameter based on the partial derivative of the loss function with respect to each parameter. These calculated gradients are used to drive the Adam optimizer to update the model parameters. The Adam optimizer, with its adaptive characteristics, helps to adjust the learning rate evenly at various learning stages, optimize the path of the loss function descent, and make the model converge to the optimal solution faster. Formulated in terms of weight update, the following rules are followed:
[0024]
[0025] Among them, w t is the weight at training step t, △w t is the weight update amount, m t is the first-order moment estimate, i.e., the exponential moving average of the gradient, v t is the second-order moment estimate, i.e., the exponential moving average of the squared gradient, η is the learning rate, and ε is a small constant introduced to increase numerical stability.
[0026] Part II: Reward Function Learning
[0027] After the pre-training of the reward function prediction model is completed, it is connected to the reinforcement learning model to replace the rule-based reward function part of the reinforcement learning model.
[0028] During the reinforcement learning strategy learning process, the current state and action are randomly sampled, and the reward function learning step is entered every K cycles. First, the algorithm is used to query human preferences M times. Based on the randomly sampled information, the input for generating human preferences is:
[0029] σ={s k ,a k ,s k+1 ,a k+1 ,...,s k+N ,a k+N}
[0030] σ is the state and action sequence randomly sampled during reinforcement learning training, s k ,a k They represent the state and action of the agent at time k, respectively. N is the sequence length. A pair of sequences is generated each time a human preference is asked:
[0031]
[0032] And humans express their preference y from it, y is the output of human preference, representing the distribution of which segment humans prefer:
[0033] y∈{(1,0),(0,1),(0.5,0.5)}
[0034] They respectively indicate that humans prefer the first sequence, the second sequence, and like the two sequences equally. This preference is expressed as a triple (σ 0 ,σ 1 ,y) is stored in data set I. In the present invention, in order to avoid the tedious human feedback process, the following equation is selected to replace human preference as the criterion:
[0035]
[0036] Among them, R' t is the real value of the comprehensive reward; μ is the discount factor, μ∈(0,1); △θ is the turning angle difference between the ego vehicle at time t and the previous moment, the purpose is to reduce unnecessary swing of the ego vehicle and improve ride comfort.
[0037] Select D t The high value is selected as the human feedback, and the human preference triplet (σ 0 ,σ 1 ,y).
[0038] The reward model is then updated using the stored preference samples First, establish a preference prediction model:
[0039]
[0040] In this paragraph, indicates the probability that humans prefer segment 1, Represents the fragment σ i Ratio fragment σ j More preferred. Intuitively, this can be interpreted as assuming that the probability of preferring a segment depends exponentially on the sum of an intrinsic reward function over the segment. is not a binary classifier, learning It is equivalent to binary classification under the label y provided by humans. Specifically, as a reward function of a neural network model with parameters, it is updated by minimizing the following cross entropy loss:
[0041]
[0042] Among them, y(i) represents human preference for the i-th sequence, E() represents expectation, (σ 0 ,σ 1 ,y)~I represents (σ 0 ,σ 1 ,y) is sampled from distribution I.
[0043] To ensure the training effect, in the loss function Then add L2 regularization, that is:
[0044]
[0045] Among them, τ is a regularization hyperparameter, which determines the penalty for complex models. In addition, the bi-norm of the weight W (that is, the square root of the sum of the squares of the weight elements) represents a measure of the complexity of the model. The larger the model weight, the more complex the model. This representation helps to control the complexity of the model, further prevent overfitting, and improve the generalization ability of the model.
[0046] After completing the update of the reward function neural network model, the data in the experience replay buffer of the reinforcement learning model is updated. Specifically, this involves using the updated reward model to recalculate the reward for each state transition in the buffer (i.e., state, action, and its corresponding subsequent state). This step ensures the synchronization and consistency of the experience replay to reflect the latest reward mechanism.
[0047] Part III: Agent Learning
[0048] The reinforcement learning model uses the ROACH model and the Proximal Policy Optimization (PPO) algorithm. bev As input; in addition, an additional measurement vector m is input rl, which includes the vehicle state not represented in the BEV, including direction, throttle, brake, transmission gear, lateral and longitudinal speed. The above raw information is then encoded through the same convolution and fully connected layers as in pre-training to help the reinforcement learning neural network understand it. The output is the predicted value of the vehicle's turning angle, throttle, and brake. The value range of steering control is defined as [-1,1], indicating the degree of left and right steering of the vehicle; and the value range of throttle control and brake control are both set to [0,1], representing the force applied by the throttle and brake pedals, 0 represents no force, and 1 represents full force.
[0049] During training, the approximate policy optimization (PPO) algorithm is used to train the model, in which the pruning technique is used to avoid excessive policy updates. Trajectories are collected during training, and each trajectory consists of a series of states (BEV images i bev and the measurement vector m rl ), action a and reward r, and termination event z, which marks the end of a trajectory. The termination event z includes the policy network and the value network, which are iteratively updated through the following objective function:
[0050]
[0051] in, The trajectory τ is obtained according to the strategy Generated, θ k+1 is the updated value of the policy parameter, the goal is to maximize the overall loss function described below, is the policy gradient loss after clipping, using the generalized advantage estimate to compute the advantage; is the maximum entropy loss, which is used to encourage exploration; is the exploration loss, guiding the policy to explore the distribution of “suggested” actions that prevent a particular terminal event z from occurring.
[0052] first, is the clipped policy gradient loss, which uses Generalized Advantage Estimation (GAE) to estimate the advantage. This loss function helps the policy network learn the most beneficial actions in the long run:
[0053]
[0054] π θ (a t ∣s t ) is the current strategy in state s t Next, perform action a t The probability of is the old policy in state st Next, perform action a t The probability of is the advantage function at time step t; clip(·,1-∈,1+∈) is a function that clips the ratio to limit the magnitude of the policy update; ∈ is a hyperparameter for clipping
[0055] The second objective function is That is, maximum entropy loss, which is a loss commonly used to promote exploration and avoid falling into local optimality. It is defined as follows:
[0056]
[0057] H(π θ (·|i bev ,m rl )) is the entropy of the strategy, which is used to quantify the randomness of the strategy; λ ent is a hyperparameter that controls the weight of the entropy loss.
[0058] By increasing the entropy of the action distribution, the action distribution is pushed toward a uniform distribution, because maximizing the entropy is equivalent to minimizing the KL divergence to a uniform distribution. u(-1,1) represents a uniform distribution on the interval [-1,1]:
[0059] H(π θ )=-KL(π θ ‖u(-1,1))
[0060] If two probability distributions have the same support set (i.e. they can produce the same set of outcomes), then maximizing the entropy (i.e. randomness) of one distribution is equivalent to minimizing its KL divergence to another uniform distribution. This generalized form is designed to encourage the algorithm to explore reasonable directions that comply with basic traffic rules. In other words, it not only encourages the algorithm to perform more and more random exploration behaviors, but also guides the algorithm to explore in directions that are more likely to comply with traffic rules. Such a design can both enhance the model's ability to explore new states and ensure that the behavior learned by the model complies with the rules of the real world. This form is named exploration loss and is defined as:
[0061]
[0062] Among them, "1" represents the indicator function, Represents an event, specifically an event that ends a sequence (episode). The event set Including collision, running red light / traffic sign violation, deviation from route and obstruction; KL(π θ (·|i bev,k ,m rl,k )‖p z ) is the strategy distribution and the predefined exploration prior p z The Kullback-Leibler divergence between exp is a hyperparameter that controls the weight of the exploration loss; is a hyperparameter that specifies the number of steps at the end of each episode. The algorithm applies a uniform prior to all moments regardless of whether event z is triggered or not. The algorithm then takes the last N z Step 2 adjusts the action towards the predefined exploration prior p z , which encodes a “recommendation” to prevent the triggered event z from happening again. In practice, we use N z =100 and If z is related to a collision or running a red light / sign, set the acceleration prior to to encourage Roach to slow down, while directional control is not affected. Conversely, if the car is stopped, use As a prior for acceleration. For course deviation, a uniform prior is applied to the directional control Although in this case this is equivalent to maximizing entropy, the exploration loss further encourages exploration of steering angles in the last 10 seconds before going off-course.
[0063] Beneficial effects of the present invention:
[0064] (1) The present invention provides an end-to-end autonomous driving control system based on human preference reinforcement learning. By implementing a reward prediction model that takes vehicle stability into consideration, the application of human preferences is introduced. Through the feedback of human preferences, with the help of the preference prediction model and the cross-entropy loss function, and combined with a carefully calibrated CARLA simulation data set, the present invention effectively overcomes the reward hacking problem commonly seen in traditional reinforcement learning processes. This mechanism ensures that the autonomous driving decision-making process is closely linked to the actual goals and values of human drivers, and achieves a natural alignment of the end-to-end autonomous driving system. Through this strategy, the generalization ability of the autonomous driving model is significantly enhanced, and the behavioral consistency and safety in complex traffic environments are ensured, maintaining a high degree of consistency with the driving preferences of human passengers.
[0065] (2) Introduction of multi-dimensional input mechanism: By integrating the bird's-eye view (BEV) based on semantic segmentation and auxiliary measurement vectors such as yaw rate, this model achieves a comprehensive understanding of the autonomous driving environment and vehicle status. Compared with traditional models, this multi-dimensional input mechanism significantly optimizes the autonomous driving system's ability to assess vehicle motion stability and safety, thereby improving the accuracy and reliability of autonomous navigation.
[0066] (3) Design of comprehensive reward function: The comprehensive reward function proposed in this paper innovatively optimizes the reward prediction mechanism of autonomous driving by integrating yaw rate and traditional reward indicators. This design not only simplifies the complex parameter setting problem, but also enhances the adaptability and generalization ability of the model in a changing environment. This comprehensive reward function makes the path planning of the autonomous driving system in an unknown environment more accurate, and effectively improves the stability control and safety performance of the vehicle in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 Schematic diagram of the end-to-end autonomous driving control system based on human preference reinforcement learning proposed in the present invention;
[0068] Figure 2 The structure diagram of the reward prediction model proposed by the present invention;
[0069] Figure 3 Schematic diagram of reinforcement learning training proposed by the present invention; DETAILED DESCRIPTION
[0070] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings, but the content of the present invention is not limited thereto.
[0071] The present invention provides an end-to-end autonomous driving control system based on human preference reinforcement learning, which adopts a reward function model with human preferences, such as Figure 1 As shown in the figure, the reward hacking problem that is prone to occur in traditional reinforcement learning is solved, and end-to-end autonomous driving that is aligned with human's real goals and values is achieved. The specific steps include:
[0072] (1) Enable the pre-training part of the reward prediction model, such as Figure 2 shown.
[0073] 1. Collection and processing of pre-training data: Use the ROACH model to perform pre-collection activities in the CARLA simulation environment to collect the required driving data, and then save it in .h5 file format. These data include six channels of semantic segmentation-based bird's-eye view (BEV) images, dynamic control information of the vehicle (including throttle, steering, brake, speed), and the actual reward value r corresponding to the ROACH model. bThe steering control uses the value range of [-1,1] to reflect the degree of left and right steering of the vehicle, while the control range of the accelerator and brake is set to [0,1] to represent the change in the pedal application force.
[0074] 2. Data encoding and feature extraction: The collected data items are processed using a specially designed neural network structure. For the BEV image data, six convolutional layers are used for encoding to obtain the encoded BEV image features. For the measurement vector data of the vehicle state, two layers of fully connected layers are used for encoding to extract features about the real-time state of the vehicle. Finally, the encoder outputs of these two parts are merged and further processed through two additional fully connected layers to produce the final latent feature vector j re .
[0075] 3. Construction and optimization of reward function model: By constructing an end-to-end neural network, the network includes an input layer, two fully connected hidden layers, and an output layer. The network design uses a unit output layer to predict the reward value. Each hidden layer is equipped with a LeakyReLU activation function to enhance nonlinear capabilities, while the output layer uses a linear activation function. Incorporating stability considerations into the reward function, the following comprehensive reward function is constructed to accurately evaluate and predict the reward value in autonomous driving, and further improve the stability and accuracy of the model:
[0076]
[0077] 4. Utilization of order-sensitive loss function: Using the time difference learning strategy, the order decay coefficient λ is introduced. This loss function not only focuses on the immediate return, but also considers the predicted reward V(s) of the future state. t ) to deepen the model's understanding of the long-term impact of the behavior sequence. The generalized sequential dependency loss function considers the entire action sequence, strengthens the accuracy of the prediction, ensures the penalty for high deviation predictions, and thus optimizes the accuracy of long-term reward predictions. The following loss function is constructed:
[0078]
[0079] 5. Optimization and update of model parameters: The back propagation algorithm is used in combination with the Adam optimizer to calculate the gradient based on the partial derivatives of the loss function with respect to each parameter to guide the update of the model parameters. The Adam optimizer uses its adaptive adjustment characteristics to balance the adjustment of the learning rate, promote the rapid decrease of the loss function and converge to the optimal solution. The weight update follows the following rules to ensure the robustness and efficiency of the model during the training process:
[0080]
[0081] (2) Enable the reinforcement learning training part, such as Figure 3 shown.
[0082] 1. Integration of pre-trained reward prediction models: Integrate pre-trained reward prediction models into reinforcement learning models to replace traditional rule-based reward functions, thereby improving the performance of the model in simulating complex decision-making environments.
[0083] 2. Environment setup and strategy initialization: define the state space and use the six-channel BEV image i bev and the measurement vector m rl As input, initialize the policy network and value network and prepare the Proximal Policy Optimization (PPO) algorithm.
[0084] 3. Data collection: Use the current strategy to perform actions in the environment and collect states (BEV images i bev and the measurement vector m rl ), action a and reward r, and a termination event z.
[0085] 4. Objective function construction: Construct the objective function θ k+1 , consists of three parts: the policy gradient loss after clipping Maximum Entropy Loss To promote exploration, and exploration loss To avoid unwanted abort events with a boot strategy:
[0086]
[0087] 5. Policy optimization: Apply the collected data to the objective function and use the gradient descent method to iteratively update the policy network. In this process, the PPO algorithm's pruning technique is used to control the magnitude of the policy update and maintain the stability of learning.
[0088] 6. Execution of periodic reward function learning: Set a periodic cycle, and execute reward function learning every K cycles. The algorithm automatically generates a state and action sequence σ = {s k ,a k ,s k+1 ,a k+1 ,...,s k+N ,a k+N} and simulates human preferences based on the following preset equation:
[0089]
[0090] 7. Take the comprehensive reward real value R' t , discount factor γ and behavior stability evaluation index constitute the optimization reward function:
[0091] 8. Establish a preference prediction model and update it: Use the collected preference data samples to establish a preference prediction model:
[0092]
[0093] 9. And iteratively update the reward function neural network model by minimizing the following loss function (including L2 regularization) to optimize the model training effect:
[0094]
[0095] 10. Synchronous update of the experience replay buffer: Using the updated reward model, a comprehensive update of the data in the experience replay buffer of the reinforcement learning model is performed to ensure the consistency of the data with the latest reward mechanism.
[0096] 11. Definition of the agent learning strategy: The Proximal Policy Optimization (PPO) algorithm is used to define the agent’s learning strategy by taking the semantically segmented six-channel BEV images and measurement vectors as input to improve the effectiveness and efficiency of training.
[0097] 12. Agent training process: During the agent training process, trajectory data consisting of states, actions, and rewards are collected, and the agent model is iteratively updated using specific objective functions (including policy gradient loss, entropy loss, and exploration loss) based on termination events.
[0098] 13. Application of objective function and exploration loss: Based on the PPO algorithm training, entropy loss is introduced to promote strategy exploration. At the same time, exploration loss is applied to guide the agent to avoid undesirable termination events. By adjusting the details of action preferences, the adaptability and behavioral compliance of the model under the rules of the real world are enhanced.
[0099] 14. Evaluation and iteration: Evaluate the performance of the updated strategy in the environment. Repeat steps 3 to 13 based on the evaluation results until the predetermined performance threshold is met or the maximum number of iterations is reached.
[0100] Based on the above control system, an embodiment of the present invention further proposes a control device for automatic driving, in which a program code of the control system is set.
[0101] In summary, the present invention provides an end-to-end autonomous driving control system and device based on human preference reinforcement learning. The reward function model with human preferences is adopted to solve the reward hacking problem that is easy to occur in traditional reinforcement learning, and realize end-to-end autonomous driving that is aligned with human's real goals and values. Based on the pre-collected data set as the input of the pre-trained reward function model, through the advantage of pre-training, the reward function model can obtain certain prior knowledge about the task before the learning begins, helping the model to converge faster; using human preferences, the learning behavior is closer to human decision-making, avoiding reward hacking behavior; based on the real-time collected vehicle-side and road-side information, as the input of the main neural network of reinforcement learning, during the training process, the neural network parameters of the reward function model are adjusted and optimized according to human preferences at fixed time periods, which helps to deal with those special situations and edge cases that may not be fully considered in the pre-training process. While ensuring that the model training efficiency and the performance of the final model are more in line with human expectations and preferences, the safety of driving is ensured.
[0102] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.
Claims
1. An end-to-end autonomous driving system based on human preference reinforcement learning, characterized in that: include: Pre-training part, reward function learning part and agent learning part; The pre-training part uses the pre-collected data as input, takes the true value of the reward function output plus the yaw angular velocity as the response ratio as the target value, and pre-trains the neural network model of the reward function; by using the pre-collected data for pre-training, the neural network model of the reward function can obtain certain prior knowledge about the task before learning; The reward function learning part uses the predicted value of the reward function neural network model to obtain the predicted preference of the reward function model, and performs cross entropy with the real preference, and obtains the reward distribution of different actions in this state and adds the L2 regularization term, which can help reduce the complexity of the model and prevent overfitting by adding the sum of squares of weights as a penalty term, thereby improving the generalization ability of the model. By introducing human preferences, the reward function learning makes the learning behavior of the model closer to human decision-making with the help of the preference prediction model and the cross entropy loss function; The intelligent agent learning part uses the multi-channel BEV after semantic segmentation as input and the throttle opening and steering angle in vector form as output for real-time training. The intelligent agent learning adopts the PPO algorithm and is trained through the vehicle-side and road-side information collected in real time.
2. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 1, characterized in that: In the pre-training part, the pre-collected data is collected by using the ROACH model in the CARLA simulator and saved as a .h5 file; Specifically include: 1) Multi-channel BEV after semantic segmentation; 2) The ego vehicle’s throttle, steering, brake, and speed. The steering control value range is defined as [-1, 1], indicating the degree of left and right steering of the vehicle. The throttle control and brake control value ranges are both set to [0, 1], representing the force applied by the throttle and brake pedals, with 0 representing no force and 1 representing full force. 3) The true value of the reward r of the ROACH model b ; 4) The yaw rate of the vehicle, in degrees per second, output as the predicted value of the reinforcement learning reward value V(s t ), which is used to update the weights of the neural network.
3. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 2, characterized in that: The multi-channel BEV image after semantic segmentation includes six channels, namely, a drivable area, an expected route, a lane boundary line, other vehicles in multiple time series, pedestrians in multiple time series, traffic lights and stop signs in multiple time series and their triggering areas; the measurement vector includes the vehicle state not represented in the BEV, including ground-truth measurement values of steering, throttle, brake, lateral and horizontal speeds; for the BEV, six convolutional layers are used for encoding processing, and for the measurement vector, two fully connected layers are used for encoding, the two encodings are merged, and processed through another two fully connected layers to generate potential features, and finally input into a neural network model with a reward function with two fully connected hidden layers for training.
4. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 3, characterized in that: The neural network model of the reward function consists of an input layer, two hidden layers and an output layer. The input layer and the output layer are both configured with a single unit, and the hidden layer is set to 128 units. Each hidden layer is followed by a layer of LeakyReLU activation function to increase nonlinear expression capabilities. A linear activation function is used in the output layer.
5. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 4, characterized in that: In order to take stability into account, a comprehensive reward function is proposed with reference to the response ratio function as follows: Among them, α represents the weight factor, α∈(0,1); R t is the actual reward observed at time t, which is replaced by the real value of the existing ROACH model in this patent; is the dimensionless yaw rate of the ego vehicle at time t; ξ is a constant to avoid the denominator being equal to zero.
6. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 4, characterized in that: For the loss function of the neural network model, a method using temporal difference learning is proposed to achieve sequential sensitivity in the reward prediction model, which not only takes into account an action and its immediate reward, but also integrates the predicted reward value of the subsequent state, thereby allowing the model to have some understanding of the long-term impact of actions in the sequence; a sequence decay coefficient λ is introduced into the loss function, which is applicable to the loss calculation of the reward function prediction model. Specifically, the reward prediction V(s t ) is updated as follows: V(s t )←V(s t )+β[R' t +γV(s t+1 )-V(s t )] Where V(s t ) is in state s at time t t Reward value prediction of β is the learning rate parameter, which is used to adjust the step size; R′ t is the actual comprehensive reward observed at time t; γ is the discount factor for future rewards, a value between 0 and 1 that measures the importance of future rewards relative to immediate rewards; s t+1 Indicates the state at the next moment. Considering the order effect in the action sequence, the generalized order dependency loss function is defined as follows Here, represents the overall sequential dependency loss function, which considers future rewards at each moment and weights them by the coefficient λ, thereby giving higher weights to predictions closer to the present moment, reflecting sequential sensitivity, while the square term strengthens the accuracy of the prediction, ensuring that the model is more severely penalized for predictions with large deviations, and promoting more accurate long-term reward predictions.
7. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 4, characterized in that: For the weight update of the neural network, the back-propagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. The gradient measures the rate of change of each parameter based on the partial derivative of the loss function with respect to each parameter. These calculated gradients are used to drive the Adam optimizer to update the model weights. The weight update follows the following rules: Among them, w t is the weight at the training step, △w t is the weight update amount, m t is the first-order moment estimate, i.e., the exponential moving average of the gradient, v t is the second-order moment estimate, i.e., the exponential moving average of the squared gradient, η is the learning rate, and ε is a small constant introduced to increase numerical stability.
8. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 1, characterized in that: The reward function learning part, after the pre-training of the reward function neural network model is completed, is connected to the reinforcement learning model to replace the rule-based reward function part in the reinforcement learning model; During the reinforcement learning model learning process, the current state and action are randomly sampled, and the reward function learning step is entered every K cycles; Specifically, First, we ask for human preferences M times, and generate the input of human preferences based on the randomly sampled information: s={s k ,a k ,s k+1 ,a k+1 ,...,s k+N ,a k+N } s is a sequence of states and actions randomly sampled during reinforcement learning training. Each time a human preference is asked, a pair of sequences is generated: And humans express their preference y from it, y is the output of human preference, representing the distribution of which segment humans prefer: y∈{(1,0),(0,1),(0.5,0.5)} They respectively indicate that humans prefer the first sequence, the second sequence, and like the two sequences equally. This preference is taken as a triple (s 0 ,s 1 ,y) is saved in dataset I. To avoid the tedious human feedback process, the following equation is used instead of human preference as the criterion: Among them, R' t is the real value of the comprehensive reward; μ is the discount factor, μ∈(0,1); △θ is the difference in the turning angle of the ego vehicle at time t and the previous moment, the purpose is to reduce unnecessary swinging of the ego vehicle and improve riding comfort; Select D t The high value is selected as the human feedback, and the human preference triplet (s 0 ,s 1 ,y); The reward model is then updated using the stored preference samples First, establish a preference prediction model: In this paragraph, "s i =s j " indicates fragment s i Segments j is preferred; as a reward function for a neural network model with parameters , updated by minimizing the following loss: To ensure the training effect, in the loss function Then add L2 regularization, that is: Among them, τ is a regularization hyperparameter, which determines the penalty for complex models. The second norm of the weight W (that is, the square root of the sum of the squares of the weight elements) represents a measure of the complexity of the model. The larger the model weight, the more complex the model. After completing the update of the reward function neural network model, the data in the experience replay buffer (ReplayBuffer) of the reinforcement learning model is updated once, and the reward is recalculated for each state transition (i.e., state, action and its corresponding subsequent state) in the buffer using the updated reward function model.
9. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 1, characterized in that: The agent learning part takes the semantically segmented six-channel BEV image as input, and also inputs a measurement vector, which includes the vehicle state not represented in the BEV, including direction, throttle, brake, transmission gear, lateral and longitudinal speed. The above input is encoded through the same convolution and fully connected layers as in pre-training to help reinforce learning understanding, and the output is the predicted value of the vehicle's turning angle, throttle, and brake, where the value range of the steering control is defined as [-1,1], indicating the degree of left and right steering of the vehicle; the value range of the throttle control and the brake control are both set to [0,1], representing the force applied by the throttle and brake pedals, 0 represents no force, and 1 represents full force; During training, the Proximal Policy Optimization (PPO) algorithm is used for training. Trajectories are collected during training. Each trajectory consists of a series of states, actions a and rewards r, as well as a termination event z, which marks the end of a trajectory. The termination event z includes the policy network and the value network, and is iteratively updated through the following objective function: in, is the policy gradient loss after clipping, using the generalized advantage estimate to compute the advantage; is the maximum entropy loss, which is used to encourage exploration; is the exploration loss, guiding the policy toward exploring the distribution of “suggested” actions that prevent a particular terminal event z from occurring.
10. An automatic driving device, characterized in that: The device is provided with program code of the system according to any one of claims 1 to 9.
Citation Information
Patent Citations
Method and arrangement for generating control commands for an autonomous road vehicle
CN110654396A
Scoring autonomous vehicle trajectory using reasonable crowd data
CN114118658A
Automatic driving decision-making method and device for reinforcement learning guided by human driving data, and medium
CN118261233A
Backward Anti-collision driving decision-making method for heavy commercial vehicle
US20230182725A1
A computer-implemented method and an apparatus for reinforcement learning
WO2023225941A1
Cited By
General artificial intelligence value function learning prediction method and system
CN119670811A
Scene-driven high-generalization federal reinforcement learning automatic driving system and method
CN119821428A
Scenario-driven highly generalized federated reinforcement learning autonomous driving system and method
CN119821428B
Priori knowledge fused end-to-end automatic driving decision-making method
CN119975417A
An end-to-end autonomous driving decision-making method integrating prior knowledge
CN119975417B
