An end-to-end autonomous driving control system and equipment based on human preference reinforcement learning

Abstract

The present invention discloses an end-to-end autonomous driving control system and equipment based on human preference reinforcement learning. In the pre-training stage, the data collected in the CARLA simulator is used to pre-train the neural network model of the reward function based on the yaw angular velocity and the true value of the existing reward function, providing certain prior knowledge for the reward function model, which helps to accelerate the convergence process of the model. In the reward function learning stage, human preferences are used to correct and optimize the reward function. The cross entropy loss of the reward prediction value and the actual preference is used and L2 regularization is added to the loss function to ensure that the learning behavior is closer to human decision-making and prevent reward hacking, thereby achieving the alignment of the decision-making of the autonomous driving system with human values. In the intelligent agent learning stage, the PPO algorithm and multi-channel BEV are used as environmental inputs, and real-time training is performed in combination with the vector output of the throttle opening and the steering angle to ensure the real-time responsiveness and safety of the autonomous driving system.

Description

Technical Field

[0001] The present invention belongs to the field of transportation and relates to an end-to-end autonomous driving control system and equipment based on human preference reinforcement learning. Background Art

[0002] In the field of autonomous driving, achieving the vision of vehicles driving safely without human intervention is a key technological pursuit today. Autonomous driving technology aims to enable vehicles to navigate safely and efficiently without human intervention through advanced perception, decision-making, and control systems. Traditional autonomous driving systems typically rely on rule-based approaches, which require complex programming and pre-defined rules for various traffic situations. However, these rule-based systems are often rigid and limited when dealing with unknown and complex scenarios, and struggle to adapt to dynamically changing environments. To provide a more flexible and adaptable solution, reinforcement learning has been introduced to the field of autonomous driving. Reinforcement learning learns through interaction with the environment, eliminating the need for pre-defined rules and offering a more flexible and intelligent autonomous driving solution. However, traditional reinforcement learning often requires manual design of the state space and reward function, which is extremely difficult and inefficient in complex autonomous driving scenarios.

[0003] End-to-end reinforcement learning has been introduced to the field of autonomous driving as a method for providing more flexible and adaptable solutions. By learning driving behaviors directly from raw sensor input, end-to-end reinforcement learning integrates environmental perception, decision-making, and control execution, avoiding complex manual feature selection and state design, and reducing error propagation in intermediate steps such as path planning and trajectory tracking. In this way, end-to-end reinforcement learning improves the system's robustness and adaptability, enabling it to more effectively cope with dynamically changing environments.

[0004] Therefore, in this context, the introduction of human preferences to train reward function models abandons overly abstract and theoretical reward function settings and instead directly incorporates human intuitive judgment and feedback. By providing humans with a sequence of two states and actions, human feedback on their preferences is obtained. The reward function model is trained using a preference prediction model and a cross-entropy loss function, making the learning process more consistent with human driver behavior and decision-making. This strategy, which integrates human intelligence, not only promotes the development of reinforcement learning algorithms in the right direction, but also significantly reduces the potential for erroneous driving patterns during the solution process, such as strategies that exploit vulnerabilities in the reward system to obtain high rewards. Therefore, this provides autonomous driving systems with a higher level of intelligent decision-making and safety assurance.

[0005] In summary, the present invention proposes a novel control strategy for end-to-end reinforcement learning. By combining end-to-end reinforcement learning with human preferences, this strategy enables autonomous driving under complex urban road conditions, providing a more efficient, safe, and generalizable autonomous driving solution. Summary of the Invention

[0006] This invention provides an end-to-end autonomous driving control system and device based on human-biased reinforcement learning. This system employs a reward function model with human preferences, addressing the reward hacking problem that often occurs in traditional reinforcement learning and achieving end-to-end autonomous driving that aligns with real-world human goals and values. By incorporating human preferences, the model not only learns behaviors consistent with human decision-making but also effectively avoids policy failures caused by flawed reward design. The incorporation of human preferences is a major innovation of this invention, ensuring that the model's decisions during actual driving align with the expectations and values ​​of human drivers, enhancing the model's real-world applicability and safety. A pre-trained reward function model uses a pre-collected dataset as input. Through pre-training, the reward function model acquires prior knowledge about the task before learning begins, accelerating model convergence. Human preferences are leveraged to align learned behaviors more closely with human decision-making, avoiding reward hacking. Real-time vehicle and roadside information is used as input to the main reinforcement learning neural network. During training, the reward function model's neural network parameters are adjusted and optimized based on human preferences at fixed intervals, helping to address special circumstances and edge cases that may not have been fully considered during pre-training. While ensuring model training efficiency and the performance of the final model are more in line with human expectations and preferences, driving safety is also ensured.

[0007] The end-to-end autonomous driving control system based on human preference reinforcement learning of the present invention includes three main parts: pre-training part, reward function learning part and intelligent agent learning part.

[0008] For pre-training, data collected in advance from the CARLA simulator is used as input. The reward function's neural network model is pre-trained by adding the true value of the existing reward function output to the yaw rate response ratio as the target value. By pre-training with high-quality, pre-collected data, the reward function model acquires a certain amount of prior knowledge about the task before learning begins. This strategy helps the model converge faster, improves initial performance, and reduces training time.

[0009] For reward function learning, the reward predictions are used to calculate the cross entropy between the reward model's predicted preferences and the actual preferences. The reward distribution for different actions in this state is then obtained, and an L2 regularization term is added. This helps reduce model complexity and prevent overfitting by adding the sum of squared weights as a penalty term, thereby improving the model's generalization capabilities. By incorporating human preferences, using a preference prediction model and a cross entropy loss function, reward function learning makes the model's learning behavior more similar to human decision-making, thus avoiding reward hacking. This step helps the model make decisions that are more consistent with human expectations in complex environments and enhances the model's generalization capabilities.

[0010] For the agent learning component, the PPO algorithm is used. Real-time training is performed in the CARLA simulator, using semantically segmented multi-channel BEVs as input and vectorized throttle angles as output. The agent learning component uses the PPO algorithm and is trained using real-time vehicle and roadside information. This ensures that the model's performance in real-world operation is more aligned with human expectations and preferences, while also ensuring driving safety. Through continuous training and optimization, the agent learning component effectively improves the autonomous driving system's decision-making capabilities and adaptability.

[0011] By integrating these three components, the proposed end-to-end autonomous driving control system not only addresses the reward hacking problem that often arises in traditional reinforcement learning, but also enables the autonomous driving system to be closely aligned with real-world human goals and values. This strategy significantly enhances the generalization capabilities of the autonomous driving model and ensures behavioral consistency and safety in complex traffic environments, maintaining a high degree of consistency with the driving preferences of human passengers.

[0012] Part 1: Pre-training

[0013] For the reward function's neural network model, a dataset was pre-collected in the CARLA simulator using the ROACH model and saved as a .h5 file. This provided semantic information about perception and the ego vehicle's state. To help the neural network understand this input, features extracted using convolutional and fully connected layers were used as input to the neural network model. The raw input information included: 1) the semantically segmented multi-channel BEV; 2) the ego vehicle's throttle, steering, brake, and speed. The steering control value range was defined as [-1, 1], indicating the degree of left or right steering; the throttle and brake control values ​​were both set to [0, 1], representing the force applied to the accelerator and brake pedals, with 0 representing no force and 1 representing full force; and 3) the true value of the ROACH model's reward, r. b ; 4) The yaw rate of the vehicle, in degrees per second. The output is the predicted value of the reinforcement learning reward value V(s t ), which is used in the model to update the weights of the neural network.

[0014] Multi-channel BEV image after semantic segmentation i bev It includes six channels: drivable area, expected route, lane boundary, other vehicles in multiple time series, pedestrians in multiple time series, traffic lights and stop signs in multiple time series and their triggering areas. re The ego vehicle states not represented in the BEV are included, including ground truth measurements of steering, throttle, brake, lateral and horizontal speed. For the BEV, six convolutional layers are used for encoding; for the measurement vector, two fully connected layers are used for encoding. The outputs of the two encoders are merged and processed through two more fully connected layers to produce the latent feature j re , and finally input into a reward function neural network model with two fully connected hidden layers for training. The reward function neural network model consists of an input layer, two hidden layers, and an output layer. The input and output layers are both configured with a single unit, while the hidden layer is set to have 128 units. Each hidden layer is followed by a LeakyReLU activation function to increase nonlinear expression capabilities; a linear activation function is used in the output layer. To take stability into account, a comprehensive reward function is proposed with reference to the response ratio function. The equation is as follows:

[0015]

[0016] Among them, α represents the weight factor, α∈(0,1); R t is the actual reward observed at time t, which is replaced by the real value of the existing ROACH model in this invention; is the dimensionless yaw rate of the ego vehicle at time t; ξ is a constant to avoid the denominator being equal to zero.

[0017] For the loss function, a strategy using temporal difference learning is proposed to achieve sequence sensitivity in the reward prediction model. This strategy not only takes into account an action and its immediate reward, but also integrates the predicted reward value of the subsequent state, allowing the model to understand the long-term impact of actions in the sequence. A sequence decay coefficient λ is introduced into the loss function, which is applicable to the loss calculation of the reward prediction model. Specifically, the reward prediction V(s) for each action taken is t ) is updated as follows:

[0018] V(s t )←V(s t )+β[R' t +γV(s t+1 )-V(s t )]

[0019] Where V(s t ) is in state s at time tt The reward value prediction; β is the learning rate parameter used to adjust the step size; R′ t is the actual comprehensive reward observed at time t; γ is the discount factor for future rewards, which is a value between 0 and 1 that measures the importance of future rewards relative to immediate rewards; s t+1 Indicates the state at the next moment.

[0020] In order to better consider the order effect in the action sequence, the following generalized order dependency loss function is defined

[0021]

[0022] Here, represents the overall sequential dependency loss function, which considers future rewards at each moment and weights them by the coefficient λ. This means that the strategy places greater weight on predictions closer to the current moment, reflecting sequential sensitivity. T is the total number of model training steps. The squared term strengthens prediction accuracy, ensuring that the model is more severely penalized for predictions with large deviations, promoting more accurate long-term reward predictions.

[0023] To update the weights of a neural network, the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. In a neural network, the gradient measures the rate of change of each parameter based on the partial derivative of the loss function with respect to each parameter. These calculated gradients are used to drive the Adam optimizer, thereby updating the model parameters. The Adam optimizer, with its adaptive nature, helps to evenly adjust the learning rate at various learning stages, optimizing the path of the loss function's descent and enabling the model to converge to the optimal solution more quickly. Formally stated, weight updates follow the following rules:

[0024]

[0025] Among them, w t is the weight at training step t, △w t is the weight update amount, m t is the first-order moment estimate, i.e. the exponential moving average of the gradient, v t is the second-order moment estimate, i.e., the exponential moving average of the squared gradient, η is the learning rate, and ε is a small constant introduced to increase numerical stability.

[0026] Part II: Reward Function Learning

[0027] After the reward function prediction model is pre-trained, it is connected to the reinforcement learning model to replace the rule-based reward function part of the reinforcement learning model.

[0028] During the reinforcement learning strategy learning process, the current state and action are randomly sampled, and the reward function learning step is entered every K cycles. First, the algorithm queries the human preferences M times. Based on the randomly sampled information, the input for generating human preferences is:

[0029] σ={s k ,a k ,s k+1 ,a k+1 ,...,s k+N ,a k+N}

[0030] σ is the state and action sequence randomly sampled during reinforcement learning training, s k ,a k They represent the state and action of the agent at time k, respectively. N is the sequence length. Each time a human preference is asked, a pair of sequences is generated:

[0031]

[0032] And humans express their preference y from it, y is the output of human preference, representing the distribution of which segment humans prefer:

[0033] y∈{(1,0),(0,1),(0.5,0.5)}

[0034] They indicate that humans prefer the first sequence, the second sequence, and like the two sequences equally. This preference is expressed as a triple (σ 0 ,σ 1 ,y) is saved in dataset I. In the present invention, in order to avoid the tedious human feedback process, the following equation is selected instead of human preference as the criterion:

[0035]

[0036] Among them, R' t is the real value of the comprehensive reward; μ is the discount factor, μ∈(0,1); △θ is the difference in the turning angle of the ego vehicle at time t and the previous moment, the purpose of which is to reduce unnecessary swing of the ego vehicle and improve ride comfort.

[0037] Select D t The higher value is selected as the human feedback, and the human preference triplet (σ 0 ,σ 1 ,y).

[0038] The reward model is then updated using the stored preference samples First, establish a preference prediction model:

[0039]

[0040] In this paragraph, Indicates the probability that humans prefer 1 segment, Represents the fragment σ i Ratio segment σ j More preferred. Intuitively, this can be interpreted as assuming that the probability of a segment being preferred depends exponentially on the sum of an intrinsic reward function over the segment. is not a binary classifier, learning It is equivalent to performing binary classification under the label y provided by humans. Specifically, as a reward function of a neural network model with parameters, it is updated by minimizing the following cross entropy loss:

[0041]

[0042] Among them, y(i) represents human preference for the i-th sequence, E() represents expectation, (σ 0 ,σ 1 ,y)~I represents (σ 0 ,σ 1 ,y) is sampled from distribution I.

[0043] To ensure the training effect, in the loss function Then add L2 regularization, that is:

[0044]

[0045] Here, τ is a regularization hyperparameter that determines the penalty for complex models. Furthermore, the bi-norm of the weights W (the square root of the sum of the squares of the weight elements) represents a measure of model complexity: larger weights indicate more complex models. This representation helps control model complexity, further preventing overfitting and improving model generalization.

[0046] After the reward function neural network model is updated, the data in the reinforcement learning model's experience replay buffer is updated. Specifically, this involves recalculating the reward for each state transition in the buffer (i.e., a state, an action, and its corresponding subsequent state) using the updated reward model. This step ensures the synchronization and consistency of the experience replay to reflect the latest reward mechanism.

[0047] Part III: Agent Learning

[0048] The reinforcement learning model uses the ROACH model and the Proximal Policy Optimization (PPO) algorithm. The six-channel BEV image i after semantic segmentation is bev As input; in addition, an additional measurement vector m is input rl, which includes the vehicle state not represented in the BEV, including direction, throttle, brake, transmission gear, lateral and longitudinal speed. The above raw information is then encoded through the same convolution and fully connected layers as in pre-training to help the reinforcement learning neural network understand it. The output is the predicted value of the vehicle's steering angle, throttle, and brake. Among them, the value range of steering control is defined as [-1,1], which indicates the degree of left and right steering of the vehicle; and the value range of throttle control and brake control is set to [0,1], representing the force applied to the throttle and brake pedals, with 0 representing no force and 1 representing full force.

[0049] During training, the approximate policy optimization (PPO) algorithm is used to train the model, in which the pruning technique is used to avoid excessive policy updates. Trajectories are collected during training, and each trajectory consists of a series of states (BEV images i bev and the measurement vector m rl ), action a and reward r, and a termination event z, which marks the end of a trajectory. The termination event z includes the policy network and the value network, which are iteratively updated through the following objective function:

[0050]

[0051] in, Denotes the trajectory τ is generated according to the strategy Generated, θ k+1 is the updated value of the policy parameters, the goal is to maximize the overall loss function described below, is the policy gradient loss after clipping, using the generalized advantage estimate to calculate the advantage; is to maximize entropy loss, which is used to encourage exploration; is the exploration loss that guides the policy toward exploring the distribution of “suggested” actions that prevent a specific terminal event z from occurring.

[0052] first, is the clipped policy gradient loss, which uses Generalized Advantage Estimation (GAE) to estimate the advantage. This loss function helps the policy network learn the most beneficial actions in the long run:

[0053]

[0054] π θ (a t ∣s t ) is the current strategy in state s t Next, perform action a t probability; is the old policy in state st Next, perform action a t probability; is the advantage function at time step t; clip(·,1-∈,1+∈) is a function that clips the ratio to limit the magnitude of the policy update; ∈ is a hyperparameter for clipping

[0055] The second objective function is That is, maximum entropy loss, which is a loss commonly used to promote exploration and avoid falling into local optimality. It is defined as follows:

[0056]

[0057] H(π θ (·|i bev ,m rl )) is the entropy of the strategy, which is used to quantify the randomness of the strategy; λ ent is a hyperparameter that controls the weight of the entropy loss.

[0058] By increasing the entropy of the action distribution, the action distribution is pushed towards a uniform distribution, because maximizing the entropy is equivalent to minimizing the KL divergence to a uniform distribution. u(-1,1) represents a uniform distribution on the interval [-1,1]:

[0059] H(π θ )=-KL(π θ ‖u(-1,1))

[0060] If two probability distributions have the same support set (i.e. they can produce the same set of outcomes), then maximizing the entropy (i.e. randomness) of one distribution is equivalent to minimizing its KL divergence to another uniform distribution. This generalized form is designed to encourage the algorithm to explore reasonable directions that comply with basic traffic rules. In other words, it not only encourages the algorithm to perform more and more random exploration behaviors, but also guides the algorithm to explore in directions that are more likely to comply with traffic rules. This design can not only enhance the model's ability to explore new states, but also ensure that the behavior learned by the model complies with the rules of the real world. This form is named exploration loss and is defined as:

[0061]

[0062] Among them, "1" represents the indicator function, Represents an event, specifically an event that ends a sequence (episode). The event set Including collision, running red light / traffic sign violation, deviation from route and obstruction; KL(π θ (·|i bev,k ,m rl,k )‖p z ) is the policy distribution and the predefined exploration prior p z The Kullback-Leibler divergence between exp Is a hyperparameter that controls the weight of the exploration loss; is a hyperparameter that specifies the number of steps at the end of each episode. The algorithm applies a uniform prior to all moments regardless of whether event z is triggered or not. The algorithm is at the last N of the sequence z Step 2: Adjust the action towards the predefined exploration prior p z , which encodes a “recommendation” to prevent the triggered event z from happening again. In practice, using N z =100 and If z is related to a collision or running a red light / sign, set the acceleration prior to To encourage Roach to slow down, while directional control is not affected. Conversely, if the car is stopped, use As a prior on acceleration. For course deviation, a uniform prior is applied on the directional control While this is equivalent to maximizing entropy in this case, the exploration loss further encourages exploration of steering angles during the last 10 seconds before going off-course.

[0063] Beneficial effects of the present invention:

[0064] (1) The present invention provides an end-to-end autonomous driving control system based on human preference reinforcement learning. By implementing a reward prediction model that takes vehicle stability into consideration, the application of human preferences is introduced. Through the feedback of human preferences, with the help of the preference prediction model and the cross-entropy loss function, and combined with a carefully calibrated CARLA simulation data set, the present invention effectively overcomes the reward manipulation ("reward hacking") problem commonly seen in traditional reinforcement learning processes. This mechanism ensures that the autonomous driving decision-making process is closely linked to the actual goals and values ​​of human drivers, achieving a natural alignment of the end-to-end autonomous driving system. Through this strategy, the generalization ability of the autonomous driving model is significantly enhanced, and the behavioral consistency and safety in complex traffic environments are ensured, maintaining a high degree of consistency with the driving preferences of human passengers.

[0065] (2) Introduction of a multi-dimensional input mechanism: By integrating a bird's-eye view (BEV) based on semantic segmentation with auxiliary measurement vectors such as yaw rate, this model achieves a comprehensive understanding of the autonomous driving environment and vehicle status. Compared with traditional models, this multi-dimensional input mechanism significantly optimizes the autonomous driving system's ability to assess vehicle motion stability and safety, thereby improving the accuracy and reliability of autonomous navigation.

[0066] (3) Design of a comprehensive reward function: The comprehensive reward function proposed in this paper innovatively optimizes the reward prediction mechanism for autonomous driving by integrating yaw rate with traditional reward metrics. This design not only simplifies the complex parameter setting problem but also enhances the model's adaptability and generalization capabilities in changing environments. This comprehensive reward function makes the autonomous driving system's path planning more accurate in unknown environments, effectively improving the vehicle's stable control and safety performance in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the end-to-end autonomous driving control system based on human preference reinforcement learning proposed in this invention;

[0068] Figure 2 The structure diagram of the reward prediction model proposed by the present invention;

[0069] Figure 3 Schematic diagram of reinforcement learning training proposed by the present invention; DETAILED DESCRIPTION

[0070] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the content of the present invention is not limited thereto.

[0071] The present invention provides an end-to-end autonomous driving control system based on human preference reinforcement learning, which adopts a reward function model with human preference, such as Figure 1 As shown in the figure, the reward hacking problem that is prone to occur in traditional reinforcement learning is solved, and end-to-end autonomous driving is achieved that is aligned with human's real goals and values. The specific steps include:

[0072] (1) Enable the pre-training of the reward prediction model, such as Figure 2 shown.

[0073] 1. Pre-training data collection and processing: Use the ROACH model to perform pre-collection activities in the CARLA simulation environment to collect the required driving data and then save it in .h5 file format. This data includes six channels of semantically segmented bird's-eye view (BEV) images, dynamic control information of the vehicle (including throttle, steering, braking, speed), and the actual reward value r corresponding to the ROACH model. band the yaw rate of the vehicle. The steering control uses a range of [-1, 1] to reflect the degree of left and right steering of the vehicle, while the throttle and brake control ranges are set to [0, 1] to represent changes in pedal application force.

[0074] 2. Data encoding and feature extraction: The collected data items are processed using a specially designed neural network structure. For the BEV image data, six layers of convolutional layers are used for encoding to obtain the encoded BEV image features. For the measurement vector data of the vehicle state, two layers of fully connected layers are used for encoding to extract features related to the real-time state of the vehicle. Finally, the encoder outputs of these two parts are merged and further processed through two additional layers of fully connected layers to produce the final latent feature vector j re .

[0075] 3. Construction and Optimization of the Reward Function Model: We constructed an end-to-end neural network consisting of an input layer, two fully connected hidden layers, and an output layer. The network design uses a unit output layer to predict reward values. Each hidden layer is followed by a LeakyReLU activation function to enhance nonlinearity, while the output layer uses a linear activation function. By incorporating stability considerations into the reward function, we constructed the following comprehensive reward function to accurately evaluate and predict reward values ​​in autonomous driving, further improving the stability and accuracy of the model:

[0076]

[0077] 4. Utilization of order-sensitive loss function: Using the time difference learning strategy, the order decay coefficient λ is introduced. This loss function not only focuses on the immediate reward, but also considers the predicted reward V(s) of the future state. t ) to deepen the model's understanding of the long-term impact of action sequences. The generalized sequential dependency loss function considers the entire action sequence, enhancing prediction accuracy and ensuring penalties for high-deviation predictions, thereby optimizing the accuracy of long-term reward predictions. This results in the following loss function:

[0078]

[0079] 5. Optimizing and updating model parameters: Using the backpropagation algorithm, combined with the Adam optimizer, we calculate gradients based on the partial derivatives of the loss function with respect to each parameter to guide model parameter updates. The Adam optimizer, through its adaptive nature, balances learning rate adjustments, promoting rapid reduction of the loss function and convergence to an optimized solution. Weight updates adhere to the following rules to ensure robustness and efficiency during model training:

[0080]

[0081] (2) Enable the reinforcement learning training part, such as Figure 3 shown.

[0082] 1. Integration of pre-trained reward prediction models: Integrating pre-trained reward prediction models into reinforcement learning models replaces traditional rule-based reward functions, thereby improving the model's performance in simulating complex decision-making environments.

[0083] 2. Environment setup and policy initialization: define the state space and use the six-channel BEV image i bev and the measurement vector m rl As input, initialize the policy network and value network and prepare the Proximal Policy Optimization (PPO) algorithm.

[0084] 3. Data collection: Use the current strategy to perform actions in the environment and collect states (BEV images i bev and the measurement vector m rl ), action a and reward r, and termination event z.

[0085] 4. Objective function construction: Construct the objective function θ k+1 , consists of three parts: the policy gradient loss after clipping Maximum Entropy Loss To promote exploration, and exploration loss To avoid unwanted aborts with a boot strategy:

[0086]

[0087] 5. Policy Optimization: Apply the collected data to the objective function and iteratively update the policy network using gradient descent. This process uses the PPO algorithm's pruning technique to control the magnitude of policy updates and maintain learning stability.

[0088] 6. Execution of periodic reward function learning: Set a periodic cycle, and execute reward function learning every K cycles. The algorithm automatically generates the state and action sequence σ={s k ,a k ,s k+1 ,a k+1 ,...,s k+N ,a k+N} and simulates human preferences based on the following preset equation:

[0089]

[0090] 7. Take the comprehensive reward real value R' t , discount factor γ and behavior stability evaluation index constitute the optimization reward function:

[0091] 8. Establish a preference prediction model and update it: Use the collected preference data samples to establish a preference prediction model:

[0092]

[0093] 9. Iteratively update the reward function neural network model by minimizing the following loss function (including L2 regularization) to optimize model training results:

[0094]

[0095] 10. Synchronous update of the experience replay buffer: Using the updated reward model, a comprehensive update of the data in the experience replay buffer of the reinforcement learning model is performed to ensure the consistency of the data with the latest reward mechanism.

[0096] 11. Definition of the agent learning strategy: The Proximal Policy Optimization (PPO) algorithm is used to define the agent learning strategy using the semantically segmented six-channel BEV image and measurement vector as input to improve the effectiveness and efficiency of training.

[0097] 12. Agent training process: During the agent training process, trajectory data consisting of states, actions, and rewards is collected, and the agent model is iteratively updated using specific objective functions (including policy gradient loss, entropy loss, and exploration loss) based on termination events.

[0098] 13. Application of Objective Function and Exploration Loss: Based on the PPO algorithm training, entropy loss is introduced to promote policy exploration. Exploration loss is also applied to guide the agent to avoid undesirable termination events. By adjusting the details of action preferences, the model's adaptability and behavioral compliance under real-world rules are enhanced.

[0099] 14. Evaluate and Iterate: Evaluate the performance of the updated policy in the environment. Based on the evaluation results, iterate steps 3 through 13 until the predetermined performance threshold is met or the maximum number of iterations is reached.

[0100] Based on the above control system, an embodiment of the present invention further proposes a control device for autonomous driving, in which a program code of the control system is set.

[0101] In summary, the present invention provides an end-to-end autonomous driving control system and equipment based on human preference reinforcement learning. It adopts a reward function model with human preferences, solves the reward hacking problem that is easy to occur in traditional reinforcement learning, and realizes end-to-end autonomous driving that is aligned with human's real goals and values. Based on the pre-collected data set as the input of the pre-trained reward function model, through the advantages of pre-training, the reward function model can obtain certain prior knowledge about the task before learning begins, helping the model to converge faster; using human preferences, the learning behavior is closer to human decision-making, avoiding reward hacking behavior; based on the real-time collected vehicle-side and road-side information as the input of the reinforcement learning main neural network, during the training process, the neural network parameters of the reward function model are adjusted and optimized according to human preferences at fixed time periods, which helps to deal with special situations and edge cases that may not be fully considered in the pre-training process. While ensuring that the model training efficiency and the performance of the final model are more in line with human expectations and preferences, driving safety is ensured.

[0102] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving system based on human preference reinforcement learning, characterized by: include: Pre-training part, reward function learning part and agent learning part; The pre-training part uses the pre-collected data as input and the true value of the reward function output and the yaw angular velocity as the response ratio as the target value to pre-train the neural network model of the reward function. By using the pre-collected data for pre-training, the neural network model of the reward function can obtain certain prior knowledge about the task before learning; In the pre-training part, the pre-collected data is collected in the CARLA simulator using the ROACH model and saved as a .h5 file; Specifically include: 1) Multi-channel BEV after semantic segmentation; 2) The ego vehicle's throttle, steering, brake, and speed. The steering control value range is defined as [-1, 1], indicating the degree of left or right steering of the vehicle. The throttle control and brake control value ranges are both set to [0, 1], representing the force applied to the accelerator and brake pedals, with 0 representing no force and 1 representing full force. 3) The true value of the reward r of the ROACH model b ; 4) The yaw rate of the vehicle, in degrees per second, output as the predicted value of the reinforcement learning reward value V(s) t ), used to update the weights of the neural network; For the loss function of the neural network model, the time difference learning method is used to achieve sequential sensitivity in the reward prediction model; a sequence decay coefficient λ is introduced into the loss function, which is applicable to the loss calculation of the reward function prediction model. Specifically, the reward prediction V(s t ) is updated as follows: V(s t )←V(s t )+β[R t '+γV(s t+1 )-V(s t )] Where V(s t ) is in state s at time t t The reward value prediction; β is the learning rate parameter used to adjust the step size; R′ t is the actual comprehensive reward observed at time t; γ is the discount factor for future rewards, which is a value between 0 and 1 that measures the importance of future rewards relative to immediate rewards; s t+1 Indicates the state at the next moment; Taking into account the order effect in the action sequence, the generalized order dependency loss function is defined as follows Here, represents the overall sequential dependency loss function, which considers future rewards at each moment and weights them by the coefficient λ, giving higher weight to predictions closer to the current moment, reflecting sequential sensitivity. The square term strengthens the accuracy of the prediction, ensuring that the model is more severely penalized for predictions with large deviations, promoting more accurate long-term reward predictions. The reward function learning part uses the predicted value of the reward function neural network model to obtain the preference predicted by the reward function model, performs cross entropy with the actual preference, obtains the reward distribution of different actions in this state, and adds an L2 regularization term. By adding the sum of squared weights as a penalty term, it reduces the model complexity and prevents overfitting, thereby improving the generalization ability of the model. By introducing human preferences, the reward function learning uses the preference prediction model and the cross entropy loss function to make the model's learning behavior closer to human decision-making; The intelligent agent learning part uses the multi-channel BEV after semantic segmentation as input and the throttle opening and steering angle in vector form as output for real-time training. The intelligent agent learning adopts the PPO algorithm and is trained through real-time collected vehicle-side and road-side information.

2. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 1, characterized in that: The semantically segmented multi-channel BEV image includes six channels, namely, a drivable area, an expected route, lane boundaries, multiple time series of remaining vehicles, multiple time series of pedestrians, and multiple time series of traffic lights and stop signs and their triggering areas. The measurement vector includes the vehicle state not represented in the BEV, including ground-truth measurements of steering, throttle, brake, lateral, and horizontal speeds. For the BEV, six convolutional layers are used for encoding processing. For the measurement vector, two fully connected layers are used for encoding. The two encodings are merged and processed through two more fully connected layers to generate latent features. Finally, the latent features are input into a neural network model with a reward function and two fully connected hidden layers for training.

3. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 2, characterized in that: The neural network model of the reward function consists of an input layer, two hidden layers, and an output layer. The input layer and the output layer are both configured with a single unit, and the hidden layer is set to 128 units. Each hidden layer is followed by a LeakyReLU activation function to increase nonlinear expression capabilities; a linear activation function is used in the output layer.

4. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 3, characterized in that: In order to take stability into account, a comprehensive reward function is proposed with reference to the response ratio function, as follows: Among them, α represents the weight factor, α∈(0,1); R t is the actual reward observed at time t, replaced by the true value of the existing ROACH model; is the dimensionless yaw rate of the ego vehicle at time t; ξ is a constant to avoid the denominator being equal to zero.

5. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 3, characterized in that: For neural network weight updates, the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. The gradient measures the rate of change of each parameter based on the partial derivative of the loss function with respect to each parameter. These calculated gradients are used to drive the Adam optimizer to update the model weights. The weight updates follow the following rules: Among them, w t is the weight at the training step, w t+1 is the weight at training step t+1, Δw t is the weight update amount, m t is the first-order moment estimate, i.e. the exponential moving average of the gradient, v t is the second-order moment estimate, i.e., the exponential moving average of the squared gradient, η is the learning rate, and ε is a small constant introduced to increase numerical stability.

6. The end-to-end autonomous driving system based on human preference reinforcement learning according to claim 1, characterized in that: The agent learning part takes the semantically segmented six-channel BEV image as input, and also an additional measurement vector, which includes the vehicle state not represented in the BEV, including direction, throttle, brake, transmission gear, lateral and longitudinal speed. The above input is encoded through the same convolution and fully connected layers as in pre-training to help reinforce learning understanding, and the output is the predicted value of the vehicle's steering angle, throttle, and brake. Among them, the steering control value range is defined as [-1, 1], indicating the degree of left and right steering of the vehicle; the throttle control and brake control value ranges are both set to [0, 1], representing the force applied to the throttle and brake pedals, with 0 representing no force and 1 representing full force. During training, the Proximal Policy Optimization (PPO) algorithm is used. Trajectories are collected during training. Each trajectory consists of a series of states, actions a, and rewards r, as well as a termination event z, which marks the end of a trajectory. The termination event z includes the policy network and the value network, and is iteratively updated using the following objective function: in, is the policy gradient loss after clipping, using the generalized advantage estimate to calculate the advantage; is to maximize entropy loss, which is used to encourage exploration; is the exploration loss that guides the policy towards exploring the distribution of “suggested” actions that prevent a specific terminal event z from occurring.

7. An automatic driving device, characterized in that: The device is provided with a program code of the system according to any one of claims 1 to 6.

Images