An unmanned driving reward learning and control method based on integrated maximum entropy deep inverse reinforcement learning

Abstract

This invention discloses an autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning, comprising the following steps: Step 1: Learning the reward function and control operation in a highway autonomous vehicle driving environment and constructing it into a Markov decision process model; Step 2: Constructing a soft Q-learning model based on value pruning to obtain expert demonstrations, and dividing the inverse reinforcement learning task into sub-tasks according to expert preferences; Step 3: Establishing a strong learner ensemble model and recovering the reward function through maximum entropy deep inverse reinforcement learning; Step 4: Achieving the fusion of reward functions of each sub-task through linear combination, thereby improving the learning accuracy of the reward function. This invention considers the problems of gradient explosion, gradient vanishing, and data overflow in soft Q-learning. It learns expert demonstrations through an improved soft Q-learning algorithm and proposes an ensemble maximum entropy deep inverse reinforcement learning algorithm based on the learned expert demonstrations, which can better achieve decision control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and reinforcement learning, and particularly relates to a reward learning and control method for highway unmanned driving based on integrated maximum entropy deep inverse reinforcement learning. BACKGROUND

[0002] In recent years, with the rapid development of high-performance computing, big data and deep learning technology, the reinforcement learning algorithm of artificial intelligence software core technology such as AlphaGo and AlphaGoZero and its application have received more extensive attention and faster development. The algorithm has been widely used in games, mechanical arms, robots, unmanned driving and other fields. In the interaction process with the environment, the reinforcement learning algorithm learns the optimal strategy to maximize the benefits or achieve a specific goal. Reinforcement learning regards learning as a trial-evaluation process, the agent selects an action to act on the environment, the environment changes the state after accepting the action, and a reinforcement signal is generated and fed back to the agent, the agent selects the next action according to the reinforcement signal and the current state of the environment, and the principle of selection is to increase the probability of receiving positive reinforcement. Therefore, the design of the reward function has become a bottleneck problem restricting the development of reinforcement learning.

[0003] The artificial design of the reward function is affected by many problems. For example, there are many factors to be considered for unmanned driving, so the reward function needs to consider speed limit reward, collision reward, driving on sidewalk reward, line crossing reward, comfortable driving reward, stop reward, etc., which is difficult to design. Therefore, researchers propose inverse reinforcement learning algorithm, which eliminates the professional knowledge and effort required for manually designing cost function, and can generate a reward function more consistent with human behavior; then make task decision. First, the researchers propose expert cloning and imitation learning inverse reinforcement learning algorithm, but these two algorithms can only completely imitate the behavior of experts with limited behavior observation. Therefore, the apprentice learning and maximum marginal inverse reinforcement learning and other marginal class inverse reinforcement learning algorithms are proposed, which make the feature expectation of the learner as close as possible to the feature expectation of the expert through learning. However, this kind of algorithm has a fuzzy problem, that is, multiple strategies correspond to one reward function. Therefore, the deep inverse reinforcement learning algorithm based on maximum entropy has been widely developed, and the core idea of this method is to improve the learning accuracy of the reward function by reducing the difference between the expected state access frequency of the learner and the expected experience state access frequency.

[0004] However, in the maximum entropy deep inverse reinforcement learning algorithm, non-optimal, finite, and imbalanced expert demonstration data affect the learning of the reward function, and the algorithm also suffers from overfitting, all of which limit the learning performance of the reward function. This invention, within the framework of the maximum entropy deep inverse reinforcement learning algorithm, proposes a maximum entropy deep inverse reinforcement learning method based on soft Q-learning and ensemble algorithms for reward function fusion in a highway autonomous driving environment. This method solves the problems of non-optimal, finite, and imbalanced expert demonstration data, as well as overfitting, improving the learning accuracy of the reward function and better realizing decision control in reinforcement learning. Summary of the Invention

[0005] To address the non-optimal, limited, and imbalanced expert demonstrations in highway autonomous driving environments, this invention proposes an improved soft Q-learning algorithm based on value pruning and scaling of exponential linear unit activation functions. This algorithm balances exploration and utilization, and solves the problems of gradient explosion, gradient vanishing, and data overflow. Consequently, it yields better and more optimized expert demonstrations for autonomous vehicles.

[0006] In the context of autonomous driving on highways, this invention addresses the low learning efficiency and overfitting issues of traditional maximum entropy deep inverse reinforcement learning algorithms. It proposes an ensemble maximum entropy deep inverse reinforcement learning method, decoupling the reward function learning task into multiple sub-tasks based on expert preferences. Then, an ensemble algorithm is used to combine each maximum entropy deep weak learner into a strong learner to learn the reward function of each sub-task. To prevent overfitting by the strong learner, a constant regularization is added to the linear combination of these weak learners.

[0007] To address the complex computational issues in ensemble maximum entropy deep inverse strong learning, this invention proposes a method to sparsify the rewards obtained by the strong learner using a truncated gradient approach, thereby reducing the complexity of the model.

[0008] To address the overfitting problem in ensemble maximum entropy deep inverse strong learning, this invention proposes a linear combination method that merges the reward functions learned from each subtask of highway autonomous driving into a unified reward function.

[0009] To address the aforementioned technical problems, this invention provides a reward learning and control method for autonomous driving based on ensemble maximum entropy deep inverse reinforcement learning. This method can learn superior expert demonstrations in inverse reinforcement learning application experiments, overcoming the expert demonstration problems of non-optimal, finite, and imbalanced autonomous vehicles. Simultaneously, based on expert preferences, the complex reward function learning task is decoupled into multiple sub-tasks. Ensemble learning is used to integrate deep maximum entropy weak learners into a strong learner. Then, the reward functions of each sub-task in autonomous driving are linearly combined into the final reward function, improving the learning efficiency of the reward function and achieving better driving decision-making performance.

[0010] To achieve the aforementioned goal of unmanned driving on highways, this invention provides the following technical solution:

[0011] This invention first provides an autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning, comprising the following four steps:

[0012] Step 1: Construct a Markov decision process-based model in the inverse reinforcement learning application scenario of highway autonomous driving, and transform the highway autonomous driving environment into solving a Markov decision process;

[0013] Step 2: To solve the Markov decision process constructed in Step 1, we first construct an improved soft Q-learning model based on value pruning and scaling exponential linear unit activation functions to obtain expert demonstrations, and decouple the inverse reinforcement learning task into sub-tasks according to expert preferences.

[0014] Step 3: Under the expert demonstrations of each subtask obtained in Step 2, establish a strong learner ensemble model, solve the reward function of each subtask through maximum entropy deep inverse reinforcement learning, and use the ensemble algorithm to combine the deep maximum entropy weak learners into a strong learner.

[0015] Step 4: The reward functions of each subtask in the autonomous driving environment learned in Step 3 are fused together using a linear combination method, thereby improving the learning accuracy of the autonomous driving reward function.

[0016] Specifically:

[0017] In step one, the reinforcement learning problem for autonomous driving on highways is modeled as a Markov decision process consisting of quintuples. Where S represents the state space of the autonomous vehicle, s i ∈S; A represents the action space of the autonomous vehicle, a i ∈A; Let represent the transition probability of the autonomous vehicle; R represent the reward function of the autonomous vehicle; and γ represent the discount factor. Based on the definition of reinforcement learning models, the inverse reinforcement learning problem of autonomous driving on highways is modeled as a quadruple. The highway autonomous driving environment consists of a three-lane highway, a red master vehicle v0, and environmental vehicles {v1, v2, v3, v4, v5}. Three conditions must be met in the highway autonomous driving system: 1) the red vehicle is faster than other environmental vehicles; 2) the drivers of different vehicles do not communicate with each other; and 3) different vehicles do not share data. The cooperation process of the red master vehicle is described as follows: First, the red master vehicle on the highway is in its current state s. i Next, take an action π(s) i Secondly, the cars in the highway environment observe the actions of the red master car and take action according to the strategy. Finally, the red master car obtains the next state s through its new position and the cars in the environment before and after v0. i+1 .

[0018] In step two, the entropy term is added to the traditional soft Q-learning algorithm to obtain the optimal objective function π for solving the reward function of autonomous driving on highways. * ,

[0019]

[0020] Where R(s) i ,a i Let ρ represent the reward function of the autonomous vehicle. π This represents the state-action edges of the trajectory distribution of the autonomous vehicle. The entropy term is derived through... The calculation is performed to represent the strategy π(·|s) of the autonomous vehicle. i The randomness in the probability distribution. σ represents the temperature factor controlling the importance of the entropy term, i.e., controlling the randomness of the actions chosen by the autonomous vehicle. The greater the entropy, the greater the randomness of the strategy chosen by the autonomous vehicle.

[0021] The soft Q-function and soft value function in the inverse reinforcement learning algorithm for autonomous driving on highways are defined as follows:

[0022]

[0023]

[0024] The discount factor γ represents the impact of future rewards on current rewards. The larger γ is, the more steps the autonomous vehicle considers, but the higher the training difficulty.

[0025] Through importance sampling, the soft Bellman error of the soft Q-function of an autonomous vehicle can be expressed in an equivalent form as minimization.

[0026]

[0027] Where ds d represents the distribution of the state space of an autonomous vehicle. a This indicates the distribution of the autonomous vehicle's action space. This represents the target Q value of the autonomous vehicle. This represents the target parameters for autonomous vehicles.

[0028] The gradient explosion problem is solved by using the scaling exponential linear unit activation function of the neural network and choosing the slope λ1 of the right half axis as 1.05.

[0029]

[0030] Where λ² is 1.67, it represents the soft saturation region. The slope.

[0031] During the experiment of autonomous driving on highways, the value function of the improved soft Q-learning algorithm grew very rapidly, resulting in a data overflow problem. Therefore, a value pruning method was invented to handle the soft Q-value function, which is used when the soft value function exceeds a defined threshold. The soft Q-value function has been updated to Otherwise, the soft Q-value function remains unchanged.

[0032]

[0033] in This represents the soft Q-value function for autonomous driving after value clipping.

[0034] In the improved soft Q-learning algorithm, the states obtained from the highway autonomous driving environment are used as input, and the action distribution is used as output. The higher the probability of an action, the greater the likelihood that the corresponding action will be selected. During the iterative learning process of the improved soft Q-learning algorithm, the learned state-action pairs and reward functions are stored in a buffer. Then, after sampling, expert demonstrations of the inverse reinforcement learning algorithm are obtained. Based on expert preferences, the complex reward function learning task of the autonomous vehicle is decoupled into multiple sub-tasks for reward function learning. Under each expert demonstration sub-task, the reward function and optimal policy of the autonomous vehicle are learned.

[0035] In step three, based on the expert demonstration of the autonomous vehicle learned using the improved soft Q-learning algorithm, the method for establishing the strong learner ensemble model is as follows: a weak learner usually refers to a learner whose generalization performance is only slightly better than random guessing. In the maximum entropy deep inverse reinforcement learning framework integrated in the highway autonomous driving environment, each maximum entropy deep inverse reinforcement learning process is regarded as a weak learner through expert demonstration under sub-tasks, inputting state features and outputting rewards. The optimal network parameters w are calculated by formula (7).

[0036]

[0037] in Expert demonstration Chinese experts demonstrate trajectory The number of Let p(s) represent the partition function. i+1 |s i ,a i ) represents the state transition probability of an autonomous vehicle. Indicates the state of the driverless car s i The characteristic function is denoted by j, which represents the number of subtasks in the learning reward function of the autonomous vehicle.

[0038] According to the chain rule, the data items in formula (7) The complete derivative with respect to the network parameter scalar w consists of two parts: one is the data term. The other is the derivative of the reward function r(w) with respect to the parameter w.

[0039]

[0040] in This indicates the efficiency of gradient backpropagation in deep neural networks during maximum entropy deep inverse reinforcement learning. This represents the expected number of experiential state visits during expert demonstrations of autonomous vehicles. This represents the expected number of state visits learned by the autonomous vehicle under expert demonstration. If the reward function is positive, c is set to 0.01; otherwise, c is set to -0.01.

[0041] In expert demonstration The maximum entropy deep inverse reinforcement learning algorithm was used to learn the reward function of an autonomous vehicle. This is treated as a weak learner. The input to a weak learner is a feature function and weights, and the output is a reward function.

[0042]

[0043] Where w represents the weight of the reward function for the autonomous vehicle. The characteristic function representing an autonomous vehicle, This shows the trajectory of an expert demonstration of a self-driving car.

[0044] During the learning process of the weak learner in highway autonomous driving, the ensemble algorithm performs the following operations:

[0045] Step 1: In the ensemble maximum entropy deep inverse reinforcement learning model for autonomous vehicles, the weight distribution of the reward function learned through the ensemble maximum entropy deep inverse reinforcement learning method is first initialized as: V1 = (w 1,1 ,w 1,2 ,…,w 1,I ), i = 1, 2, ..., I.

[0046] Step 2: Utilize the current distribution Expert demonstration of self-driving car training data to learn weak learners Weak learner is calculated The learning error rate

[0047]

[0048] in This indicates the expected frequency of state visits by the learner in the autonomous vehicle. This indicates the expected frequency of access to the experiential state for autonomous vehicles.

[0049] Step 3: Weak Learner coefficient The calculation is as follows:

[0050]

[0051] Step 3: Update the weights of the autonomous vehicle's reward function to prepare for the next weak learner.

[0052]

[0053] Where α is set to 0.01, This represents the normalization factor. Ensemble learning was used to increase the weight of data from expert demonstrations that the autonomous vehicle learned from incorrect examples, while decreasing the weight of data from expert demonstrations that learned from correct examples.

[0054] In the complex nonlinear environment of autonomous driving on highways, the maximum entropy deep inverse learning reward method suffers from computational complexity and overfitting, affecting the learning accuracy of the reward. To prevent these problems, the invented maximum entropy deep inverse learning method incorporates a gradient truncation method and a correction factor.

[0055] The reward for a weak learner in maximum entropy deep inverse reinforcement learning can be calculated as follows.

[0056]

[0057] in For gravity parameters, The larger the value, the greater the sparsity of the gradient cutoff method. And... ζ is the threshold for the gradient cutoff method, and ζ is the gravity parameter for the gradient cutoff method. The coefficient.

[0058] The indicator function TG for the gradient cutoff method is calculated as follows:

[0059]

[0060] During training, the AdaBoost algorithm exponentially increases the weights of weaker reward performers. Therefore, the next weak learner will bias towards learning these weighted rewards, making them more susceptible to noise. To prevent this problem, a correction factor ν is added to the linear combination of weak learners, forming the coefficients of the weak learners.

[0061] Furthermore, ensemble algorithms combine multiple weak learners into a strong learner through linearly weighted voting, thereby solving for the reward function of the subtask.

[0062]

[0063] Where the coefficient The value of the weak learner is indicated by v, and constant regularization is indicated by v.

[0064] In step four, the method for establishing the fusion of reward functions of various sub-tasks in the autonomous driving environment through linear combination is as follows: For multiple sub-tasks, the reward functions learned by the strong learner are fused into the final reward function through linear combination, which improves the learning accuracy of the reward function and obtains better decision performance.

[0065]

[0066] in Indicates the k-th s The coefficients of the strong learner in each subtask Compared with the prior art, the present invention has the following technical advantages.

[0067] 1) In the autonomous driving environment of highways, compared with the traditional maximum entropy deep inverse reinforcement learning, this invention not only considers the expert demonstration problem of non-optimal, finite, and unbalanced autonomous vehicles, but also considers the gradient explosion, gradient vanishing and data overflow problems in the soft Q-learning algorithm. It has invented an improved soft Q-learning expert demonstration based on value clipping and scaling exponential linear unit activation function to obtain a better expert demonstration of autonomous vehicles.

[0068] 2) Based on expert demonstrations of autonomous vehicles obtained using improved soft Q-learning, the maximum entropy deep inverse reinforcement learning algorithm is integrated. The introduction of constant regularization in the process of gradient truncation and weak learner integration improves the learning rate of the reward function of the autonomous vehicle, solves the problems of complex calculation and overfitting. Experimental results show that compared with other algorithms, the present invention effectively improves the learning performance of the reward function and better realizes the decision control of autonomous driving. Attached Figure Description

[0069] Figure 1 This is a diagram of a maximum entropy deep inverse reinforcement learning model based on improved soft Q-learning and ensemble algorithms.

[0070] Figure 2 This is a diagram of an improved soft Q-learning algorithm model based on value pruning and activation functions.

[0071] Figure 3 This is a map of an autonomous driving environment on a highway.

[0072] Figure 4 This is the input-output model diagram of the improved soft Q-learning algorithm used in autonomous driving.

[0073] Figure 5 This is a value difference graph in autonomous driving.

[0074] Figure 6 This is a difference value map for autonomous driving.

[0075] Figure 7 This is a learning error map for autonomous driving.

[0076] Figure 8 This is a graph showing the reward difference in autonomous driving.

[0077] Figure 9 This is a comparison chart of reward differences in autonomous driving. Detailed Implementation

[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0079] The present invention will be described in detail below with reference to the accompanying drawings. The technical solution adopted by the present invention is a reward learning and control method for autonomous driving based on integrated maximum entropy deep inverse reinforcement learning, which includes the following four steps:

[0080] Step 1: Construct a Markov decision process-based model in the inverse reinforcement learning application scenario of highway autonomous driving, and transform the highway autonomous driving environment into solving a Markov decision process;

[0081] Step 2: To solve the Markov decision process constructed in Step 1, we first construct an improved soft Q-learning model based on value pruning and scaling exponential linear unit activation functions to obtain expert demonstrations, and decouple the inverse reinforcement learning task into sub-tasks according to expert preferences.

[0082] Step 3: Under the expert demonstrations of each subtask obtained in Step 2, establish a strong learner ensemble model, solve the reward function of each subtask through maximum entropy deep inverse reinforcement learning, and use the ensemble algorithm to combine the deep maximum entropy weak learners into a strong learner.

[0083] Step 4: The reward functions of each subtask learned in Step 3 are fused together using a linear combination method, thereby improving the learning accuracy of the autonomous driving reward function.

[0084] Specifically:

[0085] In step one, the reinforcement learning problem for autonomous driving on highways is modeled as a Markov decision process consisting of quintuples. Where S represents the state space of the autonomous vehicle, s i ∈S; A represents the action space of the autonomous vehicle, a i ∈A; Let represent the transition probability of the autonomous vehicle; R represent the reward function of the autonomous vehicle; and γ represent the discount factor. Based on the definition of reinforcement learning models, the inverse reinforcement learning problem of autonomous driving on highways is modeled as a quadruple. like Figure 1The diagram shows a highway autonomous driving model based on maximum entropy deep inverse reinforcement learning using improved soft Q-learning and ensemble algorithms. The highway autonomous driving environment consists of a three-lane highway, a red master vehicle v0, and environment vehicles {v1, v2, v3, v4, v5}. In the highway autonomous driving system, three conditions must be met: 1) the red vehicle runs faster than other environment vehicles; 2) the drivers of different vehicles do not communicate with each other; and 3) different vehicles do not share data. The cooperation process of the red master vehicle can be described as follows: First, the red master vehicle is in its current state s... i Next, take an action π(s) i Secondly, the cars in the highway environment observe the actions of the red master car and take action according to the strategy. Finally, the red master car obtains the next state s through its new position and the cars in the environment before and after v0. i+1 .

[0086] In step two, an entropy term is added to the traditional soft Q-learning algorithm to obtain the optimal objective function for solving the reward function of autonomous driving on highways.

[0087]

[0088] Where R(s) i ,a i ) represents the reward function for autonomous driving, ρ π This represents the state-action edges of the trajectory distribution for autonomous driving. The entropy term is derived through... The calculation is performed to represent the autonomous driving strategy π(·|s) i The randomness in the probability distribution. σ represents the temperature factor that controls the importance of the entropy term, i.e., the randomness of the actions chosen by the autonomous vehicle. The greater the entropy, the greater the randomness of the strategy chosen by the autonomous vehicle, thus ensuring that the autonomous vehicle can explore the state space more thoroughly and prevent it from getting trapped in local optima too early.

[0089] The soft Q-function and soft value function in the inverse reinforcement learning algorithm for autonomous driving on highways are defined as follows:

[0090]

[0091]

[0092] The discount factor γ represents the impact of future rewards on current rewards. The larger γ is, the more steps the autonomous vehicle considers, but the higher the training difficulty.

[0093] Through importance sampling, the soft Bellman error of the soft Q-function of an autonomous vehicle can be expressed in an equivalent form as minimization.

[0094]

[0095] Where d s d represents the distribution of the state space of an autonomous vehicle. a This indicates the distribution of the autonomous vehicle's action space. This represents the target Q value of the autonomous vehicle. This represents the target parameters for autonomous vehicles.

[0096] However, traditional soft Q-learning suffers from gradient explosion and gradient vanishing problems. Therefore, a scaling exponential linear unit activation function is proposed. Compared with the previously used modified linear unit activation function, the left half-axis of this activation function is a soft saturation region, which can solve the problem of neurons not being activated, and the slope λ1 of the right half-axis is 1.05, which can better solve the gradient explosion problem.

[0097]

[0098] Where λ² is approximately 1.67, representing the soft saturation region. The slope.

[0099] During autonomous driving experiments on highways, the value function of the improved soft Q-learning algorithm grew very rapidly, leading to data overflow issues. Therefore, a value pruning method was invented to handle the soft Q-value function, where the soft value function exceeds a defined threshold. The soft Q-value function has been updated to Otherwise, the soft Q-value function remains unchanged.

[0100]

[0101] in This represents the soft Q-value function for autonomous driving after value clipping.

[0102] like Figure 2 The diagram shows the model of the improved soft Q-learning algorithm. The state obtained from the highway autonomous driving environment is used as the input to the improved soft Q-learning algorithm, and the action distribution is used as the output. The higher the probability of an action, the greater the likelihood that the corresponding action will be selected. During the iterative learning process of the improved soft Q-learning algorithm, the learned state-action pairs and reward function are stored in a buffer. Then, after sampling, expert demonstrations are obtained. Based on expert preferences, the complex reward function learning task for autonomous vehicles is decoupled into multiple sub-tasks for reward function learning. Under each expert demonstration sub-task, the reward function and optimal policy are learned.

[0103] In step three, based on the expert demonstration of the autonomous vehicle learned using the improved soft Q-learning algorithm, the method for establishing the strong learner ensemble model is as follows: a weak learner usually refers to a learner whose generalization performance is only slightly better than random guessing. In the maximum entropy deep inverse reinforcement learning framework integrated in the highway autonomous driving environment, through the expert demonstration under the sub-task, each maximum entropy deep inverse reinforcement learning process can be regarded as a weak learner, which takes state features as input and outputs rewards. The optimal network parameters w are calculated by formula (7).

[0104]

[0105]

[0106] in This represents an expert demonstration of the learning capabilities of the self-driving car. Chinese experts demonstrate trajectory The number of Let p(s) represent the partition function. i+1 |s i ,a i ) represents the state transition probability of an autonomous vehicle. Indicates the state of the driverless car s i The characteristic function is denoted by j, which represents the number of subtasks in the learning reward function of the autonomous vehicle.

[0107] According to the chain rule, the data items in formula (7) The complete derivative with respect to the network parameter scalar w consists of two parts: one is the data term. The other is the derivative of the reward function r(w) with respect to the parameter w.

[0108]

[0109] in This indicates the efficiency of gradient backpropagation in deep neural networks during maximum entropy deep inverse reinforcement learning. This represents the expected number of experiential state visits during expert demonstrations of autonomous vehicles. This represents the expected number of state visits learned by the autonomous vehicle under expert demonstration. If the reward function is positive, c is set to 0.01; otherwise, c is set to -0.01.

[0110] In expert demonstration The maximum entropy deep inverse reinforcement learning algorithm was used to learn the reward function of an autonomous vehicle. It is used as a weak learner. The input of a weak learner is a feature function and weights, and the output is a reward function.

[0111]

[0112] Where w represents the weight of the reward function for the autonomous vehicle. The characteristic function representing an autonomous vehicle, This shows the trajectory of an expert demonstration of a self-driving car.

[0113] During the learning process of the weak learner in highway autonomous driving, the ensemble algorithm performs the following operations:

[0114] Step 1: In the ensemble maximum entropy deep inverse reinforcement learning model for autonomous vehicles, the weight distribution of the reward function learned through the ensemble maximum entropy deep inverse reinforcement learning method is first initialized as: V1 = (w 1,1 ,w 1,2 ,…,w 1,I ), i = 1, 2, ..., I.

[0115] Step 2: Utilize the current distribution Expert demonstration of self-driving car training data to learn weak learners Weak learner is calculated The learning error rate

[0116]

[0117] in This indicates the expected frequency of state visits by the learner in the autonomous vehicle. This indicates the expected frequency of access to the experiential state for autonomous vehicles.

[0118] Step 3: Weak Learner coefficient The calculation is as follows:

[0119]

[0120] Step 3: Update the weights of the autonomous vehicle's reward function to prepare for the next weak learner.

[0121]

[0122] Where α is set to 0.01, This represents the normalization factor. Ensemble learning was used to increase the weight of data from expert demonstrations that the autonomous vehicle learned from incorrect examples, while decreasing the weight of data from expert demonstrations that learned from correct examples.

[0123] In the complex nonlinear environment of autonomous driving on highways, the maximum entropy deep inverse learning reward method suffers from computational complexity and overfitting, affecting the learning accuracy of the reward. To prevent these problems, a gradient truncation method and a correction factor are considered in the maximum entropy deep inverse learning.

[0124] The reward for a weak learner in maximum entropy deep inverse reinforcement learning can be calculated as follows.

[0125]

[0126] in For gravity parameters, The larger the value, the greater the sparsity of the gradient cutoff method. And... ζ is the threshold for the gradient cutoff method, and ζ is the gravity parameter for the gradient cutoff method. The coefficient.

[0127] The indicator function TG for the gradient cutoff method is calculated as follows:

[0128]

[0129] During training, the AdaBoost algorithm exponentially increases the weights of weaker reward performers. Therefore, subsequent weak learners tend to learn these weighted rewards, making them more susceptible to noise. To prevent this problem, a correction factor v is added to the linear combination of weak learners, forming the coefficients of the weak learners.

[0130] Furthermore, ensemble algorithms combine multiple weak learners into a strong learner through linearly weighted voting, thereby solving for the reward function of the subtask.

[0131]

[0132] Where the coefficient The value of the weak learner is indicated by v, and constant regularization is indicated by v.

[0133] In step four, the method for establishing the fusion of reward functions for each subtask of the autonomous vehicle through linear combination is as follows: For multiple subtasks, the reward functions learned by the strong learner are fused into the final reward function through linear combination, which improves the learning accuracy of the reward function and obtains better decision performance.

[0134]

[0135] in Indicates the k-th s The coefficients of the strong learner in each subtask

[0136] Algorithm 1 presents the overall algorithm framework for autonomous driving on highways. Based on expert demonstrations learned using an improved soft Q-learning algorithm, it provides an integrated maximum entropy deep inverse reinforcement learning algorithm to learn the reward function for autonomous driving and implement autonomous driving control.

[0137]

[0138] Performance evaluation of the algorithm of this invention

[0139] To evaluate the performance of the proposed autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning, we used Python 3.6 and PyCharm 2020 to build a simulation environment for simulation analysis. The performance of the proposed algorithm was compared with various baseline algorithms through highway autonomous vehicle driving experiments. The baseline algorithms compared in this invention include basic ensemble maximum entropy inverse reinforcement learning (AME-IRL), maximum entropy deep inverse reinforcement learning (ME-DIRL), and ensemble maximum entropy deep inverse reinforcement learning algorithm (AME-DIRL).

[0140] In a high-speed autonomous driving environment, such as Figure 4 The image shows a soft Q-learning model demonstrated by a learning expert. The input to this model is the current position Po of the red main vehicle. si The length of lane 1 is Ln1 len The length of lane 2, Ln2 len The length of lane 3 is Ln3 len And whether a collision occurred. oi Therefore, the model has a total of Rl states. Ln ×2(Ln+2) = 270, where Rl represents the road length and Ln represents the number of lanes. The output is the probability of selecting an action (left, right, stop). Fc1, Fc2, and Fc3 represent the three fully connected layers in the soft Q-learning algorithm model. Figure 5 This represents the value difference map in autonomous driving. ME-DIRL represents the maximum entropy deep inverse reinforcement learning algorithm, and AME-DIRL represents the ensemble maximum entropy deep inverse reinforcement learning algorithm. From Figure 5 It can be seen that, compared to the maximum entropy deep inverse reinforcement learning algorithm, the ensemble maximum entropy deep inverse reinforcement learning algorithm produces a box plot with median, upper quartile, and lower quartile closer to 0, and a smaller interquartile range. Therefore, on the initial expert demonstration dataset generated by the value function, the ensemble algorithm combines multiple weak learners into a strong learner, enhancing the learning ability of the reward function and improving the algorithm's decision-making ability.

[0141] Figure 6The graph represents the difference between the learner's expected state access frequency and expected experienced state access frequency in a highway autonomous driving environment, validating that expert demonstrations generated by the improved soft Q-learning algorithm can improve the performance of the ensemble maximum entropy deep inverse reinforcement learning algorithm invented in this paper. AME-IRL represents the ensemble maximum entropy inverse reinforcement learning algorithm under the initial expert demonstration, and SQL-AME-IRL represents the ensemble maximum entropy inverse reinforcement learning algorithm under the expert demonstration generated by the improved soft Q-learning algorithm. Figure 6 The difference value experiment results show that, within the maximum entropy model framework, compared to the ensemble maximum entropy inverse reinforcement learning algorithm based on the original expert demonstration data, the algorithm proposed in this invention achieves a median, upper quartile, and lower quartile closer to 0 after 10 independent runs, with a smaller interquartile range, indicating lower data dispersion. Because the algorithm proposed in this invention utilizes ensemble algorithms and linear combination methods to fuse reward functions, it overcomes the problems of non-optimal, finite, imbalanced expert demonstrations, and overfitting. Therefore, the reward function learned using this invention for autonomous driving is superior to the baseline algorithm, and the difference in reward function for highway autonomous driving is closer to 0 than that of the baseline algorithm.

[0142] Within the framework of the maximum entropy model, by Figure 7 The experimental results of the learning error of autonomous driving on highways show that the improved soft Q-learning algorithm helps the integrated maximum entropy inverse reinforcement learning to obtain better expert demonstrations. Therefore, the median, upper quartile, and lower quartile of the learning error rate of the maximum entropy inverse reinforcement learning algorithm based on soft Q-learning and the integrated algorithm are smaller. Thus, the algorithm we invented has a smaller learning error in the autonomous driving environment on highways.

[0143] pass Figure 8 The experimental results of the reward function for autonomous driving on highways, as shown, reveal that the line graph of the difference in reward function between ensemble maximum entropy deep inverse reinforcement learning based on soft Q-learning and ensemble algorithms is closer to 0 than the line graph of the difference in reward function between ensemble maximum entropy deep inverse reinforcement learning based on the original expert demonstration data. Therefore, the algorithm proposed in this invention can overcome the shortcomings of non-optimal, finite, and imbalanced expert demonstrations. ME-DIRL represents the ensemble maximum entropy deep inverse reinforcement learning algorithm under the initial expert demonstrations, and SQL-AME-DIRL represents the ensemble maximum entropy deep inverse reinforcement learning algorithm under the expert demonstrations generated by the improved soft Q-learning algorithm. Figure 9Experimental results on the reward discrepancy values ​​of autonomous driving on highways show that, compared to the initial expert demonstration, the reward function discrepancy of the ensemble maximum entropy deep inverse reinforcement learning algorithm under the improved soft Q-learning algorithm has a smaller median, upper quartile, and lower quartile. Therefore, the algorithm proposed in this invention performs better. Compared to the box plot of the reward function discrepancy values ​​obtained by the maximum entropy deep inverse reinforcement learning algorithm, the median of the box plot of the reward function discrepancy values ​​obtained by the algorithm proposed in this invention is closer to 0. However, the box plot of the reward function discrepancy values ​​obtained by the ensemble maximum entropy deep inverse reinforcement learning algorithm based on the original expert demonstration data has 7 outliers, while the box plot of the reward function discrepancy values ​​obtained by the algorithm proposed in this invention has 2 outliers. In summary, the algorithm proposed in this invention can learn a better reward function for autonomous driving on highways, thereby obtaining a better driving control strategy.

[0144] The simulation results from the above high-speed autonomous driving environment show that, compared with the baseline algorithm, our invented algorithm can learn a better reward function and achieve better driving strategy control.

Claims

1. A method for reward learning and control of autonomous driving based on ensemble maximum entropy deep inverse reinforcement learning, characterized in that: It includes the following four steps, Step 1: Construct a Markov decision process-based model in the inverse reinforcement learning application scenario of highway autonomous driving, and transform the highway autonomous driving environment into solving a Markov decision process; Step 2: To solve the Markov decision process constructed in Step 1, we first construct an improved soft Q-learning model based on value pruning and scaling exponential linear unit activation functions to obtain expert demonstrations, and decouple the inverse reinforcement learning task into sub-tasks according to expert preferences. Step 3: Under the expert demonstrations of each subtask obtained in Step 2, establish a strong learner ensemble model, solve the reward function of each subtask through maximum entropy deep inverse reinforcement learning, and use the ensemble algorithm to combine the deep maximum entropy weak learners into a strong learner. Step 4: The reward functions of each subtask in the autonomous driving environment learned in Step 3 are fused together using a linear combination method, thereby improving the learning accuracy of the autonomous driving reward function; In step two, the entropy term is added to the soft Q-learning algorithm to obtain the optimal objective function for solving the reward function of autonomous driving on highways. , (1)； in This represents the reward function for autonomous vehicles. The state-action edges represent the trajectory distribution of the autonomous vehicle; the entropy term is expressed through... Perform calculations to represent the strategy of the autonomous vehicle. Randomness in probability distribution; The temperature factor represents the importance of the entropy term, which controls the randomness of the actions chosen by the autonomous vehicle; the greater the entropy, the greater the randomness of the strategy chosen by the autonomous vehicle. The soft Q-function and soft value function in the inverse reinforcement learning algorithm for autonomous driving on highways are defined as follows: (2)； (3)； Among the discount factors This indicates the impact of future rewards on current rewards, i.e. The larger the size, the more steps the autonomous vehicle needs to take forward. The more; Through importance sampling, the soft Bellman error of the soft Q-function of an autonomous vehicle can be expressed in an equivalent form as minimization. (4)； in This represents the distribution of the state space of an autonomous vehicle. This indicates the distribution of the autonomous vehicle's action space. This represents the target Q value of the autonomous vehicle. Indicates the target parameters of the autonomous vehicle; By scaling the exponential linear unit activation function using the activation function of the neural network, the slope of the right half-axis is selected. Version 1.05 addresses the gradient explosion problem. (5)； in A value of 1.67 indicates a soft saturation region. The slope; During the experiment of autonomous driving on highways, the value function of the improved soft Q-learning algorithm grew very rapidly, resulting in a data overflow problem. Therefore, a value pruning method was invented to handle the soft Q-value function, which is used when the soft value function exceeds a defined threshold. The soft Q-value function is updated to ; Otherwise, the soft Q-value function remains unchanged; (6)； in The soft Q-value function represents the soft Q-value of autonomous driving after value clipping; In the improved soft Q-learning algorithm, the state obtained from the highway autonomous driving environment is used as the input of the improved soft Q-learning algorithm, and the action distribution is used as the output of the improved soft Q-learning algorithm. In the iterative learning process of the improved soft Q-learning algorithm, the learned state-action pairs and reward functions are stored in a buffer; then, after sampling, expert demonstrations of the inverse reinforcement learning algorithm are obtained; based on expert preferences, the complex reward function learning task of the autonomous vehicle is decoupled into multiple sub-tasks for reward function learning; under each expert demonstration sub-task, the reward function and optimal policy of the autonomous vehicle are learned.

2. The autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning according to claim 1, characterized in that: In step one, the reinforcement learning problem for autonomous driving on highways is modeled as a Markov decision process consisting of quintuples. ,in This represents the state space of an autonomous vehicle. ; This indicates the operating space of an autonomous vehicle. ; This represents the transition probability of an autonomous vehicle; Represents the reward function for autonomous vehicles; Indicates the discount factor; Based on the definition of reinforcement learning models, the inverse reinforcement learning problem for autonomous driving on highways is modeled as a quadruple. The highway autonomous driving environment consists of a three-lane highway, a red master vehicle v0, and environmental vehicles {v1, v2, v3, v4, v5}. In the highway autonomous driving system, three conditions must be met: 1) the red vehicle is faster than other environmental vehicles; 2) the drivers of different vehicles do not communicate with each other; and 3) different vehicles do not share data. The cooperation process of the red master vehicle is described as follows: First, the red master vehicle on the highway is in its current state... Next action Secondly, the cars in the highway environment observe the actions of the red master car and take action according to the strategy; finally, the red master car obtains the next state through the new position and the cars in the environment before and after v0. .

3. The autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning according to claim 1, characterized in that: In step three, based on the expert demonstrations of the autonomous vehicle learned using the improved soft Q-learning algorithm, the method for establishing the strong learner ensemble model is as follows: a weak learner typically refers to a learner whose generalization performance is only slightly better than random guessing; in the maximum entropy deep inverse reinforcement learning framework integrated in the highway autonomous driving environment, each maximum entropy deep inverse reinforcement learning process is regarded as a weak learner through expert demonstrations under sub-tasks, taking state features as input and outputting rewards; optimal network parameters... Calculated using formula (7), (7)； in Expert demonstration Chinese experts demonstrate trajectory The number of Represents the partition function. This represents the state transition probability of an autonomous vehicle. Indicates the autonomous vehicle status , characteristic function The number of subtasks for learning the reward function for autonomous vehicles; According to the chain rule, the data items in formula (7) scalar of network parameters The complete derivative consists of two parts: one is the data term. For reward function The derivative of the function, and the other is the reward function. For parameters The derivative; (8)； in This indicates the efficiency of gradient backpropagation in deep neural networks during maximum entropy deep inverse reinforcement learning. This represents the expected number of experiential state visits during expert demonstrations of autonomous vehicles. This represents the expected number of state visits learned by the autonomous vehicle under expert demonstration; if the reward function is positive, then... Set to 0.01; otherwise It was set to -0.01; In expert demonstration The maximum entropy deep inverse reinforcement learning algorithm was used to learn the reward function of an autonomous vehicle. This is considered a weak learner; the input of a weak learner is a feature function and weights, and the output is a reward function. (9)； Where w represents the weight of the reward function for the autonomous vehicle. The characteristic function representing an autonomous vehicle, This shows the trajectory of an expert demonstration of a self-driving car.

4. The autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning according to claim 1, characterized in that: During the learning process of the weak learner in highway autonomous driving, the ensemble algorithm performs the following operations: Step 1: In the ensemble maximum entropy deep inverse reinforcement learning model for autonomous vehicles, the weight distribution of the reward function learned through the ensemble maximum entropy deep inverse reinforcement learning method is first initialized as follows: ; Step 2: Utilize the current distribution Expert demonstration of self-driving car training data to learn weak learners The weak learner is calculated. The learning error rate (10)； in This indicates the expected frequency of state visits by the learner in the autonomous vehicle. This represents the expected frequency of accessing the experiential state for the autonomous vehicle. Step 3: Weak Learner coefficient The calculation is as follows: (11)； Step 3: Update the weights of the autonomous vehicle's reward function to prepare for the next weak learner. (12)； in It was set to 0.

01. This represents the normalization factor; ensemble learning was used to increase the weight of expert demonstrations that the autonomous vehicle learned from incorrect examples and decrease the weight of expert demonstrations that learned from correct examples. In the complex nonlinear environment of autonomous driving on highways, the maximum entropy deep inverse learning reward process suffers from computational complexity and overfitting, which affects the learning accuracy of the reward. To prevent these problems, the invented maximum entropy deep inverse learning method incorporates gradient truncation and correction factors. The reward for a weak learner in maximum entropy deep inverse reinforcement learning can be calculated as follows. (13)； in For gravity parameters, The larger the value, the greater the sparsity of the gradient cutoff method; and The threshold for the gradient cutoff method. Gravity parameters for the gradient cutoff method The coefficient; Indicator function of gradient cutoff method The calculation is as follows: (14)； During training, the AdaBoost algorithm exponentially increases the weights of weaker reward performers; therefore, the next weak learner will be biased towards learning these increased weights, making them susceptible to noise interference. To prevent this problem, a correction factor is added to the linear combination of weak learners. , which constitutes the coefficient of a weak learner; Furthermore, ensemble algorithms combine multiple weak learners into a strong learner through linearly weighted voting, thereby solving for the reward function of the subtask. (15)； Where the coefficient This indicates the importance of weak learners. This indicates constant regularization.

5. The autonomous driving reward learning and control method based on ensemble maximum entropy deep inverse reinforcement learning according to claim 1, characterized in that: In step four, the method for establishing the fusion of reward functions of various sub-tasks in the autonomous driving environment through linear combination is as follows: For multiple sub-tasks, the reward functions learned by the strong learner are fused into the final reward function through linear combination, which improves the learning accuracy of the reward function and obtains better decision performance. (16)； in Indicates the first The coefficients of the strong learner in each subtask .

Images