A Deep Reinforcement Learning Approach to Reduce Vehicle Hyperparameter Tuning

The deep reinforcement learning method addresses the challenge of constructing nonlinear reward functions for autonomous vehicles by integrating handling stability rewards with driving rules and using multi-process asynchronous training, enhancing vehicle stability and control in complex conditions.

CN118569350BInactive Publication Date: 2024-12-06CHANGCHUN UNIV OF TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411023989.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing autonomous driving technologies face challenges in constructing nonlinear reward functions for vehicle hyperparameters, particularly in complex conditions, which affect vehicle handling stability and tasks like lane keeping and overtaking.

Method used

A deep reinforcement learning method using an integrated decision-making and control strategy that combines nonlinear handling stability rewards with driving rules, employing a double-layer deep inverse reinforcement learning structure and multi-process asynchronous training to optimize vehicle control.

Benefits of technology

This approach simplifies the construction of handling stability reward functions, enhances training efficiency, and ensures safe vehicle operation in highway scenarios by improving the stability and control of autonomous vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118569350B_ABST
    Figure CN118569350B_ABST
Patent Text Reader

Abstract

A deep reinforcement learning method for reducing vehicle hyperparameter adjustment relates to the technical field of reducing hyperparameter adjustment for autonomous driving vehicles based on deep reinforcement learning. A nonlinear reward learning architecture for vehicle handling stability based on deep inverse reinforcement learning is proposed, and an autonomous driving integrated decision-making and control strategy for the nonlinear handling stability reward and driving rule reward of the vehicle under highway scenarios is further proposed, which greatly reduces the hyperparameter setting of the vehicle handling stability reward function during the training of the integrated decision-making and control strategy. The steps of the present invention are: obtaining the expert strategy required for deep inverse reinforcement learning training; using deep inverse reinforcement learning to obtain vehicle handling stability rewards; using a multi-process asynchronous method to accelerate the training speed of deep inverse reinforcement learning; and integrated decision-making and control of vehicles in high-speed and complex scenarios. The present invention can realize integrated decision-making and control with the ability to maintain vehicle stability in high-speed scenarios without relying on a large number of vehicle stability hyperparameter settings.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention proposes a deep reinforcement learning method for reducing vehicle hyperparameter adjustment, and relates to the technical field of reducing hyperparameter adjustment of autonomous driving vehicles based on deep reinforcement learning. Background Art

[0002] With the rapid development of economic globalization and the rapid expansion of urbanization, the number of cars has increased dramatically, but the incidence of traffic accidents has also increased accordingly. Autonomous driving is expected to make significant contributions to reducing traffic accidents, improving road efficiency and alleviating traffic congestion. However, this technology faces many technical challenges in its implementation. In order to ensure the driving safety of autonomous vehicles in highway environments, it is necessary to build an algorithm based on deep reinforcement learning and design a reward function to evaluate the impact of vehicle hyperparameters during training, such as the impact of hyperparameters such as speed, yaw angle and steering wheel angle on vehicle handling stability. To ensure vehicle stability, it is also necessary to quickly achieve complex tasks such as lane keeping, lane change and overtaking. At present, most handling stability reward functions rely on empirical assumptions and vehicle dynamics models. The setting of reward functions for a large number of hyperparameters has become a problem, especially for vehicles with tires in complex working conditions. The nonlinear reward function of hyperparameters is even more difficult to adjust. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a nonlinear reward learning architecture for vehicle handling stability based on deep inverse reinforcement learning, and further proposes an integrated decision-making and control strategy for autonomous driving that combines the nonlinear handling stability reward of the vehicle with the driving rule reward in highway scenarios, thereby greatly reducing the hyperparameter setting of the vehicle handling stability reward function during the training of the integrated decision-making and control strategy.

[0004] A deep reinforcement learning method for reducing vehicle hyperparameter adjustment according to the present invention includes the following specific related steps:

[0005] Step S1: Obtain the expert strategy required for deep inverse reinforcement learning training;

[0006] Step S2: Using deep inverse reinforcement learning to obtain vehicle handling stability rewards;

[0007] Step S3: Use a multi-process asynchronous method to accelerate the training speed of deep inverse reinforcement learning;

[0008] Step S4: Integrated vehicle decision-making and control in high-speed complex scenarios.

[0009] Step S1 specifically includes the following contents:

[0010] Step S11: The original expert data is the vehicle handling stability evaluation value P corresponding to different vehicle slip rates λ at different front wheel steering angles, longitudinal vehicle speeds, and road friction coefficients.eff , P eff The larger the value, the better the vehicle's handling stability. Each set of different front wheel steering angles, longitudinal vehicle speeds, and fixed values ​​of road friction coefficient is a working condition. Each working condition corresponds to a λ-P eff In order to obtain the expert strategy in the form of probability distribution function, the original expert data needs to be preprocessed to prevent numerical overflow. eff Too large, causing numerical calculation overflow, P in each working condition eff All need to subtract the minimum P under the current working conditions eff The value is set as follows:

[0011] P eff ←P eff -min(P eff )

[0012] Step S12: Use the softmax method to calculate the P values ​​under each working condition. eff The value is converted to a probability distribution Q, and the sum of the probabilities is 1. The expression is set as shown below:

[0013] Q(λ)=softmax(P eff )

[0014]

[0015] Step S13: Use the parameterized normal distribution P θ To approximate Q, that is, λ obeys the normal distribution P(μ θ ,σ θ ),μ θ and σ θ is the mean and variance of the normal distribution. The specific approximation method is achieved by minimizing the KL divergence. The expression is set as shown in the following formula:

[0016]

[0017] Step S14: The specific method of calculating the parameter θ is to use the following gradient descent algorithm to achieve the optimal solution for θ:

[0018]

[0019] Where α is the learning rate during parameter update;

[0020] Step S15: When the probability distribution function P θ The peak probability between the original probability distribution Q and the optimal slip rate λ best When there is an error, it is necessary to θ Make a correction and change P θThe mean of is corrected to the optimal slip rate corresponding to the probability distribution Q, while keeping the variance unchanged. The corrected probability distribution P′ θ The final expert strategy π E =P′ θ .

[0021] Step S2 specifically includes the following contents:

[0022] Step S21: Calculate the feature expectation μ(π E ) and μ(π), for a set of expert strategy data D, the expression is set as follows:

[0023]

[0024] here represents the i-th expert trajectory, s is the state of the vehicle, and the length of the expert trajectory is k. Therefore, the feature expectation of the expert strategy is calculated as follows:

[0025]

[0026] Here γ is the discount coefficient 0<γ<1, φ is the state feature, φ is 1 when s meets a specific state, otherwise it is 0, and the learning strategy μ(π) is the feature expectation of the vehicle during the deep reinforcement learning training process, and the calculation method is consistent with the feature expectation of the expert strategy;

[0027] Step S22: Use deep inverse reinforcement learning to obtain the vehicle handling stability reward. The handling stability reward function is approximated by a neural network, that is, r ts (s,λ;θ), where θ is the parameter of the neural network, s,λ is the input of the neural network, and the reward r is ts The problem of obtaining the parameter θ is transformed into an optimization problem, and the expression is set as shown in the following formula:

[0028]

[0029] Here θ Reward for stable operation ts The probability value of the parameter, the above formula is equivalent to a maximum a posteriori estimation problem, which can be regarded as the likelihood function L of the expert strategy data D The combination of the parameter prior θ,

[0030]

[0031] By simplifying,

[0032]

[0033] Here L2 represents the reward r ts The regularization function of the parameters, μ(π E ) and μ(π) are both calculated within a time range Δk. Considering Δk = 1, we have μ(π E )-μ(π)=π E -π;

[0034] Step S23: Use the gradient descent algorithm to solve the parameter θ, and set the expression as shown in the following formula:

[0035]

[0036] By setting the learning rate to β, the optimal stable reward r can be obtained ts .

[0037] Step S3 specifically includes the following contents:

[0038] Step S31: Multi-process asynchronous training makes full use of the multi-core CPU performance of a single computer and distributes training tasks to multiple CPU cores, thereby speeding up training and reducing training time. The multi-process asynchronous training learning environment is mainly divided into two parts: the main process and the sub-process. There are multiple sub-processes, each of which has two neural networks, Actorl strategy network and Criticl Value Network There is only one main process, and there are two neural networks, Actorg strategy network and Criticg Value Network These neural networks have the same structure, but the parameter update method is different, where s is the state of the vehicle and a is the slip ratio of the vehicle;

[0039] Step S32: Initialize the network parameters of Actorg and Criticg, and assign the network parameters to Actorl and Criticl respectively, set the subprocess update step number K, the maximum number of training rounds N, the maximum capacity of the experience pool C, and initialize the action step number k=0, the training round number i=0;

[0040] Step S33: In each sub-process, the current vehicle state s t According to the strategy Select action a t , calculate the vehicle state s at the next moment t+1 , then through the state s t ,a t Calculate the vehicle's reward value r t (s t ,a t ), k=k+1, and the data (s t ,a t ,s t+1 ,rt , done) is put into the experience pool. Each time a new data is put into the experience pool, it is saved in the first position, and the original data is moved back in sequence. Here done indicates whether this round of training is finished during the training process. If it is finished, done = True, otherwise done = False. In the child process, when done = True, i = i + 1;

[0041] Step S34: In each subprocess, whenever k=K, data is randomly extracted from the experience pool to calculate the Actor1 and Critic1 network loss functions, and the expressions are set as shown below:

[0042]

[0043] Among them H is strategy Entropy, ε is the clipping coefficient ε∈(0.1,0.5), clip(r(x),1-ε,1+ε) represents the clipping function r(x)∈[1-ε,1+ε], γ is the discount factor γ∈(0,1);

[0044] Step S35: Whenever the Actor1 and Critic1 loss functions are calculated, the gradients of the two networks are calculated, and the Actorg and Criticg network parameters x and w are updated. The expressions are set as shown below:

[0045]

[0046] Here a and b are the learning rates of the gradient descent algorithm. Each subprocess can independently update the parameters of the two networks in the main process, thereby realizing asynchronous updates. Whenever a subprocess updates the parameters in the main process, k needs to be reassigned to 0, k = 0, and the network parameters in the main process need to be assigned to the network in the subprocess that provides gradient information, x Actorl ←x Actorg , w Criticl ←w Criticg ;

[0047] Step S36: When i=N, the training process ends. is the optimal strategy, and then Returning to step S22, the parameter θ update process of deep inverse reinforcement learning is added, thereby realizing the process of accelerating deep inverse reinforcement learning training using a multi-process asynchronous method.

[0048] Step S4 specifically includes the following contents:

[0049] Step S41: The vehicle decision-making problem in the high-speed scene is regarded as a Markov problem, and the tuple (S, A, P, r, γ) is used to represent the integrated decision-making of the autonomous driving vehicle in the high-speed scene, where S is the state space, A is the action space of the autonomous driving vehicle, P is the probability of environmental state transition, r is the immediate reward of the vehicle action, and γ∈(0,1) is the discount factor. In the Markov decision process, the goal of the vehicle is to maximize its own benefits through the strategy π:S→A

[0050] Step S42: By designing multiple sub-rewards and calculating the linear combination of multiple sub-rewards to guide the vehicle to achieve integrated decision-making control, the road progress reward r cp , speed reward r v , heading angle reward r ha , road center deviation reward r loc , distance keeping reward r d , vehicle torque distribution reward r T , steering wheel angle reward r dwa , vehicle handling reward ts , the total reward r = d1r cp +d2r v +d3r ha +d4r loc +d5r d +d6r T +d7r dwa +d8r ts ;

[0051] Step S43: By creating a neural network that shares the parameters of the Actor and Critic networks, a neural network is implemented to output the integrated control strategy π and the value function V for evaluating the strategy at one time. The loss function expression is as follows:

[0052]

[0053] Where L is calculated π Such as L Actorl (x), calculate L V Such as L Criticl (w), ω is the neural network parameter, τ is the hyperparameter;

[0054] Step S44: Update the neural network parameters by using the gradient descent algorithm, and set the expression as shown in the following formula:

[0055]

[0056] Where μ is the learning rate hyperparameter.

[0057] The beneficial effects of the present invention are:

[0058] 1. A vehicle handling stability control strategy based on deep inverse reinforcement learning is proposed. Aiming at the difficulty of constructing the handling stability reward function in autonomous driving, the double-layer structure of upper-layer deep inverse reinforcement learning and lower-layer deep reinforcement learning is used to realize the learning of the vehicle handling stability reward function and the final vehicle handling stability control. This method not only simplifies the construction of the handling stability reward function and expands the research methods of vehicle handling stability control, but also provides important technical support for the safe driving of autonomous vehicles in highway scenarios.

[0059] 2. A multi-process asynchronous parallel training method is introduced. This method realizes parallel processing of data and rapid updating of control strategies by running agents on multiple CPU cores simultaneously. In this way, agents can learn and explore independently in different environment instances, which significantly improves training efficiency. In addition, the multi-process learning environment and algorithm are designed in detail to ensure the stability of the training process and the convergence of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Expert strategies for partial acquisition;

[0061] Figure 2 It is the overall error change between the expert strategy and the learning strategy during the deep inverse reinforcement learning training process;

[0062] Figure 3 The degree of approximation between the final partial learning strategy of deep inverse reinforcement learning and the expert strategy;

[0063] Figure 4 Training framework for multi-process asynchronous methods;

[0064] Figure 5 The training time difference of deep inverse reinforcement learning in different processes in multi-process asynchronous training;

[0065] Figure 6 To evaluate the vehicle handling stability under different vehicle driving conditions in multi-process asynchronous training;

[0066] Figure 7 A schematic diagram of the driving environment of a vehicle in a high-speed scenario;

[0067] Figure 8 Schematic diagram of the lane keeping task in a high-speed scenario;

[0068] Fig. 9 This is a schematic diagram of the lane changing and overtaking task in a high-speed scenario;

[0069] Fig.10 Integrate decision-making control algorithm neural network structure for high-speed scenarios;

[0070] Fig.11 To integrate the convergence change curve of the decision control algorithm in high-speed scenarios;

[0071] Fig.12 It is the state change curve of the vehicle in the lane keeping task in the high-speed scenario;

[0072] Fig.13 It is the state change curve of the vehicle in the lane changing and overtaking task in the high-speed scenario. DETAILED DESCRIPTION

[0073] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments are merely for explaining the present invention but not for limiting it.

[0074] The specific implementation steps are as follows:

[0075] Step S1: Obtain the expert strategy required for training;

[0076] Step S2: Using deep inverse reinforcement learning to obtain vehicle handling stability rewards;

[0077] Step S3: Use a multi-process asynchronous method to accelerate the training speed of deep inverse reinforcement learning;

[0078] Step S4: Integrated vehicle decision-making and control in high-speed complex scenarios.

[0079] In step S1, in order to obtain the required expert strategy, the following steps need to be performed:

[0080] Step S11: The original expert data is different front wheel steering angles delta f , longitudinal speed speed, road friction coefficient mu, vehicle handling stability evaluation value P corresponding to different vehicle slip rates λ eff , P eff The larger the value, the better the vehicle's handling stability. Each set of different front wheel steering angles, longitudinal vehicle speeds, and fixed values ​​of road friction coefficient is a working condition. Each working condition corresponds to a λ-P eff In order to obtain the expert strategy in the form of probability distribution function, the original expert data needs to be preprocessed to prevent numerical overflow. eff Too large, causing numerical calculation overflow, P in each working condition eff All need to subtract the minimum P under the current working conditions eff The value is set as follows:

[0081] P eff ←P eff -min(P eff )

[0082] Step S12: Use the softmax method to calculate the P values ​​under each working condition. eff The value is converted to a probability distribution Q, and the sum of the probabilities is 1. The expression is set as shown below:

[0083] Q(λ)=softmax(P eff )

[0084]

[0085] Step S13: Use the parameterized normal distribution P θ To approximate Q, that is, λ obeys the normal distribution P(μ θ ,σ θ ),μ θ and σ θ is the mean and variance of the normal distribution. The specific approximation method is achieved by minimizing the KL divergence. The expression is set as shown in the following formula:

[0086]

[0087] Step S14: The specific method of calculating the parameter θ is to use the following gradient descent algorithm to achieve the optimal solution for θ:

[0088]

[0089] Where α is the learning rate during parameter update;

[0090] Step S15: Figure 1 As shown, when the probability distribution function P θ The peak probability between the original probability distribution Q and the optimal slip rate λ best When there is an error, it is necessary to θ Make a correction and change P θ The mean of is corrected to the optimal slip rate corresponding to the probability distribution Q, while keeping the variance unchanged. The corrected probability distribution P′ θ The final expert strategy π E =P′ θ .

[0091] In step S2, deep inverse reinforcement learning is used to obtain vehicle handling stability rewards, which specifically includes the following steps:

[0092] Step S21: Calculate the feature expectation μ(π E ) and μ(π), for a set of expert strategy data D, the expression is set as follows:

[0093]

[0094] here represents the i-th expert trajectory, s is the state of the vehicle, and the length of the expert trajectory is k. Therefore, the feature expectation of the expert strategy is calculated as follows:

[0095]

[0096] Here γ is the discount coefficient 0<γ<1, φ is the state feature, φ is 1 when s meets a specific state, otherwise it is 0, and the learning strategy μ(π) is the feature expectation of the vehicle during the deep reinforcement learning training process, and the calculation method is consistent with the feature expectation of the expert strategy;

[0097] Step S22: Use deep inverse reinforcement learning to obtain the vehicle handling reward. The handling reward function is approximated by a neural network, that is, r ts (s,λ;θ), where θ is the parameter of the neural network, s,λ is the input of the neural network, and the reward r is ts The problem of obtaining the parameter θ is transformed into an optimization problem, and the expression is set as shown in the following formula:

[0098]

[0099] Here θ Reward for stable operation ts The probability value of the parameter, the above formula is equivalent to a maximum a posteriori estimation problem, which can be regarded as the likelihood function L of the expert strategy data D The combination of the parameter prior θ,

[0100]

[0101] By simplifying,

[0102]

[0103] Here L2 represents the reward r ts The regularization function of the parameters, μ(π E ) and μ(π) are both calculated within a time range Δk. Considering Δk = 1, we have μ(π E )-μ(π)=π E -π;

[0104] Step S23: Use the gradient descent algorithm to solve the parameter θ, and set the expression as shown in the following formula:

[0105]

[0106] By setting the learning rate to β, we can get the optimal control stability reward r ts .

[0107] like Figure 2 and Figure 3 As shown in the figure, in the final learning result, the error and approximation between the expert strategy and the learning strategy are evaluated; Figure 2 In the overall error change in , 200 π are sampled E -π The smaller the error sum of the samples, the better the learning effect. Figure 3 In the figure, the actual degree of fit between some expert strategies and the learning strategy under different vehicle speed conditions is plotted.

[0108] In step S3, multi-process asynchronous training is used to accelerate deep inverse reinforcement learning. The specific steps are as follows:

[0109] Step S31: Multi-process asynchronous training makes full use of the multi-core CPU performance of a single computer and distributes training tasks to multiple CPU cores, thereby speeding up training and reducing training time. The multi-process asynchronous training learning environment is mainly divided into two parts: the main process and the sub-process. There are multiple sub-processes, each of which has two neural networks, Actorl strategy network and Criticl Value Network These two networks together form a sub-network; there is only one main process, and there are also two neural networks, Actorg strategy network and Criticg Value Network Together they form the main network. These neural networks have the same structure, but the parameter update methods are different, where s is the state of the vehicle and a is the slip rate of the vehicle;

[0110] Step S32: Initialize the network parameters of Actorg and Criticg, and assign the network parameters to Actorl and Criticl respectively, set the subprocess update step number K, the maximum number of training rounds N, the maximum capacity of the experience pool C, and initialize the action step number k=0, the training round number i=0;

[0111] Step S33: In each sub-process, the current vehicle state s t According to the strategy Select action a t , calculate the vehicle state s at the next moment t+1 , then through the state s t ,a t Calculate the vehicle's reward value r t (s t ,a t ), k=k+1, and the data (s t ,a t ,s t+1 ,r t, done) is put into the experience pool. Each time a new data is put into the experience pool, it is saved in the first position, and the original data is moved back in sequence. Here done indicates whether this round of training is finished during the training process. If it is finished, done = True, otherwise done = False. In the child process, when done = True, i = i + 1;

[0112] Step S34: In each subprocess, whenever k=K, data is randomly extracted from the experience pool to calculate the Actor1 and Critic1 network loss functions, and the expressions are set as shown below:

[0113]

[0114] Among them H is strategy Entropy, ε is the clipping coefficient ε∈(0.1,0.5), clip(r(x),1-ε,1+ε) represents the clipping function r(x)∈[1-ε,1+ε], γ is the discount factor γ∈(0,1);

[0115] Step S35: Whenever the Actor1 and Critic1 loss functions are calculated, the gradients of the two networks are calculated, and the Actorg and Criticg network parameters x and w are updated. The expressions are set as shown below:

[0116]

[0117] Here a and b are the learning rates of the gradient descent algorithm. Each subprocess can independently update the parameters of the two networks in the main process, thereby realizing asynchronous updates. Whenever a subprocess updates the parameters in the main process, k needs to be reassigned to 0, k = 0, and the network parameters in the main process need to be assigned to the network in the subprocess that provides gradient information, x Actorl ←x Actorg , w Criticl ←w Criticg , the specific multi-process asynchronous training framework is as follows Figure 4 As shown;

[0118] Step S36: When i=N, the training process ends. is the optimal strategy, and then Returning to step S22, a parameter θ updating process of deep inverse reinforcement learning is added, thereby realizing the process of accelerating deep inverse reinforcement learning training using a multi-process asynchronous method;

[0119] exist Figure 5The difference in training time of deep inverse reinforcement learning in different processes in multi-process asynchronous training is shown in Figure 2. As the number of sub-processes increases, the total time of deep inverse reinforcement learning gradually decreases. Figure 6 The 3D diagram of the vehicle's handling stability under different vehicle conditions is shown in the figure. The safety value represents the vehicle's handling stability. The larger the safety value, the better the vehicle performance.

[0120] In step S4, the vehicle integrated decision-making control in high-speed complex scenes is as follows:

[0121] Step S41: The vehicle decision-making problem in the high-speed scene is regarded as a Markov problem, and the tuple (S, A, P, r, γ) is used to represent the integrated decision-making of the autonomous driving vehicle in the high-speed scene, where S is the state space, A is the action space of the autonomous driving vehicle, P is the probability of environmental state transition, r is the immediate reward of the vehicle action, and γ∈(0,1) is the discount factor. In the Markov decision process, the goal of the vehicle is to maximize its own benefits through the strategy π:S→A Among them, the high-speed scene environment of vehicle driving is as follows Figure 7 As shown in the figure, the lane keeping task is divided into Figure 8 and lane change and overtaking tasks such as Fig. 9 , the width of each lane is 3.2m;

[0122] Step S42: By designing multiple sub-rewards and calculating the linear combination of multiple sub-rewards to guide the vehicle to achieve integrated decision-making control, the road progress reward r cp , speed reward r v , heading angle reward r ha , road center deviation reward r loc , distance keeping reward r d , vehicle torque distribution reward r T , steering wheel angle reward r dwa , vehicle handling stability reward r ts , the total reward r = d1r cp +d2r v +d3r ha +d4r loc +d5r d +d6r T +d7r dwa +d8r ts ;

[0123] Step S43: Create a neural network that shares the Actor and Critic network parameters such as Fig.10, so that a neural network can output the integrated control strategy π and the value function V for evaluating the strategy at one time, and the feature extraction of state information is realized in the hidden layer. The loss function expression is shown as follows:

[0124]

[0125] Where L is calculated π Such as L Actorl (x), calculate L V Such as L Criticl (w), ω is the neural network parameter, τ is the hyperparameter; Fig.11 In the figure, the convergence of the algorithm during the training process is shown. As the number of training times increases, the return value (i.e., reward) obtained by the vehicle gradually increases and tends to converge, and the loss function gradually decreases and tends to converge;

[0126] Step S44: Update the neural network parameters by using the gradient descent algorithm, and set the expression as shown in the following formula:

[0127]

[0128] Where μ is the learning rate hyperparameter;

[0129] like Fig.12 As shown in the figure, the state change curve of the vehicle in the lane keeping task at a speed of 30m / s is shown in the figure. In order to compare the state change curve of the vehicle in the lane keeping task with the handling stability reward r ts (i.e., the reward for steady handling in the figure) The performance of the vehicle is specially added with the same training without r ts Vehicle performance at the time; Fig.12 It can be seen that in order to counteract the deviation characteristics of the vehicle, after adding r ts The vehicle performs better when the vehicle is in a more stable position, both in terms of reducing lane centerline deviation and maintaining a low yaw rate.

[0130] like Fig.13 As shown in the figure, it is the state change curve of the vehicle in the lane changing and overtaking task. It can be seen that in order to implement the lane changing and overtaking task on the road, the vehicle has r ts Its performance is better when there is no r ts When the vehicle is more likely to run off the track; ts When the vehicle's slip rate is maintained better, the tires are less likely to slip, and the acceleration changes more smoothly. ts The vehicle's slip rate is maintained worse, the tires are more likely to slip, and its acceleration changes are very drastic, which is not conducive to maintaining the vehicle's handling stability.

Claims

1. A deep reinforcement learning method to reduce vehicle hyperparameter tuning, It is characterized in that The method comprises the following steps: Step S1: Obtain the expert strategy required for training; Step S2: Using deep inverse reinforcement learning to obtain vehicle handling stability rewards; Step S3: Use a multi-process asynchronous method to accelerate the training speed of deep inverse reinforcement learning, specifically including the following steps: Step S31: Construct a multi-process asynchronous training learning environment. Multi-process asynchronous training fully utilizes the multi-core CPU performance of a single computer and distributes training tasks to multiple CPU cores, thereby speeding up training and reducing training time. The multi-process asynchronous training learning environment is mainly divided into two parts: the main process and the sub-process. There are multiple sub-processes, and each sub-process has two neural networks, Actorl strategy network and Criticl Value Network There is only one main process, and there are two neural networks, Actorg strategy network and Criticg Value Network These neural networks have the same structure, but the parameter update method is different, where s is the state of the vehicle and a is the slip ratio of the vehicle; Step S32: Initialize the network parameters of Actorg and Criticg, and assign the network parameters to Actorl and Criticl respectively, set the subprocess update step number K, the maximum number of training rounds N, the maximum capacity of the experience pool C, and initialize the action step number k=0, the training round number i=0; Step S33: In each sub-process, the current vehicle state s t According to the strategy Select action a t , calculate the vehicle state s at the next moment t+1 , then through the state s t ,a t Calculate the vehicle's reward value r t (s t ,a t ), k=k+1, and the data (s t ,a t ,s t+1 ,r t , done) is put into the experience pool. Each time a new piece of data is put into the experience pool, it is saved in the first position, and the original data is moved back in sequence. Here done indicates whether this round of training is completed during the training process. If it is completed, done = True, otherwise done = False. In the child process, when done = True, i = i + 1; Step S34: In each subprocess, whenever k=K, data is randomly extracted from the experience pool to calculate the Actor1 and Critic1 network loss functions, and the expressions are set as shown below: Among them H is strategy Entropy, ε is the clipping coefficient ε∈(0.1,0.5), clip(r(x),1-ε,1+ε) represents the clipping function r(x)∈[1-ε,1+ε], γ is the discount factor γ∈(0,1); Step S35: Whenever the Actor1 and Critic1 loss functions are calculated, the gradients of the two networks are calculated, and the Actorg and Criticg network parameters x and w are updated. The expressions are set as shown below: Here a and b are the learning rates of the gradient descent algorithm. Each subprocess can independently update the parameters of the two networks in the main process, thereby realizing asynchronous updates. Whenever a subprocess updates the parameters in the main process, k needs to be reassigned to 0, k = 0, and the network parameters in the main process need to be assigned to the network in the subprocess that provides gradient information, x Actorl ←x Actorg , w Criticl ←w Criticg ; Step S36: When i=N, the training process ends. is the optimal strategy, and then Returning to step S22, a parameter θ updating process of deep inverse reinforcement learning is added, thereby realizing the process of accelerating deep inverse reinforcement learning training using a multi-process asynchronous method; Step S4: Vehicle integrated decision-making and control in high-speed complex scenarios, specifically including the following steps: Step S41: The vehicle decision-making problem in the high-speed scene is regarded as a Markov problem, and the tuple (S, A, P, r, γ) is used to represent the integrated decision-making of the autonomous driving vehicle in the high-speed scene, where S is the state space, A is the action space of the autonomous driving vehicle, P is the probability of environmental state transition, r is the immediate reward of the vehicle action, and γ∈(0,1) is the discount factor. In the Markov decision process, the goal of the vehicle is to maximize its own benefits through the strategy π:S→A Step S42: By designing multiple sub-rewards and calculating the linear combination of multiple sub-rewards to guide the vehicle to achieve integrated decision-making control, the road progress reward r cp , speed reward r v , heading angle reward r ha , road center deviation reward r loc , distance keeping reward r d , vehicle torque distribution reward r T , steering wheel angle reward r dwa , vehicle handling reward ts , the total reward r = d 1 r cp +d 2 r v +d 3 r ha +d 4 r loc +d 5 r d +d 6 r T +d 7 r dwa +d 8 r ts ; Step S43: By creating a neural network that shares the parameters of the Actor and Critic networks, a neural network is implemented to output the integrated control strategy π and the value function V for evaluating the strategy at one time. The loss function expression is as follows: Where L is calculated π Such as L Actorl (x), calculate L V Such as L Criticl (w), ω is the neural network parameter, τ is the hyperparameter; Step S44: Update the neural network parameters by using the gradient descent algorithm, and set the expression as shown in the following formula: Where μ is the learning rate hyperparameter.

2. A deep reinforcement learning method for reducing vehicle hyperparameter adjustment according to claim 1, It is characterized in that The original expert data in step S1 is the vehicle handling stability evaluation value P corresponding to different vehicle slip rates λ under different front wheel steering angles, longitudinal vehicle speeds and road friction coefficients. eff , P eff The larger the value, the better the vehicle's handling stability. Each set of different front wheel steering angles, longitudinal vehicle speeds, and fixed values ​​of road friction coefficient is a working condition. Each working condition corresponds to a λ-P eff When using the curve, the above original expert data needs to be converted into a form described by a probability distribution function, so the following processing is required: Step S11: To prevent P eff Too large, causing numerical calculation overflow, P in each working condition eff All need to subtract the minimum P under the current working conditions eff The value is set as follows: P eff ←P eff -min(P eff ) Step S12: Use the softmax method to calculate the P values ​​under each working condition. eff The value is converted to a probability distribution Q, and the sum of the probabilities is 1. The expression is set as shown below: Q(λ)=softmax(P eff ) Step S13: Use the parameterized normal distribution P θ To approximate Q, that is, λ follows the normal distribution P(μ θ ,σ θ ),μ θ and σ θ is the mean and variance of the normal distribution. The specific approximation method is achieved by minimizing the KL divergence. The expression is set as shown in the following formula: Step S14: The specific method of calculating the parameter θ is to use the following gradient descent algorithm to achieve the optimal solution for θ: Where α is the learning rate during parameter update; Step S15: When the probability distribution function P θ The peak probability between the original probability distribution Q and the optimal slip rate λ best When there is an error, it is necessary to θ Make a correction and change P θ The mean of is corrected to the optimal slip rate corresponding to the probability distribution Q, while keeping the variance unchanged. The corrected probability distribution P′ θ The final expert strategy π E =P′ θ .

3. A deep reinforcement learning method for reducing vehicle hyperparameter adjustment according to claim 1, It is characterized in that The specific process of using deep inverse reinforcement learning to obtain vehicle handling stability rewards in step S2 is as follows: Step S21: Calculate the feature expectation μ(π E ) and μ(π), for a set of expert strategy data D, the expression is set as follows: here represents the i-th expert trajectory, s is the state of the vehicle, and the length of the expert trajectory is k. Therefore, the feature expectation of the expert strategy is calculated as follows: Here γ is the discount factor 0<γ<1, φ is the state feature, φ is 1 when s meets a specific state, otherwise it is 0, and the learning strategy μ(π) is the feature expectation of the vehicle during the deep reinforcement learning training process, and the calculation method is consistent with the feature expectation of the expert strategy; Step S22: Use deep inverse reinforcement learning to obtain the vehicle handling reward. The handling reward function is approximated by a neural network, that is, r ts (s,λ;θ), where θ is the parameter of the neural network, s,λ is the input of the neural network, and the reward r is ts The problem of obtaining the parameter θ is transformed into an optimization problem, and the expression is set as shown in the following formula: Here θ Reward for stable operation ts The probability value of the parameter, the above formula is equivalent to a maximum a posteriori estimation problem, which can be regarded as the likelihood function L of the expert strategy data D The combination of the parameter prior θ, By simplifying, Here L 2 Reward r ts The regularization function of the parameters, μ(π E ) and μ(π) are both calculated within a time range Δk. Considering Δk = 1, we have μ(π E )-μ(π)=π E -π; Step S23: Use the gradient descent algorithm to solve the parameter θ, and set the expression as shown in the following formula: By setting the learning rate to β, the optimal stable reward r can be obtained ts .

Citation Information

Patent Citations

  • Multi-model training method and device, electronic equipment and storage medium

    CN112541570A

  • Composite working condition chassis integrated control method based on handling stability probability distribution

    CN115525019A

  • Redundant vehicle control system based on tire sensor load estimation

    CN116215559A

  • Lane keeping prediction control method and system for man-machine co-driving

    CN117734725A

  • Techniques for navigating an autonomous vehicle based on perceived risk

    GB202117359D0