Vehicle path tracking control method based on improved deep reinforcement learning

Abstract

The invention discloses a vehicle path tracking control method based on improved deep reinforcement learning. The method comprises the steps that a path tracking control optimization model of the 4WIS-4WID vehicle is established, then current actions of four wheels of the 4WIS-4WID vehicle are input and processed, an improved deep reinforcement learning DRL controller based on a swarm intelligence collaborative experience playback mechanism is used for training the model, and the model outputs optimized actions; after the optimized action is transmitted to the vehicle, action control over four wheels is conducted, then the next action of the vehicle is updated, and the steps are repeated to achieve real-time tracking control over the vehicle path. According to the method, the priority of the samples can be adaptively adjusted, exploration and utilization are balanced, the sample utilization efficiency is improved, and the convergence speed and generalization ability of a deep reinforcement learning algorithm are improved. According to the method, importance sampling correction is introduced, so that the deviation caused by priority sampling introduction is effectively reduced, and the stability of 4WIS-4WID vehicle path tracking control is improved.

Description

Technical Field

[0001] The present invention relates to a vehicle path tracking control method, and in particular to a vehicle path tracking control method based on improved deep reinforcement learning. Background Art

[0002] Deep reinforcement learning (DRL) combines the advantages of reinforcement learning (RL) and deep learning (DL), and has shown broad application potential in systems that require real-time control. In the DRL framework, the agent is modeled through a deep neural network and interacts continuously with the environment and control system. Each action is rewarded based on the feedback from the environment, and the training goal is to maximize the total reward accumulated over time. Through continuous interactive learning with the environment and control system, DRL can adapt to the decision-making needs of four-wheel independent steering and independent drive (4WIS-4WID) vehicles under various external disturbances.

[0003] The experience replay mechanism of deep reinforcement learning (DRL) stores the transfer samples obtained by the agent's interaction in the environment into the replay buffer, and randomly extracts small batches of samples from it to update the parameters of the value network or policy network. It can break the temporal correlation between samples, increase the independent and identically distributed nature of samples, and facilitate the convergence of the agent. In the standard reinforcement learning process, the agent generates experience (state-action-reward-new state sequence) through continuous interaction with the environment. These experiences usually have strong temporal correlation, and directly using these sequences for training may cause the model to fall into a local optimal solution or unstable training. In order to solve this problem, the experience replay mechanism came into being.

[0004] The basic principle of the experience replay mechanism is to store the experience generated during the interaction between the agent and the environment in a fixed-size experience replay pool, and then randomly sample small batches of experience from the experience pool for training during training. This random sampling breaks the temporal correlation between experiences, which can make the training process more stable, improve the diversity of samples, and promote the generalization ability of the model. In addition, the experience replay mechanism can also introduce priority sampling, that is, determine the sampling priority according to the importance of the sample (such as the temporal difference TD error). In this way, samples with larger errors will be sampled more frequently, which helps the model converge to the optimal strategy more quickly. However, when controlling complex high-dimensional agents, some states appear less frequently, resulting in an unbalanced distribution of the collected interaction samples. This imbalance increases the sparsity of interaction samples and affects the deep reinforcement learning DRL algorithm's learning of rare states. It leads to an imbalance between exploration and utilization, thereby affecting the convergence effect of deep reinforcement learning DRL. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention provides a vehicle path tracking control method based on improved deep reinforcement learning. The method of the present invention is specifically designed to utilize sample grouping and non-dominated sorting, perform intra-group collaboration and inter-group collaboration according to the convergence situation, realize the exploration and utilization of balanced playback samples, and improve the convergence of deep reinforcement learning for vehicle path tracking control.

[0006] The technical solution adopted by the present invention is:

[0007] The vehicle path tracking control method based on improved deep reinforcement learning of the present invention comprises:

[0008] Step 1: Establish a path tracking control optimization model for a four-wheel independent drive and independent steering 4WIS-4WID vehicle.

[0009] Step 2: Input the current actions of the four wheels of the four-wheel independent drive and independent steering 4WIS-4WID vehicle into the path tracking control optimization model, and use the improved deep reinforcement learning DRL controller based on the swarm intelligence collaborative experience playback mechanism to train the path tracking control optimization model, and the path tracking control optimization model outputs the optimized actions.

[0010] Step 3, passing the optimized action to the four-wheel independent drive and independent steering 4WIS-4WID vehicle and then controlling the action of the four wheels, thereby updating the next action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle, and repeating steps 2-3 to achieve real-time tracking control of the vehicle path.

[0011] In step 1, the path tracking control optimization model of the four-wheel independent drive and independent steering 4WIS-4WID vehicle is as follows:

[0012]

[0013] sta lb ≤a i ≤a ub

[0014] U lb ≤U i ≤U ub

[0015] s lb ≤s i ≤s ub

[0016]

[0017] Wherein, a^ represents the action set of the optimized four-wheel independent drive and independent steering 4WIS-4WID vehicle; X^ represents the reference path state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle; f H () represents the vehicle dynamics model of a four-wheel independent drive and independent steering 4WIS-4WID vehicle; s i 、s ub and lb Respectively represent the current i-th action a i The state quantity and upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when v xi 、v yi and γ zi Respectively represent the current i-th action a i The longitudinal velocity, lateral acceleration and yaw rate of the four-wheel independent drive and independent steering 4WIS-4WID vehicle at the time of i and Respectively represent the current i-th action a i The lateral displacement and heading angle of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when refi Indicates the current i-th action a i The tracking error between the vehicle state and the ideal state of a four-wheel independent drive and independent steering 4WIS-4WID vehicle is e xi , and e vxi Respectively represent the current i-th action a i The lateral displacement error, heading angle error and longitudinal velocity error of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when fc () represents the transfer function of the improved deep reinforcement learning DRL controller; U i , U ub and U lb Respectively represent the current i-th action a i The disturbance variables and their upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when ub and a lb They represent the current i-th action a of the four-wheel independent drive and independent steering 4WIS-4WID vehicle. i The upper and lower limits of a i =[T dfli T dfri T drli T drri δ dfli δ dfri δ drli δ drri ], T dfli , T dfri , T drli and T drri Respectively represent the current i-th action a i The driving torque of the left front wheel, right front wheel, left rear wheel and right rear wheel of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when dfli , δ dfri , δ drli and δ drri Respectively represent the current i-th action a i The steering angles of the left front wheel, right front wheel, left rear wheel and right rear wheel of the four-wheel independent drive independent steering 4WIS-4WID vehicle; i 、e ub and e lb Respectively represent the current i-th action a i Path tracking error and its upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle.

[0018] State quantity s i Including measurable state o and unmeasurable disturbance d, deep reinforcement learning DRL is used to perform end-to-end control of the vehicle. Therefore, it is assumed that the unmeasurable disturbance d is known, and the vehicle's motion control process is directly expressed in the form of Markov decision process MDP (Markov Decision Process): p(s′︱s=o, a), s′ represents the derivative of the state quantity s, which is conducive to the training of deep reinforcement learning DRL.

[0019] The step 2 is as follows:

[0020] In step 2.1, first build the agent and experience replay pool of the improved deep reinforcement learning (DRL) controller.

[0021] Step 2.2, use the intelligent agent to interact with the control environment of the four-wheel independent drive and independent steering 4WIS-4WID vehicle to obtain the transfer sample G of the current interaction, and then obtain the initial priority of the transfer sample G of the current interaction, store the transfer sample G of the current interaction and its initial priority in the experience replay pool, and then update the total priority of the experience replay pool.

[0022] Step 2.3, repeat step 2.2. When the number of transfer samples in the experience replay pool is equal to the preset threshold, divide the transfer samples in the experience replay pool into three groups: discoverer group, joiner group and risk group according to the initial priority of each transfer sample and the initial preset division ratio, and update the priorities of the three groups of transfer samples.

[0023] Step 2.4, according to the updated priorities of the three groups of transfer samples, in the training process of the improved deep reinforcement learning DRL controller, the division ratio of the three groups of transfer samples is dynamically adjusted and re-divided, and small batch samples are obtained by sampling according to the adjusted division ratio of the three groups of transfer samples in the re-divided three groups of transfer samples, and the small batch samples are input into the intelligent agent, and then input into the path tracking control optimization model for training until the loss function of the path tracking control optimization model converges, and the training is completed, and the path tracking control optimization model outputs the optimized action.

[0024] In the step 2.2, the initial priority of the transfer sample G of the current interaction is obtained according to the transfer sample G of the current interaction. First, the time difference TD error τ is obtained according to the transfer sample G of the current interaction. t and advantage function value A t , G=(s t , a t , r t ,s t+1 ), s t and t+1 They represent the state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th and t+1-th transfer samples G, respectively. t represents the action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th transfer sample G, r t Represents the reward function under the t-th transfer sample G; then according to the temporal difference TD error τ t and advantage function value A t Get the initial priority P of the transfer sample G of the current interaction t ,as follows:

[0025] P t=|τ t |+ν|A t |

[0026] Wherein, ν represents the preset priority coefficient.

[0027] In step 2.2, the reward function r under the t-th transfer sample G is t as follows:

[0028] r t =-(K r e ref T e ref +K f (a t -a t-1 ) T (a t -a t-1 ))‖e ref ‖<e max ∩s L <s t <s U

[0029] r t =-K b other

[0030] Among them, K r represents the positive gain parameter of the path tracking error of the improved deep reinforcement learning DRL controller, K f represents the positive gain parameter that improves the continuous control smoothness of the deep reinforcement learning DRL controller; e ref Represents the tracking error between the vehicle state and the ideal state of a four-wheel independent drive and independent steering 4WIS-4WID vehicle; a t-1 represents the action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-1th transfer sample G, a t -a t-1 Indicates the change range of the auxiliary control variable, its purpose is to ensure that the control process has a certain smoothness; e max represents the tracking error e ref The limit value of s t 、s U and L They represent the state quantity and upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th transfer sample G; K b Represents a penalty term.

[0031] The action of DRL is then passed to the vehicle body to achieve closed-loop control of the 4WIS4WID vehicle path tracking. The reward function is designed as the error of the path tracking control target and the smoothness of the control variable. Deep reinforcement learning DRL specifically uses the Twin Delayed Deep Deterministic policy gradient algorithm TD3 (Twin Delayed Deep Deterministic policy gradient algorithm).

[0032] In step 2.3, when the number of transfer samples in the experience replay pool is equal to the preset threshold, firstly, the transfer samples are arranged from large to small according to their initial priorities, and then the transfer samples in the experience replay pool are arranged in turn according to the preset first division ratio S α , preset the second division ratio S β and the preset third division ratio S γ Divided into three groups: discoverer group, joiner group and risk group, S α , S β , S γ ∈[0,1],S α +S β +S γ =1; for each transfer sample, the timing difference TD error τ of the transfer sample is t and advantage function value A t They are defined as the dominant distance and non-dominated distance respectively, and then the Pareto frontier F is obtained k , according to the Pareto frontier F k Initial update of the initial priority P of the transfer sample t Then get the intermediate priority P k Then according to the initial priority P t and the intermediate priority P k Update and finally update the priority of the transferred samples as follows:

[0033] P t ′=P t +minP k

[0034] Among them, P t ′ represents the priority of the tth transfer sample that is finally updated.

[0035] Discoverer Group D di The priority is in the front S α Percentile samples are mainly used to explore new valuable state-action pairs. j The priority is in the front S α +S β Percentile but not in the front S αPercentile samples are mainly used to develop near existing valuable state-action pairs. da The priority is after S γ The samples of quantiles are mainly those that are difficult to learn or bring negative effects.

[0036] In step 2.4, during the training process, the division ratios of the three groups of transfer samples are first normalized to avoid probability value overflow, and then the normalized division ratios of the three groups of transfer samples are dynamically adjusted and re-divided according to the current number of training steps tt and the total number of training steps TT, as follows:

[0037] S αtt =S α (1-tt / TT)×(exp(avg di ) / (exp(avg di )+exp(avg j )+exp(avg da )))

[0038] S βtt =(S β (1-tt / TT)+S α (tt / TT))×(exp(avg j ) / (exp(avg di )+exp(avg j )+exp(avg da )))

[0039] S γtt =(1-S α -S β )×(exp(avg da ) / (exp(avg di )+exp(avg j )+exp(avg da )))

[0040] Among them, S αt , S βtt and S γtt Respectively represent the proportions of the dynamically adjusted discoverer group, joiner group and risk group; S α , S β and S γ Represents the preset first, second and third division ratios of the discoverer group, the joiner group and the risk group respectively; avg da 、avg j and avg di The average values ​​of the updated priorities of the transfer samples of the discoverer group, the participant group, and the risk group are respectively represented.

[0041] Among the three groups of transfer samples after redivision, S was sampled in the discoverer group. αt Transfer samples, sample S in the joiner group βtt The transfer sample is S from the risk group. γtt The transferred samples are then combined into a small batch of samples, which are then input into the agent to realize the experience replay process.

[0042] During the training process, the sampling ratios of the three groups are dynamically adjusted to achieve inter-group collaboration and balance between exploration and utilization. αt Larger, encouraging exploration of new valuable state-action pairs. As training progresses, S αt Gradually decreases, S βtt Gradually increase, gradually transfer the valuable samples obtained from exploration to the joiner group for full use. γtt Keep it unchanged and ensure that a certain proportion of difficult samples participate in training to improve the generalization ability of the model.

[0043] In step 2.4, the loss function of the path tracking control optimization model is as follows:

[0044] L w =(∑ t∈B ω t L t ) / B

[0045] ω t =(1 / |SUM|P t ′) ξ

[0046] ξ=ξ0+(1-ξ0)tt / TT

[0047] Where B represents the total number of mini-batch samples; ω t represents the importance weight of the tth transfer sample in the mini-batch sample; L t represents the mean square error loss function of the tth transfer sample in the mini-batch sample; SUM represents the sum of the priorities of all transfer samples that are finally updated; P t ′ represents the priority of the final update of the tth transfer sample; ξ and ξ0 represent the importance sampling weight parameter and its initial value respectively. ξ controls the intensity of importance sampling correction and increases linearly to 1 during training.

[0048] In order to correct the bias introduced by priority sampling, importance sampling correction is required for the loss function during training. After each training step, the priority of the sample is updated according to its TD error, and then the updated priority is backfilled into the experience replay pool, and the total priority is updated.

[0049] The electronic device of the present invention comprises: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method as described above.

[0050] The computer-readable storage medium of the present invention stores program data thereon, and is characterized in that the program data implements the method described above when executed by a processor.

[0051] The method of the present invention stores the sample data obtained during the control of the vehicle into the experience pool. All the data in the experience pool are regarded as an intelligent group. The samples in the experience pool are divided into three different functional groups by time series difference TD error and advantage function value, and then non-dominated sorting and training progress are introduced to perform intra-group collaboration and inter-group collaboration to optimize data playback and storage. The function of the discoverer group is to provide a better direction for the convergence of the deep reinforcement learning DRL training process, and give priority to learning to explore novel state-action pairs; the function of the joiner group is to replay excellent samples with a large proportion around the discoverer, strengthen and optimize known high-value strategies; and the dangerous group is to screen out samples with poor quality to avoid learning high-risk samples that may cause overfitting. However, a small proportion of samples will still be extracted to ensure the diversity of the playback samples, providing other possible global optimal convergence directions for the training process.

[0052] The beneficial effects of the present invention are:

[0053] The method of the present invention can adaptively adjust the priority of samples, balance exploration and utilization, improve sample utilization efficiency, and improve the convergence speed and generalization ability of deep reinforcement learning algorithms. Through the group collaborative sampling strategy, the exploration value and utilization value of samples are comprehensively considered to accelerate the strategy learning process. In addition, the method introduces importance sampling correction, which effectively reduces the deviation introduced by priority sampling and improves the stability of 4WIS-4WID vehicle path tracking control. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall flow chart of the method of the present invention;

[0055] Figure 2 It is a schematic diagram of the group intelligent collaborative experience playback mechanism of the method of the present invention;

[0056] Figure 3 is a schematic diagram of a vehicle body dynamics model of the present invention;

[0057] Figure 4 is a schematic diagram of the tire dynamics model of the present invention;

[0058] Figure 5 This is a comparison chart of different DRL algorithms. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 As shown, the vehicle path tracking control method based on improved deep reinforcement learning of the present invention is specifically as follows:

[0061] First, the path tracking control optimization model of the four-wheel independent drive and independent steering 4WIS-4WID vehicle is established as follows:

[0062] a ^ = arg min∑ N i＝1 ‖X ^ -f H (s i , f c (s i , U i )‖ 2

[0063] sta lb ≤a i ≤a ub

[0064] U lb ≤U i ≤U ub

[0065] s lb ≤s i ≤s ub

[0066]

[0067] Among them, a ^ represents the action set of the optimized four-wheel independent drive and independent steering 4WIS-4WID vehicle; X^ represents the reference path state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle; f H () represents the vehicle dynamics model of a four-wheel independent drive and independent steering 4WIS-4WID vehicle; s i 、s ub and lb Respectively represent the current i-th action a i The state quantity and upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when v xi 、v yi and γ zi Respectively represent the current i-th action a iThe longitudinal velocity, lateral acceleration and yaw rate of the four-wheel independent drive and independent steering 4WIS-4WID vehicle at the time of i and Respectively represent the current i-th action a i The lateral displacement and heading angle of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when refi Indicates the current i-th action a i The tracking error between the vehicle state and the ideal state of a four-wheel independent drive and independent steering 4WIS-4WID vehicle is e xi , and e vxi Respectively represent the current i-th action a i The lateral displacement error, heading angle error and longitudinal velocity error of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when f c () represents the transfer function of the improved deep reinforcement learning DRL controller; U i , U ub and U lb Respectively represent the current i-th action a i The disturbance variables and their upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when ub and a lb They represent the current i-th action a of the four-wheel independent drive and independent steering 4WIS-4WID vehicle. i The upper and lower limits of a i =[T dfli T dfri T drli T drri δ dfli δ dfri δ drli δ drri ], T dfli , T dfri , T drli and T drri Respectively represent the current i-th action a i The driving torque of the left front wheel, right front wheel, left rear wheel and right rear wheel of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when dfli , δ dfri , δ drli and δ drri Respectively represent the current i-th action a i The steering angles of the left front wheel, right front wheel, left rear wheel and right rear wheel of the four-wheel independent drive independent steering 4WIS-4WID vehicle; i 、e ub and e lb Respectively represent the current i-th action ai Path tracking error and its upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle.

[0068] State quantity s i Including measurable state o and unmeasurable disturbance d, deep reinforcement learning DRL is used to perform end-to-end control of the vehicle. Therefore, it is assumed that the unmeasurable disturbance d is known, and the vehicle's motion control process is directly expressed in the form of Markov decision process MDP (Markov Decision Process): p(s′︱s=o, a), s′ represents the derivative of the state quantity s, which is conducive to the training of deep reinforcement learning DRL.

[0069] The vehicle dynamics model of the four-wheel independent drive and independent steering 4WIS-4WID vehicle includes the body dynamics model and the tire dynamics model. Figure 3 and Figure 4 OXYZ and O v X v Y v Z v Denote the earth inertial reference frame and the vehicle body inertial reference frame, respectively. Assuming that the path tracking occurs on a flat road, the yaw motion of the vehicle body is mainly controlled by the longitudinal force and lateral force on each wheel. For the convenience of notation, the four tires are indexed as j = fl, fr, rl and rr, which are the left front wheel, right front wheel, left rear wheel and right rear wheel, respectively. Figure 3 As shown in the figure, in the tire coordinate system, the longitudinal force, lateral force, vertical force and steering angle of each tire are represented by F lj 、F Lj 、F Nj and δ j Therefore, the vehicle body dynamics model is as follows:

[0070] M(v x ′-v y γ)=∑ j fl，fr，rl，rr (F lj cosδ j -F Lj sinδ j )

[0071] M(v y ′-v x γ)=∑ j fl，fr，rl，rr (F lj sinδ j +F Lj cosδ j )

[0072] I γ γ′=∑ jfl，fr，rl，rr [L bj (F lj cosδ j -F Lj sinδ j )+l bj (F lj sinδ j +F Lj cosδ j )]

[0073] Among them, M() and I γ They represent the vehicle mass and moment of inertia of the four-wheel independent drive and independent steering 4WIS-4WID vehicle respectively; v x and v x ′ represents the longitudinal speed and longitudinal acceleration of the four-wheel independent drive and independent steering 4WIS-4WID vehicle, v y and v y ′ represents the lateral velocity and lateral acceleration of the four-wheel independent drive and independent steering 4WIS-4WID vehicle, γ and γ′ represent the yaw angular velocity and yaw angular acceleration of the four-wheel independent drive and independent steering 4WIS-4WID vehicle, respectively; L bj and l bj Respectively represent the distance of each tire from the center of gravity of the vehicle in the vertical forward direction and the forward direction in the body coordinate system.

[0074] The vehicle acceleration in the body coordinate system is calculated by the forces applied by the tires. In the design of the path tracking controller, the longitudinal force and lateral force at each tire are determined by controlling the steering angle and the wheel drive torque, as shown in the tire dynamics model in 4. In the tire coordinate system, the Magic Formula tire dynamics model is used to represent the longitudinal force F of each tire. lj and the lateral force F Lj The nonlinear characteristics are as follows:

[0075] m=Dsin(Carctan{Hx-E[Hx-arctan(Hx)]})

[0076] Where m represents the longitudinal force F of each tire lj Or lateral force F Lj ; x represents the longitudinal slip rate λ j Or slip angle; D, C, H and E represent the peak factor, shape factor, stiffness coefficient and curvature coefficient respectively.

[0077] Longitudinal slip rate λ j as follows:

[0078] λ j =(R w ω j -uwj ) / (R w ω j ) R w ω j ≥u wj

[0079] λ j =(R w ω j -u wj ) / u wj R w ω j <u wj

[0080] Among them, R w and ω j Respectively represent the radius and angular velocity of the tire; u wj Indicates the actual speed at the center of the tire.

[0081] The actual speed u at the center of the tire wj as follows:

[0082] u wj =(v x ±L bj γ)cosδ j +(v y ±l bj γ)sinδ j

[0083] Then, the longitudinal and lateral dynamic models of the tire can be obtained:

[0084] T j =(F lj -F Nj f w )R w +ω j I w

[0085] δ j =arctan((v y ±ω j l bj ) / (v x ±ω j L bj ))-α j

[0086] Among them, T j 、f w ,I w and F Nj Respectively represent the driving torque, rolling friction coefficient, moment of inertia and vertical force of each tire; α j Indicates the slip angle of each tire.

[0087] The result is a complex vehicle dynamics model with seven degrees of freedom, with up to eight controllable inputs, namely four drive torques and four steering angles.

[0088] Then the current actions of the four wheels of the four-wheel independent drive and independent steering 4WIS-4WID vehicle are input into the path tracking control optimization model, and the path tracking control optimization model is trained using an improved deep reinforcement learning DRL controller based on swarm intelligence collaborative experience playback mechanism, as shown in Figure 2 As shown in Figure 1, we first construct an agent and experience replay pool of the improved deep reinforcement learning DRL controller, and then use the agent to interact with the control environment of the four-wheel independent drive independent steering 4WIS-4WID vehicle to obtain the transfer sample G of the interaction, and then obtain the initial priority of the transfer sample G of the interaction. First, we obtain the temporal difference TD error τ according to the transfer sample G of the interaction. t and advantage function value A t , G=(s t , a t , r t ,s t+1 ), s t and t+1 They represent the state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th and t+1-th transfer samples G, respectively. t represents the action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th transfer sample G, r t Represents the reward function under the t-th transfer sample G; then according to the temporal difference TD error τ t and advantage function value A t Get the initial priority P of the transfer sample G of the current interaction t ,as follows:

[0089] P t =|τ t |+ν|A t |

[0090] Wherein, ν represents the preset priority coefficient.

[0091] Timing difference TD error τ t and advantage function value A t The details are as follows:

[0092] τ t =r t +κQ θ′ (s t+1 , μ θ′ (s t+1 ))-Q θ (st , a t )

[0093] A t =Q θ (s t , a t )-V θ (s t )

[0094] Where κ represents the discount factor; Q θ and Q θ′ They represent the critic Q network and the target critic Q network respectively; μ θ′ represents the target actor network; V θ Represents a state-value function.

[0095] The reward function r under the tth transfer sample G t as follows:

[0096] r t =-(K r e ref T e ref +K f (a t -a t-1 ) T (a t -a t-1 ))‖e ref ‖<e max ∩s L <s t <s U

[0097] r t =-K b other

[0098] Among them, K r represents the positive gain parameter of the path tracking error of the improved deep reinforcement learning DRL controller, K f represents the positive gain parameter that improves the continuous control smoothness of the deep reinforcement learning DRL controller; e ref Represents the tracking error between the vehicle state and the ideal state of a four-wheel independent drive and independent steering 4WIS-4WID vehicle; a t-1 represents the action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-1th transfer sample G, a t -a t-1 Indicates the change range of the auxiliary control variable, its purpose is to ensure that the control process has a certain smoothness; e max represents the tracking error e ref The limit value of s t 、sU and L They represent the state quantity and upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th transfer sample G; K b Represents a penalty term.

[0099] The action of DRL is then passed to the vehicle body to achieve closed-loop control of the 4WIS4WID vehicle path tracking. The reward function is designed as the error of the path tracking control target and the smoothness of the control variable. Deep reinforcement learning DRL specifically uses the Twin Delayed Deep Deterministic policy gradient algorithm TD3 (Twin Delayed Deep Deterministic policy gradient algorithm).

[0100] The transfer sample G of the current interaction and its initial priority are stored in the experience replay pool to update the total priority of the experience replay pool.

[0101] Repeat the above steps. When the number of transfer samples in the experience replay pool is equal to the preset threshold, firstly, arrange the transfer samples from large to small according to their initial priorities, and then arrange the transfer samples in the experience replay pool in turn according to the preset first division ratio S. α , preset the second division ratio S β and the preset third division ratio S γ Divided into three groups: discoverer group, joiner group and risk group, S α , S β , S γ ∈[0,1],S α +S β +S γ =1; for each transfer sample, the timing difference TD error τ of the transfer sample is t and advantage function value A t They are defined as the dominant distance and non-dominated distance respectively, and then the Pareto frontier F is obtained k , according to the Pareto frontier F k Initial update of the initial priority P of the transfer sample t Then get the intermediate priority P k Then according to the initial priority P t and the intermediate priority P k Update and finally update the priority of the transferred samples as follows:

[0102] P t ′=P t +minP k

[0103] Among them, P t ′ represents the priority of the tth transfer sample that is finally updated.

[0104] Discoverer Group D di The priority is in the top 30% of the samples, mainly used to explore new valuable state-action pairs. j Priority is in the top 30%-70% percentile but not in the top S α Percentile samples are mainly used to develop near existing valuable state-action pairs. da These are samples with a priority in the bottom 30% percentile, which are mainly samples that are difficult to learn or have negative impacts.

[0105] In the early stage of training, we encourage the exploration of new valuable state-action pairs. As the training progresses, we gradually transfer the explored valuable samples to the joiner group for utilization, keeping the proportion of the dangerous group unchanged to improve the generalization ability of the model.

[0106] The timing difference TD error τ of the transferred samples t and advantage function value A t The dominant distance and non-dominated distance, respectively, can more comprehensively and accurately evaluate the quality of samples within the group. The Pareto frontier F k The details are as follows:

[0107]

[0108] Among them, D k Indicates the discoverer group D di , Joiner Group D j or risk group D da ;s j represents the state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the j-th transfer sample G; τ t and τ j Denote the timing difference TD error under the t-th and j-th transfer sample G, A t and A j Represent the advantage function values ​​under the t-th and j-th transfer samples G respectively.

[0109] According to the updated priorities of the three groups of transfer samples, in the training process of the improved deep reinforcement learning DRL controller, the division ratio of the three groups of transfer samples is dynamically adjusted and re-divided, and small batch samples are sampled according to the adjusted division ratio of the three groups of transfer samples in the re-divided three groups of transfer samples. The small batch samples are input into the intelligent agent, and then input into the path tracking control optimization model for training until the loss function of the path tracking control optimization model converges, then the training is completed, and the path tracking control optimization model outputs the optimized action.

[0110] During the training process, the division ratios of the three groups of transfer samples are first normalized to avoid probability value overflow. Then, according to the current number of training steps tt and the total number of training steps TT, the normalized division ratios of the three groups of transfer samples are dynamically adjusted and re-divided as follows:

[0111] S αtt =S α (1-tt / TT)×(exp(avg di ) / (exp(avg di )+exp(avg j )+exp(avg da )))

[0112] S βtt =(S β (1-tt / TT)+S α (tt / TT))×(exp(avg j ) / (exp(avg di )+exp(avg j )+exp(avg da )))

[0113] S γtt =(1-S α -S β )×(exp(avg da ) / (exp(avg di )+exp(avg j )+exp(avg da )))

[0114] Among them, S αt , S βtt and S γtt Respectively represent the proportions of the dynamically adjusted discoverer group, joiner group and risk group; S α , S β and S γ Represents the preset first, second and third division ratios of the discoverer group, the joiner group and the risk group respectively; avg da 、avg j and avg di The average values ​​of the updated priorities of the transfer samples of the discoverer group, the participant group, and the risk group are respectively represented.

[0115] Among the three groups of transfer samples after redivision, S was sampled in the discoverer group. αt Transfer samples, sample S in the joiner group βtt The transfer sample is S from the risk group. γttThe transferred samples are then combined into a small batch of samples, which are then input into the agent to realize the experience replay process.

[0116] During the training process, the sampling ratios of the three groups are dynamically adjusted to achieve inter-group collaboration and balance between exploration and utilization. αt Larger, encouraging exploration of new valuable state-action pairs. As training progresses, S αt Gradually decreases, S βtt Gradually increase, gradually transfer the valuable samples obtained from exploration to the joiner group for full use. γtt Keep it unchanged and ensure that a certain proportion of difficult samples participate in training to improve the generalization ability of the model.

[0117] The loss function of the path tracking control optimization model is as follows:

[0118] L w =(∑ t∈B ω t L t ) / B

[0119] ω t =(1 / |SUM|P t ′) ξ

[0120] ξ=ξ0+(1-ξ0)tt / TT

[0121] Where B represents the total number of mini-batch samples; ω t represents the importance weight of the tth transfer sample in the mini-batch sample; L t represents the mean square error loss function of the tth transfer sample in the mini-batch sample; SUM represents the sum of the priorities of all transfer samples that are finally updated; P t ′ represents the priority of the final update of the tth transfer sample; ξ and ξ0 represent the importance sampling weight parameter and its initial value respectively. ξ controls the intensity of importance sampling correction and increases linearly to 1 during training.

[0122] In order to correct the bias introduced by priority sampling, importance sampling correction is required for the loss function during training. After each training step, the priority of the sample is updated according to its TD error, and then the updated priority is backfilled into the experience replay pool, and the total priority is updated.

[0123] The present invention selects the twin-delayed policy gradient algorithm TD3 to implement the deep reinforcement learning DRL part. TD3 (TwinDelayed Deep Deterministic Policy Gradient) is a reinforcement learning algorithm based on the deep deterministic policy gradient DDPG (Deep Deterministic Policy Gradient), which is specifically used to solve control problems in continuous action space. The twin-delayed policy gradient algorithm TD3 reduces the over-estimation problem in traditional algorithms and improves the stability and performance of learning by introducing mechanisms such as dual Q networks, delayed policy updates, and target action noise. The core idea is to use the experience replay pool to store interaction samples with the environment, break the time correlation through random sampling, and train the strategy and Q network more efficiently.

[0124] First, initialize the policy network π c , two Q networks Q θ1 and Q θ2 and their corresponding target policy networks π c′ , two target Q networks Q θ′1 and Q θ′2 The initial parameters of the target network are the same as those of the current network. The experience replay pool has already started collecting samples in the first step, and these samples will be used in the subsequent training phase. Then, according to the current policy network and state s t Select action a t . Perform action a t After that, record the instant reward r t and the new state s t+1 , and the new transfer sample (s t , a t , r t ,s t+1 ) is stored in the experience replay pool.

[0125] Extract replay samples from the experience replay pool and then use the target policy network to generate the next state s t+1 The target action a′ is taken and noise is introduced for smoothing to prevent the policy network from overfitting the action:

[0126] a′=π c′ (s t+1 )+ε

[0127] Here, ε represents motion noise.

[0128] Two target Q networks are used to calculate the target Q value and take the minimum value to reduce the over-estimation problem of the Q value.

[0129] n=r t +κmin p＝1,2 Qθp′ (s t+1 , a′)

[0130] Where n is the target Q value, r t is the immediate reward, κ is the discount factor, and a′ is the target action. Stochastic gradient descent is used to minimize the loss function of the two Q networks. The loss function is the mean square error L(θp) between the Q value estimate and the target Q value:

[0131] L(p)=E[(Q θ′p (s t ,a t )-n) 2 ]

[0132] Among them, E[] means taking expectation.

[0133] By optimizing this loss function, TD3 gradually adjusts the parameters θ1 and θ2 of the Q network to improve the accuracy of Q value estimation.

[0134] The double-delayed policy gradient algorithm TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated once every two updates of the Q network. c . Policy Network π c The goal is to maximize the Q value, and the update formula is:

[0135] c←c-η c ▽ c J(c)

[0136] J(c)=E[Q θ1 (s,π c (s))]

[0137] Where J(c) represents the policy network π c The objective function of c represents the policy network π c The network coefficient, η c Represents the policy network π c The learning rate, ▽ c Represents the policy network π c gradient.

[0138] By delaying updates, the double-delayed policy gradient algorithm TD3 can optimize the policy network based on a more stable Q-value estimation, avoiding the policy network from relying on inaccurate Q-values ​​too early.

[0139] Use the soft update method to update the parameters of the target Q network and the target policy network, so that the parameters of the target network slowly approach the parameters of the current network, as follows:

[0140] θ′p←ρθp+(1-ρ)θ′p

[0141] c′p←ρcp+(1-ρ)c′p, p=1,2

[0142] Among them, ρ represents the soft update step size, which is usually small.

[0143] After training is complete, the policy network π c It has been optimized to the optimal strategy. At this point, for a given state s, the optimal action a can be directly calculated through the policy network without adding noise:

[0144] a=π c (s)

[0145] This action a is used to guide the agent's actual decision-making in the environment.

[0146] The optimized action is transmitted to the four-wheel independent drive and independent steering 4WIS-4WID vehicle, and then the action of the four wheels is controlled, and then the next action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle is updated, and the steps are repeated to realize real-time tracking control of the vehicle path.

[0147] like Figure 5 As shown, the improved deep reinforcement learning GER-TD3 has a larger replay reward value than the TD3 algorithm AER-TD3 based on average experience replay and the TD3 algorithm PER-TD3 based on priority experience replay, that is, it converges faster.

[0148] The method of the present invention regards the experience pool as an intelligent group, and divides the group into three groups: discoverers, joiners and dangerous ones through the sample priority established by the time series difference TD error and the advantage function. The training progress and non-dominated sorting are used to coordinate the dynamics between and within the group, adaptively balance exploration and utilization, improve the convergence speed of deep reinforcement learning DRL, and improve the control accuracy of the path tracking of the four-wheel independent drive independent steering 4WIS-4WID vehicle. The method of the present invention can achieve the balance of playback sample exploration-utilization, and adaptively adjust the weight according to the training process.

[0149] The contents described in the implementation cases of this specification are merely descriptions of the process of the group intelligent collaborative experience playback mechanism. The protection scope of the present invention should not be regarded as limited to the specific forms and parameters stated in the implementation cases. The protection scope of the present invention also involves group collaboration modes of different experience playback mechanisms.

Claims

1. A vehicle path tracking control method based on improved deep reinforcement learning, characterized in that: include: Step 1, establishing a path tracking control optimization model for a four-wheel independent drive and independent steering 4WIS-4WID vehicle; Step 2: Input the current actions of the four wheels of the four-wheel independent drive and independent steering 4WIS-4WID vehicle into the path tracking control optimization model, and use the improved deep reinforcement learning DRL controller based on the swarm intelligence collaborative experience playback mechanism to train the path tracking control optimization model, and the path tracking control optimization model outputs the optimized actions; Step 3, passing the optimized action to the four-wheel independent drive and independent steering 4WIS-4WID vehicle and then controlling the action of the four wheels, thereby updating the next action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle, and repeating steps 2-3 to achieve real-time tracking control of the vehicle path.

2. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 1, characterized in that: In step 1, the path tracking control optimization model of the four-wheel independent drive and independent steering 4WIS-4WID vehicle is as follows: a^=arg min∑ N i＝1 ‖X^-f H (s i ,f c (s i ,U i )‖ 2 s.t.a lb ≤a i ≤a ub IN lb ≤U i ≤U ub s lb ≤s i ≤s ub Wherein, a^ represents the action set of the optimized four-wheel independent drive and independent steering 4WIS-4WID vehicle; X^ represents the reference path state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle; f H () represents the vehicle dynamics model of a four-wheel independent drive and independent steering 4WIS-4WID vehicle; s i 、s ub and lb Respectively represent the current i-th action a i The state quantity and upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when v xi 、v yi and γ zi Respectively represent the current i-th action a i The longitudinal velocity, lateral acceleration and yaw rate of the four-wheel independent drive and independent steering 4WIS-4WID vehicle at the time of i and Respectively represent the current i-th action a i The lateral displacement and heading angle of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when refi Indicates the current i-th action a i The tracking error between the vehicle state and the ideal state of a four-wheel independent drive and independent steering 4WIS-4WID vehicle is e xi , and e vxi Respectively represent the current i-th action a i The lateral displacement error, heading angle error and longitudinal velocity error of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when f c () represents the transfer function of the improved deep reinforcement learning DRL controller; U i , U ub and U lb Respectively represent the current i-th action a i The disturbance variables and their upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when ub and a lb They represent the current i-th action a of the four-wheel independent drive and independent steering 4WIS-4WID vehicle. i The upper and lower limits of a i =[T dfli T dfri T drli T drri δ dfli δ dfri δ drli δ drri ], T dfli , T dfri , T drli and T drri Respectively represent the current i-th action a i The driving torque of the left front wheel, right front wheel, left rear wheel and right rear wheel of the four-wheel independent drive and independent steering 4WIS-4WID vehicle when dfli , δ dfri , δ drli and δ drri Respectively represent the current i-th action a i The steering angles of the left front wheel, right front wheel, left rear wheel and right rear wheel of the four-wheel independent drive independent steering 4WIS-4WID vehicle; i 、e ub and e lb Respectively represent the current i-th action a i Path tracking error and its upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle.

3. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 1, characterized in that: The step 2 is as follows: Step 2.1, first build the agent and experience replay pool of the improved deep reinforcement learning DRL controller; Step 2.2, using the intelligent agent to interact with the control environment of the four-wheel independent drive and independent steering 4WIS-4WID vehicle to obtain the transfer sample G of the current interaction, and then obtain the initial priority of the transfer sample G of the current interaction, and store the transfer sample G of the current interaction and its initial priority in the experience replay pool to update the total priority of the experience replay pool; Step 2.3, repeat step 2.2, when the number of transfer samples in the experience replay pool is equal to the preset threshold, divide the transfer samples in the experience replay pool into three groups: the discoverer group, the joiner group and the risk group according to the initial priority of each transfer sample and the initial preset division ratio, and update the priorities of the three groups of transfer samples; Step 2.4, according to the updated priorities of the three groups of transfer samples, in the training process of the improved deep reinforcement learning DRL controller, the division ratio of the three groups of transfer samples is dynamically adjusted and re-divided, and small batch samples are obtained by sampling according to the adjusted division ratio of the three groups of transfer samples in the re-divided three groups of transfer samples, and the small batch samples are input into the intelligent agent, and then input into the path tracking control optimization model for training until the loss function of the path tracking control optimization model converges, and the training is completed, and the path tracking control optimization model outputs the optimized action.

4. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 3 is characterized in that: In the step 2.2, the initial priority of the transfer sample G of the current interaction is obtained according to the transfer sample G of the current interaction. First, the time difference TD error τ is obtained according to the transfer sample G of the current interaction. t and advantage function value A t , G=(s t , a t , r t ,s t+1 ), s t and t+1 They represent the state of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th and t+1-th transfer samples G, respectively. t represents the action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th transfer sample G, r t Represents the reward function under the t-th transfer sample G; then according to the temporal difference TD error τ t and advantage function value A t Get the initial priority P of the transfer sample G of the current interaction t ,as follows: P t =|τ t |+ν|A t | Wherein, ν represents the preset priority coefficient.

5. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 4 is characterized in that: In step 2.2, the reward function r under the t-th transfer sample G is t as follows: r t =-(K r and ref T and ref +K f (the t -the t-1 ) T (the t -the t-1 ))‖and ref ‖<and max ∩s L <s t <s U r t =-K b other Among them, K r represents the positive gain parameter of the path tracking error of the improved deep reinforcement learning DRL controller, K f represents the positive gain parameter that improves the continuous control smoothness of the deep reinforcement learning DRL controller; e ref Represents the tracking error between the vehicle state and the ideal state of a four-wheel independent drive and independent steering 4WIS-4WID vehicle; a t-1 represents the action of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-1th transfer sample G; e max represents the tracking error e ref The limit value of s t 、s U and L They represent the state quantity and upper and lower limits of the four-wheel independent drive and independent steering 4WIS-4WID vehicle under the t-th transfer sample G; K b Represents a penalty term.

6. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 3 is characterized in that: In step 2.3, when the number of transfer samples in the experience replay pool is equal to the preset threshold, firstly, the transfer samples are arranged from large to small according to their initial priorities, and then the transfer samples in the experience replay pool are arranged in turn according to the preset first division ratio S α , preset the second division ratio S β and the preset third division ratio S γ Divided into three groups: discoverer group, joiner group and risk group, S α , S β , S γ ∈[0,1],S α +S β +S γ =1; for each transfer sample, the timing difference TD error τ of the transfer sample is t and advantage function value A t They are defined as the dominant distance and non-dominated distance respectively, and then the Pareto frontier F is obtained k , according to the Pareto frontier F k Initial update of the initial priority P of the transfer sample t Then get the intermediate priority P k Then according to the initial priority P t and the intermediate priority P k Update and finally update the priority of the transferred samples as follows: P t ′=P t +minP k Among them, P t ′ represents the priority of the tth transfer sample that is finally updated.

7. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 3 is characterized in that: In step 2.4, during the training process, the division ratios of the three groups of transfer samples are first normalized, and then the normalized division ratios of the three groups of transfer samples are dynamically adjusted and re-divided according to the current number of training steps tt and the total number of training steps TT, as follows: S αtt =S α (1-tt / TT)×(exp(avg di ) / (exp(avg di )+exp(avg j )+exp(avg da ))) S βtt =(S β (1-tt / TT)+S α (tt / TT))×(exp(avg j ) / (exp(avg di )+exp(avg j )+exp(avg da ))) S γtt =(1-S α -S β )×(exp(avg da ) / (exp(avg di )+exp(avg j )+exp(avg da ))) Among them, S αt , S βtt and S γtt Respectively represent the proportions of the dynamically adjusted discoverer group, joiner group and risk group; S α , S β and S γ Represents the preset first, second and third division ratios of the discoverer group, the joiner group and the risk group respectively; avg da 、avg j and avg di represent the average values ​​of the updated priorities of the transfer samples of the discoverer group, the joiner group, and the risk group, respectively; Among the three groups of transfer samples after redivision, S was sampled in the discoverer group. αt Transfer samples, sample S in the joiner group βtt The transfer sample is S from the risk group. γtt The transferred samples are then combined into a small batch of samples and then input into the agent.

8. The vehicle path tracking control method based on improved deep reinforcement learning according to claim 3 is characterized in that: In step 2.4, the loss function of the path tracking control optimization model is as follows: L w =(∑ t∈B oh t L t ) / B oh t =(1 / |SUM|P t ′) ξ ξ=ξ0+(1-ξ0)tt / TT Where B represents the total number of mini-batch samples; ω t represents the importance weight of the tth transfer sample in the mini-batch sample; L t represents the mean square error loss function of the tth transfer sample in the mini-batch sample; SUM represents the sum of the priorities of all transfer samples that are finally updated; P t ′ represents the final updated priority of the tth transfer sample; ξ and ξ0 represent the importance sampling weight parameter and its initial value respectively.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images