Adaptive dynamic window method based on deep reinforcement learning

Abstract

The present invention discloses an adaptive dynamic window method based on deep reinforcement learning. Firstly, a deep reinforcement learning model for a DWA algorithm is constructed, which specifically includes elements such as an intelligent agent, an environment, an action and state space, a reward function, and a network architecture. Then, according to the principle of the DWA algorithm, the movement of an intelligent vehicle in a randomly generated obstacle map is simulated to obtain a training set for the gradient descent of a neural network. Subsequently, gradient descent is performed according to the idea of ​​a proximal policy optimization (PPO) algorithm to maximize the reinforcement learning reward. The training finally results in a converged neural network, which is then fused with the DWA algorithm evaluation function to complete the construction of an adaptive DWA algorithm.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving path planning and autonomous navigation technology, and in particular to an adaptive dynamic window method based on deep reinforcement learning. Specifically, the present invention relates to a solution to the defect that the three weights (speed weight, obstacle weight, and heading angle weight) of the evaluation function of the current dynamic window method (DWA) do not dynamically adjust with the environment in which the intelligent vehicle is located and its own motion state when facing different obstacle environments, which results in the intelligent body often being unable to find the end point or calculate the optimal path when facing different obstacle environments. Background Art

[0002] Path planning is an important part of autonomous driving smart cars. Path planning technology can be summarized as selecting the optimal obstacle avoidance path from the starting point to the end point in the mission area. In essence, it is the optimal solution under several constraints. Path planning is a key part of the unmanned navigation technology of smart cars. Path planning algorithms can be divided into global planning based on understanding the complete regional information and local planning based on understanding the local regional information. Dynamic Window Approach (DWA) is a local path planning method that takes into account the motion performance of smart cars and is widely used in path navigation of smart cars.

[0003] The dynamic window method speed-angular velocity combination command drives the smart car to move forward. The workflow is as follows: consider the constraints of various conditions on the speed-angular velocity space at time t, and obtain the speed-angular velocity range that the smart car can reach at time t; discretize it; combine the discretized speed-angular velocity; the smart car traverses all combinations and simulates moving forward m Δ according to the given motion model. t The evaluation function gives the scores of all simulated trajectories in the simulated trajectory set τ, and the combination corresponding to the highest-scoring trajectory is selected; the combination is used to drive the smart car forward for a time Δ t Arrive at t+1 and repeat this cycle until the end. m is the number of sampling steps of the DWA algorithm simulation trajectory, Δ t is the sampling time interval. The decision-making function is its evaluation function, which includes three parts: the heading angle function, the obstacle function, and the speed function. The evaluation function is the weighted sum of these three sub-functions. The weights corresponding to the three functions in the original DWA algorithm are fixed values. However, when the smart car is exploring the end point, the obstacle environment around it is complex and changeable. Different obstacle distributions require different weights, which leads to the original DWA algorithm fixed weight value method that easily makes the smart car fall into a local optimum or the target unreachable. Summary of the invention

[0004] The purpose of the present invention is to provide a method for adaptively and dynamically adjusting the evaluation function of the DWA algorithm based on a deep reinforcement learning method, in order to solve the problem that the weight parameters in the evaluation function of the existing DWA algorithm path planning algorithm are fixed values ​​or cannot be adjusted in real time according to the motion state of the smart car and the surrounding obstacle environment to obtain the optimal evaluation function suitable for the current moment.

[0005] The present invention first constructs a deep reinforcement learning model for the DWA algorithm, which specifically includes elements such as the agent, environment, action and state space, reward function, and network architecture; then, according to the principle of the DWA algorithm, simulates the movement of the intelligent car in a randomly generated obstacle map to obtain a training set for the gradient descent of the neural network; then, according to the idea of ​​the proximal policy optimization (PPO) algorithm, gradient descent is performed to maximize the reinforcement learning reward; the final result of the training is a convergent neural network, which is integrated with the DWA algorithm evaluation function to complete the construction of the adaptive DWA algorithm. Finally, the effectiveness of the algorithm is verified by simulation on the Python3 platform.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] An adaptive dynamic window method based on deep reinforcement learning, characterized by comprising the following steps:

[0008] Step S1: Establish a DWA algorithm model, based on the Ackerman smart car, including: speed range [v min ,v max ]、Angular velocity range [w min ,w max ], Acceleration range Angular acceleration range Parameters and elements of the DWA algorithm;

[0009] Step S2: Build a deep reinforcement learning model for the DWA algorithm, including the agent, environment, action and state space, reward function, and deep reinforcement learning architecture;

[0010] Step S3: Randomly initialize the obstacle map environment and initialize the smart car speed v 0 , angular velocity w 0 , orientation angle θ 0 is 0, and the smart car is placed at the starting point p 0 , the end point is set to g;

[0011] Step S4: Training set collection phase: simulate the movement of the intelligent agent in the randomly generated obstacle map and collect the trajectory set ψ for the gradient descent of the neural network;

[0012] Step S5: Network parameter update phase: Use the trajectory set ψ collected in step S4 to update the parameters of each network architecture; Step S6: Determine whether the average round score of the agent under the current network architecture is greater than the threshold. If so, terminate the loop, otherwise re-enter step S3 for a new generation of training.

[0013] Furthermore, in step S1, the core elements of the DWA algorithm include: speed space constraint, motion model, and evaluation function:

[0014] Velocity space constraint V v

[0015] (1) Limit speed constraint V m : The feasible speed of smart cars shall not exceed the limit range:

[0016] V m ={(v,w)|v∈[v min ,v max ]∧w∈[w min ,w max ]}

[0017] In the formula [v min ,v max ]Linear speed range, [w min ,w max ] is the angular velocity range;

[0018] (2) Acceleration constraint V d : The speed and angular velocity increments of the smart car at adjacent moments are subject to the acceleration and angular acceleration constraints:

[0019]

[0020] v c 、w c is the current linear velocity and angular velocity, is the linear acceleration range, is the angular acceleration range, Δ t is the movement time interval, and also the simulation trajectory sampling time interval;

[0021] (3) Braking distance constraint V a : The smart car should be able to decelerate to 0 before colliding with an obstacle:

[0022]

[0023] dist(v, w) is the shortest distance from the obstacle to the trajectory corresponding to the velocity-angular velocity combination (v, w);

[0024] (4) Minimum turning radius constraint w a : The turning radius of the smart car during steering shall not be less than the minimum turning radius:

[0025]

[0026] v is the speed of the smart car, α lim is the maximum turning angle of the smart car, r min is the minimum turning radius of the smart car; in summary, the speed space constraint V v It is expressed as:

[0027] V v =V m ∩V d ∩V a ∩w a

[0028] Motion Model

[0029] When the time interval between movements is small enough, the trajectory between two adjacent points will be close to uniform linear motion. The specific formula is as follows:

[0030]

[0031] x t ,y t represents the horizontal and vertical coordinates of the smart car at time t, represents the heading angle at time t, v t 、w t represents the velocity-angular velocity combination at time t, Δ t is the exercise time interval;

[0032] Evaluation function

[0033] G(v,w)=σ(αheading(v,w)+ζdist(v,w)+γvel(v,w))

[0034] Where σ() is a smoothing function, represents the heading angle of the smart car, φ is the angle between the line connecting the position of the smart car and the target point and the positive direction of the x-axis, dist(v,w) is the Euclidean distance from the simulated trajectory to the nearest obstacle, vel(v,w) represents the linear velocity of the smart car, and α, ζ, and γ are three weight coefficients.

[0035] Furthermore, in step S2:

[0036] The intelligent agent is specifically an Ackerman intelligent car, and the environment is specifically a virtual obstacle environment;

[0037] The action and state space are specifically:

[0038] The output action space A of the neural network is defined as [α, ζ, γ], and the final output is obtained by transformation through the activation function f(x) = max(0.01, x), which corresponds to the weight parameters of the three sub-functions heading(v, w), dist(v, w), and vel(v, w) in the evaluation function G(v, w); the state space S is defined as the five obstacles with distances from the smart car from near to far and the position coordinates of the smart car at time t (x t ,y t ) and the orientation angle of the smart car Combinations such as Figure 2 As shown, the numerical value is normalized, S is an 11-dimensional space, as shown in the following formula:

[0039]

[0040] Where x m 、x n is the upper and lower bounds of the obstacle’s horizontal coordinate, y m ,y n are the upper and lower bounds of the obstacle’s ordinate;

[0041] The reward function is specifically:

[0042] The reward function of the smart car at time t consists of 8 parts, defined as follows:

[0043]

[0044] Is the reward given to the smart car whether it reaches the target point:

[0045]

[0046] Are rewards associated with smart car collisions:

[0047]

[0048] is the reward associated with the number of steps taken by the smart car:

[0049]

[0050] is the reward associated with the change in position of the smart car:

[0051]

[0052] Is a reward related to the smart car line speed:

[0053]

[0054] is the reward related to the direction of the smart car:

[0055]

[0056] is the reward associated with the distance from the smart car to the nearest obstacle:

[0057]

[0058] In the formula =-1, indicating the cost of each step of the smart car, p t represents the coordinates of the smart car at time t, g represents the coordinates of the end point, ob t represents the coordinates of the obstacle closest to the smart car at time t, d t represents the distance between the smart car and the destination at time t, d t-1 represents the distance between the smart car and the end point at time t-1, v t represents the linear velocity of the smart car at time t, Φ t represents the angle between the line connecting the smart car and the end point and its own orientation angle at time t;

[0059] The network architecture specifically includes: an agent, an environment, a critic module, and an actor module;

[0060] The critic module includes a value network objective function L(ω), a value network q(s,a;ω), a target value network q T (s,a;ω);

[0061] The actor module includes a policy network π(a|s; θ), a policy network objective function J(θ);

[0062] The intelligent agent is a smart car. At time t, the smart car uses the perception and positioning system to observe the state s from the environment. t , the policy network π(a|s;θ) receives s t Then output an action A t The Gaussian probability distribution π(a t |s t ; θ), and then randomly extract the action a from the probability distribution t , action a t The information is passed to the value network q(s,a;ω), which is consistent with the state s observed from the environment t The inputs that form q(s,a;ω) are composed; the action a is determined by π(a|s;θ) t The information is also passed to the target value network q T (s,a;ω), and the state s observed from the environmentt Form q T (s, a; ω) input; according to the Gaussian probability distribution π(A t |s t ;θ) and action a t Information to calculate the state s t Next, perform action a t The probability p θ (a t |s t ), which is used to calculate the objective function J(θ) of the actor module; π(a|s; θ) determines the action a t The information is also transmitted to the smart car, and the evaluation function G of the DWA algorithm with window reduction at time t is obtained. t (v,w),G t (v,w) completes the simulation trajectory set τ of the DWA algorithm at time t t The speed-angular velocity combination (v t ,w t ), the speed-angular velocity information is transmitted to the motion control module of the smart car to drive the smart car to move; at this point, the position, orientation angle, and surrounding obstacle distribution information of the smart car have changed, and the environment is converted to state s t+1 The reward function also feeds back the critic module reward r based on the changed information t The role of the two objective functions L(ω) and J(θ) is to update the network parameters q(s,a;ω) and π(a|s;θ).

[0063] Furthermore, the training set collection process, trajectory set ψ, and network architecture parameter update in step S4 and step S5 are specifically as follows:

[0064] Complete value network q(s,a;ω), target value network q T After the construction of the strategy network π(a|s;θ), the multi-step temporal difference algorithm MTD is used to update the value network parameter ω, and the proximal policy optimization algorithm PPO is used to update the strategy network parameter θ; let the current parameter be ω now , θ now , repeat the following steps to update the network parameters q(s,a;ω),π(a|s;θ):

[0065] (1) The initial state s is actually observed 0 ,π(a|s;θ) outputs an action A 0 The Gaussian probability distribution π(A 0 |s 0 ; θ), and then randomly extract the action a from the probability distribution 0 , the smart car performs action a0 , observe the reward r 0 、s 1 , and so on, until the terminal state s is observed n , generate data χ 0 :[s 0 0 ,a 0 0 ,r 0 0 …,s n-1 0 ,a n-1 0 ,r n-1 0 ,s n 0 ], and so on, until i data sets χ=[χ 0 ,x 1 ,…,χ i ], i represents round;

[0066] (2) Repeat the following steps within the specified number of updates (Z):

[0067] Randomly extract M from the data set χ I (minimum batch size) states s N I , and then traverse the M I States N I :

[0068] Using the target value network q T (s,a;ω) calculates the state s N I is the K-step timing difference (TD) error G from the starting point N I :

[0069]

[0070] Use the value network q(s,a;ω) to calculate the state s N I The action value estimate q t :

[0071] q t =q(s N I ,a N I ;ω now )

[0072] L(ω) is calculated according to the following formula:

[0073]

[0074] Calculate J(θ) according to the following:

[0075]

[0076] Update the parameters of the value network, policy network, and target value network:

[0077]

[0078] is the advantage function, which is used to measure the superiority of performing action a N I in state s N I compared with other actions:

[0079]

[0080] s N I represents the state of the Nth step in episode I, and n I represents the maximum number of steps in episode I. α, β, μ, λ, κ are hyperparameters, and p θ (a N I |s N I ) represents the probability of performing a N I in state s N I by the new policy network; represents the probability under the old policy network. clip(a, b, c) means that when a ∈ [b, c], output a; when a < b, output b; when a > c, output c.

[0081] Furthermore, the average score of each episode referred to in step S6 is specifically: During the process of the intelligent vehicle exploring the end point from the starting point, the cumulative sum of the dense rewards obtained for each step and the final sparse reward. The dense rewards specifically include the five items in the reward function. Among them, the dense reward is the feedback obtained by the intelligent vehicle for each step, and the sparse reward specifically includes the The sparse reward is the reward given when the intelligent vehicle reaches the termination state; the termination state includes: reaching the end point, colliding with an obstacle, or exceeding the maximum number of steps in the episode.

[0082] Compared with the prior art, the adaptive DWA algorithm proposed by the present invention and its preferred solution can effectively avoid local optima and reach the end point, while the optimal fixed-weight parameter DWA algorithm falls into local optima. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic diagram of an obstacle environment according to an embodiment of the present invention;

[0084] Figure 2 This is a schematic diagram of a state vector of an embodiment of the present invention;

[0085] Figure 3 This is a diagram showing the results of the adaptive DWA algorithm according to an embodiment of the present invention;

[0086] Figure 4 This is a diagram showing the optimal fixed weight parameter DWA algorithm result according to an embodiment of the present invention;

[0087] Figure 5 Schematic diagram of parameter change curve of adaptive DWA algorithm according to an embodiment of the present invention;

[0088] Figure 6 This is a schematic diagram of a deep reinforcement learning architecture model according to an embodiment of the present invention;

[0089] Figure 7 The figure is a schematic diagram of the overall flow of the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0091] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0092] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0093] like Figure 7 As shown, the steps and process of the solution provided in this embodiment are as follows:

[0094] Step S1: Establish a DWA algorithm model, and determine the speed range, angular velocity range, acceleration range, angular acceleration range and other parameters and the core elements of the DWA algorithm according to the Ackerman smart car.

[0095] The core elements of the DWA algorithm described in step S1 include speed space constraints, motion model, and evaluation function:

[0096] Velocity space constraint V v

[0097] (1) Limit speed constraint V m : The feasible speed of smart cars shall not exceed the limit range:

[0098] V m ={(v,w)|v∈[v min ,v max ]∧w∈[w min ,w max ]}

[0099] In the formula [v min ,v max ]Linear speed range, [w min ,w max ] is the angular velocity range;

[0100] (2) Acceleration constraint V d : The speed and angular velocity increments of the smart car at adjacent moments are subject to the acceleration and angular acceleration constraints:

[0101]

[0102] v c 、w c is the current linear velocity and angular velocity, is the linear acceleration range, is the angular acceleration range, Δ t is the movement time interval, and also the simulation trajectory sampling time interval;

[0103] (3) Braking distance constraint V a : The smart car should be able to decelerate to 0 before colliding with an obstacle:

[0104]

[0105] dist(v, w) is the shortest distance from the obstacle to the trajectory corresponding to the velocity-angular velocity combination (v, w);

[0106] (4) Minimum turning radius constraint w a : The turning radius of the smart car during steering shall not be less than the minimum turning radius:

[0107]

[0108] v is the speed of the smart car, α lim is the maximum turning angle of the smart car, r min is the minimum turning radius of the smart car; in summary, the speed space constraint V v It is expressed as:

[0109] V v =Vm ∩V d ∩V a ∩w a

[0110] Motion Model

[0111] When the time interval between movements is small enough, the trajectory between two adjacent points will be close to uniform linear motion. The specific formula is as follows:

[0112]

[0113] x t ,y t represents the horizontal and vertical coordinates of the smart car at time t, represents the heading angle at time t, v t 、w t represents the velocity-angular velocity combination at time t, Δ t is the exercise time interval;

[0114] Evaluation function

[0115] G(v,w)=σ(αheading(v,w)+ζdist(v,w)+γvel(v,w))

[0116] Where σ() is a smoothing function, represents the heading angle of the smart car, φ is the angle between the line connecting the position of the smart car and the target point and the positive direction of the x-axis, dist(v,w) is the Euclidean distance from the simulated trajectory to the nearest obstacle, vel(v,w) represents the linear velocity of the smart car, and α, ζ, and γ are three weight coefficients.

[0117] Step S2: Step S2: Construct a deep reinforcement learning model for the DWA algorithm, including elements such as agent, environment, action and state space, reward function, and network architecture.

[0118] The intelligent agent described in step S2 is specifically the Ackerman intelligent car, and the environment is specifically a virtual obstacle environment, such as Figure 1 shown.

[0119] The actions and state spaces described in step S2 are specifically:

[0120] The present invention aims to enable the DWA algorithm to adaptively adjust the three weight parameters in the evaluation function when facing different obstacle environments, so that the smart car can select the optimal solution in the current state. Therefore, the output action space A of the neural network is defined as [α, ζ, γ], and the final output is obtained by the activation function f(x) = max(0.01, x) transformation, which corresponds to the weight parameters of the three sub-functions heading(v, w), dist(v, w), and vel(v, w) in the evaluation function G(v, w); the state space S is defined as the five obstacles with distances from the smart car from near to far and the position coordinates of the smart car at time t (x t ,y t ) and the orientation angle of the smart car Combinations such as Figure 2 As shown, the numerical value is normalized, S is an 11-dimensional space, as shown in the following formula:

[0121]

[0122] Where x m 、x n is the upper and lower bounds of the obstacle’s horizontal coordinate, y m ,y n are the upper and lower bounds of the obstacle's ordinate.

[0123] The reward function described in step S2 is specifically:

[0124] The reward function of the smart car at time t consists of 8 parts, defined as follows:

[0125]

[0126] Is the reward given to the smart car whether it reaches the target point:

[0127]

[0128] Are rewards associated with smart car collisions:

[0129]

[0130] is the reward associated with the number of steps taken by the smart car:

[0131]

[0132] is the reward associated with the change in position of the smart car:

[0133]

[0134] Is a reward related to the smart car line speed:

[0135]

[0136] is the reward related to the direction of the smart car:

[0137]

[0138] is the reward associated with the distance from the smart car to the nearest obstacle:

[0139]

[0140] In the formula =-1, indicating the cost of each step of the smart car, p t represents the coordinates of the smart car at time t, g represents the coordinates of the end point, ob t represents the coordinates of the obstacle closest to the smart car at time t, d t represents the distance between the smart car and the destination at time t, d t-1 represents the distance between the smart car and the end point at time t-1, v t represents the linear velocity of the smart car at time t, Φ t It represents the angle between the line connecting the smart car and the end point and its own orientation angle at time t.

[0141] The network architecture described in step S2 is specifically:

[0142] The deep reinforcement learning architecture model referred to in the present invention is as follows Figure 6 , including agent, environment, critic module, and actor module.

[0143] The critic module includes the value network objective function L(ω), the value network q(s,a;ω), the target value network q T (s, a; ω); the actor module includes the policy network π(a|s; θ) and the policy network objective function J(θ); the intelligent agent is the smart car, which uses the perception and positioning system to observe the state s from the environment at time t t , the policy network π(a|s;θ) receives s t Then output an action A t The Gaussian probability distribution π(a t |s t ; θ), and then randomly extract the action a from the probability distribution t , action a t The information is passed to the value network q(s,a;ω), which is consistent with the state s observed from the environment tThe inputs that form q(s,a;ω) are composed; the action a is determined by π(a|s;θ) t The information is also passed to the target value network q T (s,a;ω), and the state s observed from the environment t Form q T (s, a; ω) input; according to the Gaussian probability distribution π(A t |s t ;θ) and action a t Information to calculate the state s t Next, perform action a t The probability p θ (a t |s t ), which is used to calculate the objective function J(θ) of the actor module; π(a|s; θ) determines the action a t The information is also transmitted to the smart car, and the evaluation function G of the DWA algorithm with window reduction at time t is obtained. t (v,w),G t (v,w) completes the simulation trajectory set τ of the DWA algorithm at time t t The speed-angular velocity combination (v t ,w t ), the speed-angular velocity information is transmitted to the motion control module of the smart car to drive the smart car to move; at this point, the position, orientation angle, and surrounding obstacle distribution of the smart car have changed, and the environment has changed to state s t+1 The reward function will also give feedback to the critic module based on the changed information. t The role of the two objective functions L(ω) and J(θ) is to update the network parameters q(s,a;ω) and π(a|s;θ).

[0144] Step S4: Training set collection phase: simulate the movement of the intelligent agent in a randomly generated obstacle map and collect the trajectory set ψ for the gradient descent of the neural network.

[0145] Step S5: Network parameter update phase: Use the trajectory set ψ collected in step S4 to update various network architecture parameters.

[0146] The training set collection process, trajectory set ψ, and network architecture parameter update referred to in steps S4 and S5 are specifically as follows:

[0147] Complete value network q(s,a;ω), target value network q T After building the value network parameter ω (s, a; ω) and the policy network π (a|s; θ), the multi-step temporal difference algorithm (MTD) is used to update the value network parameter ω, and the proximal policy optimization algorithm (PPO) is used to update the policy network parameter θ; let the current parameter be ωnow , θ now , repeat the following steps to update the network parameters q(s,a;ω),π(a|s;θ):

[0148] (1) The initial state s is actually observed 0 ,π(a|s;θ) outputs an action A 0 The Gaussian probability distribution π(A 0 |s 0 ; θ), and then randomly extract the action a from the probability distribution 0 , the smart car performs action a 0 , observe the reward r 0 、s 1 , and so on, until the terminal state s is observed n , generate data χ 0 :[s 0 0 ,a 0 0 ,r 0 0 …,s n-1 0 ,a n-1 0 ,r n-1 0 ,s n 0 ], and so on, until i data sets χ=[χ 0 ,x 1 ,…,χ i ], i represents round;

[0149] (2) Repeat the following steps within the specified number of updates (Z):

[0150] Randomly extract M from the data set χ I (minimum batch size) states s N I , and then traverse the M I States N I :

[0151] Using the target value network q T (s,a;ω) calculates the state s N I is the K-step timing difference (TD) error G from the starting point N I :

[0152]

[0153] Calculate the state s using the value network q(s,a;ω). N I The action-value estimate q at time s t :

[0154] q t =q(s N I ,a N I ;ω now )

[0155] Calculate L(ω) according to the following formula:

[0156]

[0157] Calculate J(θ) according to the following:

[0158]

[0159] Update the parameters of the value network, policy network, and target value network:

[0160]

[0161] is the advantage function, which is used to measure the superiority of performing action a N I compared to other actions in state s N I :

[0162]

[0163] s N I represents the state at the Nth step of episode I, n I represents the maximum number of steps in episode I, and α, β, μ, λ, κ are hyperparameters. p θ (a N I |s N I ) represents the probability of performing a N I in the new policy network in state s N I ; represents the probability under the old policy network. clip(a,b,c) means that when a ∈ [b,c], a is output; when a < b, b is output; when a > c, c is output.

[0164] Step S6: Determine whether the average score of the agent under the current network architecture in an episode is greater than the threshold. If so, terminate the loop; otherwise, re-enter Step S3 for a new generation of training.

[0165] The round average score referred to in step S6 is specifically the cumulative sum of the dense reward obtained by the intelligent car at each step in the process of exploring the end point from the starting point and the final sparse reward. The dense reward specifically includes The dense reward is the feedback that the smart car will get every time it takes a step, and the sparse reward specifically includes The sparse reward is the reward given when the intelligent car reaches the terminal state (reaches the end point, collides with an obstacle, exceeds the maximum number of steps in the round).

[0166] In order to verify the effectiveness of the algorithm proposed in this paper, a complex static obstacle environment is randomly initialized based on the Python 3 language platform. Figure 1 As shown, the starting point of the smart car is set to (-25, 25), the end point is set to (30, 30), the black circle is a randomly initialized obstacle, and the radius of the obstacle itself is 0.5m. A series of comparative experiments are carried out including: the adaptive DWA algorithm proposed in the present invention, the random weight parameter DWA algorithm, and the optimal fixed weight parameter DWA algorithm. Each algorithm is tested 6000 times in a randomly generated map environment.

[0167] Table 1 Static obstacle environment simulation experiment table

[0168]

[0169] It can be seen that the arrival rate of the adaptive DWA algorithm is improved by 14.89% and 37.57% respectively compared with the optimal fixed weight parameter and random weight parameter DWA algorithms; the path efficiency is improved by 2.79% and 0.41% respectively; and the time efficiency is improved by 6.60% and 2.36% respectively. Figure 3 This is a simulation of the adaptive DWA algorithm in a random static obstacle environment. Figure 5 The corresponding weight parameter change curve. Figure 4 The simulation of the optimal fixed weight parameter DWA algorithm under the same environment. The adaptive DWA algorithm proposed in this invention effectively avoids the local optimum and reaches the end point, while the optimal fixed weight parameter DWA algorithm falls into the local optimum.

[0170] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0171] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of adaptive dynamic window methods based on deep reinforcement learning under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. An adaptive dynamic window method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1: Establish a DWA algorithm model, based on the Ackerman smart car, including: speed range [v min ,v max ]、Angular velocity range [w min ,w max ], Acceleration range Angular acceleration range Parameters and elements of the DWA algorithm; Step S2: Build a deep reinforcement learning model for the DWA algorithm, including the agent, environment, action and state space, reward function, and deep reinforcement learning architecture; Step S3: randomly initialize the obstacle map environment, initialize the smart car speed v0, angular velocity w0, and heading angle θ0 to 0, place the smart car at the starting point p0, and set the end point to g; Step S4: Training set collection phase: simulate the movement of the intelligent agent in the randomly generated obstacle map and collect the trajectory set ψ for the gradient descent of the neural network; Step S5: Network parameter update phase: Use the trajectory set ψ collected in step S4 to update each network architecture parameter; Step S6: Determine whether the average round score of the agent under the current network architecture is greater than the threshold. If so, terminate the loop, otherwise re-enter step S3 for the next generation of training; In step S1, the core elements of the DWA algorithm include: speed space constraint, motion model, and evaluation function: Velocity space constraint V v (1) Limit speed constraint V m : The feasible speed of smart cars shall not exceed the limit range: V m ={(v,w)|v∈[v min ,v max ]∧w∈[w min ,w max ]} In the formula [v min ,v max ]Linear speed range, [w min ,w max ] is the angular velocity range; (2) Acceleration constraint V d : The speed and angular velocity increments of the smart car at adjacent moments are subject to the acceleration and angular acceleration constraints: v c 、w c is the current linear velocity and angular velocity, is the linear acceleration range, is the angular acceleration range, Δ t is the movement time interval, and also the simulation trajectory sampling time interval; (3) Braking distance constraint V a : The smart car should be able to decelerate to 0 before colliding with an obstacle: dist(v, w) is the shortest distance from the obstacle to the trajectory corresponding to the velocity-angular velocity combination (v, w); (4) Minimum turning radius constraint w a : The turning radius of the smart car during steering shall not be less than the minimum turning radius: v is the speed of the smart car, α lim is the maximum turning angle of the smart car, r min is the minimum turning radius of the smart car; in summary, the speed space constraint V v It is expressed as: V v =V m ∩V d ∩V a ∩w a Motion Model When the time interval between movements is small enough, the trajectory between two adjacent points will be close to uniform linear motion. The specific formula is as follows: x t ,y t represents the horizontal and vertical coordinates of the smart car at time t, θ t represents the heading angle at time t, v t 、w t represents the velocity-angular velocity combination at time t, and Δt is the time interval of motion; Evaluation function G(v,w)=σ(αheading(v,w)+ζdist(v,w)+γvel(v,w)) Where σ() is a smoothing function, heading(v,w)=π-|θ-φ|, θ represents the heading angle of the smart car, φ is the angle between the line connecting the position of the smart car and the target point and the positive direction of the x-axis, dist(v,w) is the Euclidean distance from the simulated trajectory to the nearest obstacle, vel(v,w) represents the linear velocity of the smart car, and α, ζ, and γ are three weight coefficients; In step S2: The intelligent agent is specifically an Ackerman intelligent car, and the environment is specifically a virtual obstacle environment; The action and state space are specifically: The output action space A of the neural network is defined as [α, ζ, γ], and the final output is obtained by transformation through the activation function f(x) = max(0.01, x), which corresponds to the weight parameters of the three sub-functions heading(v, w), dist(v, w), and vel(v, w) in the evaluation function G(v, w); the state space S is defined as the five obstacles with distances from the smart car from near to far and the position coordinates of the smart car at time t (x t ,y t ) and the orientation angle θ of the smart car t The combination of normalized values, S is an 11-dimensional space, as shown in the following formula: Where x m 、x n is the upper and lower bounds of the obstacle’s horizontal coordinate, y m ,y n are the upper and lower bounds of the obstacle’s ordinate; The reward function is specifically: The reward function of the smart car at time t consists of 8 parts, defined as follows: Is the reward given to the smart car whether it reaches the target point: Are rewards associated with smart car collisions: is the reward associated with the number of steps taken by the smart car: is the reward associated with the change in position of the smart car: Is a reward related to the smart car line speed: is the reward related to the direction of the smart car: is the reward associated with the distance from the smart car to the nearest obstacle: In the formula =-1, indicating the cost of each step of the smart car, p t represents the coordinates of the smart car at time t, g represents the coordinates of the end point, ob t represents the coordinates of the obstacle closest to the smart car at time t, d t represents the distance between the smart car and the destination at time t, d t-1 represents the distance between the smart car and the end point at time t-1, v t represents the linear velocity of the smart car at time t, Φ t represents the angle between the line connecting the smart car and the end point and its own orientation angle at time t; The network architecture specifically includes: an agent, an environment, a critic module, and an actor module; The critic module includes a value network objective function L(ω), a value network q(s,a;ω), a target value network q T (s,a;ω); The actor module includes a policy network π(a|s; θ), a policy network objective function J(θ); The intelligent agent is a smart car. At time t, the smart car uses the perception and positioning system to observe the state s from the environment. t , the policy network π(a|s;θ) receives s t Then output an action A t The Gaussian probability distribution π(a t |s t ; θ), and then randomly extract the action a from the probability distribution t , action a t The information is passed to the value network q(s,a;ω), which is consistent with the state s observed from the environment t The inputs that form q(s,a;ω) are composed; the action a is determined by π(a|s;θ) t The information is also passed to the target value network q T (s,a;ω), and the state s observed from the environment t Form q T (s, a; ω) input; according to the Gaussian probability distribution π(A t |s t ;θ) and action a t Information to calculate the state s t Next, perform action a t The probability p θ (a t |s t ), which is used to calculate the objective function J(θ) of the actor module; π(a|s; θ) determines the action a t The information is also transmitted to the smart car, and the evaluation function G of the DWA algorithm with window reduction at time t is obtained. t (v,w),G t (v,w) completes the simulation trajectory set τ of the DWA algorithm at time t t The speed-angular velocity combination (v t ,w t ), the speed-angular velocity information is transmitted to the motion control module of the smart car to drive the smart car to move; at this point, the position, orientation angle, and surrounding obstacle distribution information of the smart car have changed, and the environment is converted to state s t+1 The reward function also feeds back the critic module reward r based on the changed information t The role of the two objective functions L(ω) and J(θ) is to update the network parameters q(s,a;ω) and π(a|s;θ).

2. The adaptive dynamic window method based on deep reinforcement learning according to claim 1, characterized in that: The training set collection process, trajectory set ψ, and network architecture parameter update in step S4 and step S5 are as follows: Complete value network q(s,a;ω), target value network q T After the construction of the strategy network π(a|s;θ), the multi-step temporal difference algorithm MTD is used to update the value network parameter ω, and the proximal policy optimization algorithm PPO is used to update the strategy network parameter θ; let the current parameter be ω now , θ now , repeat the following steps to update the network parameters q(s,a;ω),π(a|s;θ): (1) The initial state s0 is actually observed, π(a|s; θ) outputs a Gaussian probability distribution π(A0|s0; θ) about action A0, and then randomly extracts action a0 from the probability distribution. The smart car performs action a0 and observes rewards r0 and s1, and repeats this process until the terminal state s is observed. n , generate data χ 0 :[s0 0 ,a0 0 ,r0 0 …,s n-1 0 ,a n-1 0 ,r n-1 0 ,s n 0 ], and so on, until i data sets χ=[χ 0 ,x 1 ,…,χ i ], i represents round; (2) Repeat the following steps within the specified number of updates (Z): Randomly draw a minimum batch size M from the data set χ I States N I , and then traverse the M I State S N I : Using the target value network q T (s,a;ω) calculates the state s N I is the K-step timing difference (TD) error G from the starting point N I : Use the value network q(s,a;ω) to calculate the state s N I The action value estimate q t : q t =q(s N I ,a N I ;ω now ) L(ω) is calculated according to the following formula: J(θ) is calculated as follows: Update the parameters of the value network, policy network, and target value network: is the advantage function, which is used to measure N I Next, perform action a N I Superiority compared to other actions: s N I represents the state of the Nth step in round I, n I represents the maximum number of steps in round I, and α, β, μ, λ, κ are hyperparameters, p θ (a N I |s N I ) represents the probability of executing a N I under the new policy network in state s N I ; p θk (a N I |s N I ) represents the probability under the old policy network. clip(a, b, c) means that when a ∈ [b, c], output a; when a < b, output b; when a > c, output c.

3. The adaptive dynamic window method based on deep reinforcement learning according to claim 2, characterized in that: The round average score in step S6 is specifically the cumulative sum of the dense reward obtained by the intelligent car at each step in the process of exploring the end point from the starting point and the final sparse reward. The dense reward specifically includes R in the reward function. t cost , R t p , R t vel , R t θ , R t ob The dense reward is the feedback obtained by the smart car every time it takes a step, and the sparse reward specifically includes R in the reward function. t goal , R t obs , R t step , the sparse reward is the reward given when the smart car reaches the terminal state; the terminal state includes: reaching the end point, colliding with an obstacle, or exceeding the maximum number of steps in the round.

Images