Unmanned vehicle adaptive path planning method based on dynamic window method and near-end strategy

Abstract

The application relates to an unmanned vehicle adaptive path planning method based on a dynamic window method and a proximal strategy. First, an agent-environment interaction model for an unmanned vehicle is constructed, a proximal strategy optimization learning (PPO) model based on an actor-critic framework is established, a reward function is defined according to the principle of a dynamic window method algorithm (DWA) and main evaluation elements, model parameters in the input layer, the output layer size, the number of hidden layers and the number of neurons are determined, and a DWA-PPO deep reinforcement learning model is constructed. Then, the established DWA-PPO deep reinforcement learning model is used for iterative training, and finally, a network model capable of representing the potential relationship between surrounding environment information and evaluation function weight parameters is converged, and the construction of the adaptive PPO-ADWA algorithm is completed. Finally, the feasibility and effectiveness of the unmanned vehicle adaptive path planning strategy based on the PPO-ADWA are verified through simulation comparison experiments.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving path planning and autonomous navigation technology, specifically to an adaptive path planning method for unmanned vehicles based on dynamic windowing and proximity strategies. Background Technology

[0002] In recent years, with the rapid development of science and technology, a new round of technological and industrial revolution, represented by the Internet, artificial intelligence, and big data, is redefining all sectors of society, and the traditional automotive industry is facing profound industrial transformation. Traditional automobiles are developing towards intelligence and automation, with intelligent connected vehicles and autonomous vehicles becoming strategic directions for the global automotive industry. Intelligent driving technology mainly includes environmental perception, navigation and positioning, path planning, and control decision-making. Path planning is a crucial component of intelligent driving and is of great significance to the development of intelligent driving technology.

[0003] Path planning is a crucial component of autonomous vehicles. Path planning technology can be summarized as the process of planning a safe, feasible, collision-free path within a known environment using algorithms, selecting the optimal obstacle-avoidance path connecting the starting point to the destination. Essentially, it's about finding the optimal solution under several constraints, and it's a key part of autonomous vehicle navigation technology. Path planning algorithms can be further divided into global planning based on complete area information understanding and local planning based on local area information understanding. The Dynamic Window Approach (DWA), as a local path planning method considering the motion performance of autonomous vehicles, is widely used in autonomous vehicle path navigation. The decision-making mechanism in the DWA algorithm is its evaluation function, which includes three parts: orientation angle function, obstacle function, and velocity function. The evaluation function is a weighted sum of these three sub-functions. In the classic DWA algorithm, the weights corresponding to these three functions are fixed values. However, during the process of exploring the destination, the surrounding obstacle environment of the autonomous vehicle is complex and variable. Different obstacle distributions require different weights. The fixed weight value method of the classic DWA algorithm can easily cause the autonomous vehicle to get trapped in local optima or the target to become unreachable. Therefore, the classic DWA algorithm is improved by using the proximal policy optimization algorithm in deep reinforcement learning. Summary of the Invention

[0004] The purpose of this invention is to address the problem that intelligent agents often fail to find the destination or calculate the optimal path when facing different obstacle environments because the weight coefficients in the evaluation function cannot be dynamically adjusted. This invention provides an adaptive path planning method for unmanned vehicles based on the dynamic window method and a proximal strategy. An improvement is proposed on the classic DWA algorithm, combining the weight parameters in the classic DWA algorithm with the proximal strategy optimization in deep reinforcement learning. Through learning and training, model parameters applicable to different static obstacles are obtained, thus completing the construction of the adaptive PPO-ADWA algorithm.

[0005] To achieve the above objectives, the technical solution of the present invention is: an adaptive path planning method for unmanned vehicles based on dynamic window method and near-end strategy, comprising the following steps:

[0006] Step 1: Construct an agent-environment interaction model, with the autonomous vehicle acting as the agent in deep reinforcement learning and the obstacle map as the environment;

[0007] Step 2: Establish the DWA algorithm model, and determine the parameters including: speed range, angular velocity range, acceleration range, angular acceleration range, as well as the main elements and evaluation functions of the DWA algorithm based on the Ackerman intelligent vehicle.

[0008] Step 3: Establish a proximal policy optimization learning (PPO) model based on the actor-critic framework, simulate the actual application scenario of the autonomous vehicle as the learning environment of the model, and determine the state and action in the model according to the application scenario.

[0009] Step 4: Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main reward and the sub-objective reward; and determine the parameters of the DWA-PPO deep reinforcement learning model, including the size of the input layer, the output layer, the number of hidden layers and the number of neurons, and complete the instantiation of the DWA-PPO deep reinforcement learning model.

[0010] Step 5: Construct the adaptive PPO-ADWA algorithm. Using the established DWA-PPO deep reinforcement learning model, simulate the navigation planning of the unmanned vehicle in a randomly generated complex static obstacle environment to collect a training set for training the DWA-PPO deep reinforcement learning model. Through repeated iterations, converge a model that can output corresponding weight parameters according to the changes in the distribution of surrounding obstacles, and complete the construction of the adaptive PPO-ADWA algorithm.

[0011] Step 6: Demonstrate the adaptive adjustment capability of unmanned vehicle path planning based on the adaptive PPO-ADWA algorithm through simulation comparison experiments.

[0012] Compared with existing technologies, this invention has the following advantages: The method of this invention addresses the problem that the weight coefficients in the evaluation function of the traditional DWA algorithm do not dynamically adjust according to the environment and motion state of the intelligent vehicle. It uses the proximal policy optimization algorithm in deep reinforcement learning to construct a DWA-PPO deep reinforcement learning model. Through iterative training, the network model is obtained, thereby outputting the model parameters of the corresponding weight parameters, completing the construction of the adaptive PPO-ADWA algorithm. This invention also solves the problem that when facing different obstacle environments, the agent often cannot find the destination or calculate the optimal path because the weight coefficients in the evaluation function cannot be dynamically adjusted. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of an agent-environment interaction model.

[0014] Figure 2 This is a schematic diagram illustrating the principle of the DWA algorithm.

[0015] Figure 3 For velocity and angular velocity windows.

[0016] Figure 4 for Diagram with δ.

[0017] Figure 5 A schematic diagram of the actor critic framework.

[0018] Figure 6 Let s be the state.

[0019] Figure 7 This is a policy network structure.

[0020] Figure 8 It is a value network structure.

[0021] Figure 9 It is the DWA-PPO model.

[0022] Figure 10 The curve shows the change in score versus arrival rate.

[0023] Figure 11 This is a simulation environment.

[0024] Figure 12 This is a classic DWA.

[0025] Figure 13 for PPO-ADWA.

[0026] Figure 14 This is the curve showing the change in the weighting parameters.

[0027] Figure 15 This is a flowchart of the method of the present invention. Detailed Implementation

[0028] The following is in conjunction with the appendix Figure 1-15 The technical solution of the present invention will be described in detail below.

[0029] like Figure 15 As shown, this invention provides an adaptive path planning method for autonomous vehicles based on dynamic windowing and proximity strategies, comprising the following steps:

[0030] Step 1: Construct an agent-environment interaction model, with the autonomous vehicle acting as the agent in deep reinforcement learning and the obstacle map as the environment;

[0031] Step 2: Establish the DWA algorithm model, and determine the parameters including: speed range, angular velocity range, acceleration range, angular acceleration range, as well as the main elements and evaluation functions of the DWA algorithm based on the Ackerman intelligent vehicle.

[0032] Step 3: Establish a proximal policy optimization learning (PPO) model based on the actor-critic framework, simulate the actual application scenario of the autonomous vehicle as the learning environment of the model, and determine the state and action in the model according to the application scenario.

[0033] Step 4: Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main reward and the sub-objective reward; and determine the parameters of the DWA-PPO deep reinforcement learning model, including the size of the input layer, the output layer, the number of hidden layers and the number of neurons, and complete the instantiation of the DWA-PPO deep reinforcement learning model.

[0034] Step 5: Construct the adaptive PPO-ADWA algorithm. Using the established DWA-PPO deep reinforcement learning model, simulate the navigation planning of the unmanned vehicle in a randomly generated complex static obstacle environment to collect a training set for training the DWA-PPO deep reinforcement learning model. Through repeated iterations, converge a model that can output corresponding weight parameters according to the changes in the distribution of surrounding obstacles, and complete the construction of the adaptive PPO-ADWA algorithm.

[0035] Step 6: Demonstrate the adaptive adjustment capability of unmanned vehicle path planning based on the adaptive PPO-ADWA algorithm through simulation comparison experiments.

[0036] The specific implementation of each step is as follows:

[0037] Step 1, such as Figure 1 As shown, an agent-environment interaction model is constructed, with the autonomous vehicle as the agent in deep reinforcement learning and the obstacle map as the environment.

[0038] In deep reinforcement learning systems, the intelligent agent plays the role of decision-making and learning, primarily responsible for outputting action information and receiving rewards and states. The environment is the interactive object of the intelligent agent, and its interaction process includes the following three steps:

[0039] (1) The agent is influenced by the environmental state Observed information The state space is the set of possible values ​​for the environment states. Let be the observation space, which is the set of values ​​for the agent's observations.

[0040] (2) The agent is composed of known O t Make appropriate decisions and decide on the actions to be taken in response to the environment. It is the set of action values.

[0041] (3) The environment is affected by A t Impact, own state S t Transfer to S t+1 And reward the intelligent agent. It is the set of possible reward values. Therefore, the discretized agent-environment interaction model can be represented by the following sequence:

[0042] S0,O0,A0,R0,S1,O1,A1,R1,S2,O2,A2,R2,…,S T =S 终止

[0043] When the state of the environment can be fully observed by the agent, then S t =O t To simplify:

[0044] S0,A0,R0,S1,A1,R1,S2,A2,R2,…,S T =S 终止

[0045] Step 2: Establish the DWA algorithm model, and determine the parameters including: speed range, angular velocity range, acceleration range, angular acceleration range, and evaluation function of the main elements of the DWA algorithm based on the Ackerman intelligent vehicle.

[0046] The Direct Path Analysis (DWA) algorithm is a local path planning method that provides an intuitive understanding of the map environment in which an autonomous vehicle is located from a velocity space perspective. Its workflow is as follows: considering the constraints of various conditions on velocity and angular velocity at time t, it calculates the velocity and angular velocity window V that the autonomous vehicle can reach at time t. win Discretize it, and combine the discretized velocities and angular velocities; the unmanned vehicle traverses all combinations and simulates moving forward m Δ steps according to the given motion model. t The duration is used to obtain the simulated trajectory set τ, which is a series of point sets; the evaluation function gives the score of all simulated trajectories in the simulated trajectory set τ, and the trajectory τ with the highest score is selected. b The corresponding combination; the time Δ for driving the autonomous vehicle forward using this combination. t The sampling process continues until time t+1 is reached; this cycle repeats until the endpoint is reached. m is the number of sampling steps, Δ... t For example, the sampling interval is... Figure 2 As shown.

[0047] At time t, the V of the autonomous vehicle win Constrained by its own hardware limitations and the surrounding environment, the following three constraints should be considered:

[0048] (1) Limiting velocity and angular velocity constraints:

[0049] V lim ={(v,w)|v∈[v min ,v max ]^w∈[w min ,w max ]}

[0050] (2) Acceleration-limited velocity and angular velocity constraints:

[0051]

[0052] (3) Speed ​​and angular velocity constraints for braking distance limitation:

[0053]

[0054] Above, v min v max For the limiting linear velocity, w min w max This is the limiting angular velocity. cu w cu The current linear velocity and angular velocity are... For the limit linear acceleration, Let be the limiting angular acceleration. `dist(v,w)` is the closest distance between the simulated trajectory corresponding to the velocity-angular velocity combination (v,w) and the obstacle. Finally, at time `t`, V... win Represented as:

[0055] V win =V lim ∩V acc ∩V dis

[0056] Specifically, such as Figure 3 As shown, the evaluation function includes three sub-functions, which comprehensively consider three factors: the autonomous vehicle's speed, obstacle collision risk, and autonomous vehicle heading, as detailed below:

[0057] G(v,w)=σ(αheading(v,w)+ηdist(v,w)+γvel(v,w))

[0058] in The yaw angle of the autonomous vehicle is represented by δ, which is the angle between the line connecting the autonomous vehicle and the target point and the positive x-axis. Figure 4 As shown.

[0059] `dist(v,w)` represents the Euclidean distance from the simulated trajectory to the nearest obstacle, `vel(v,w)` represents the linear velocity of the autonomous vehicle, and `α`, `η`, and `γ` are three weighting coefficients. As shown above, the evaluation function is composed of sub-functions with different dimensions. The normalization function `σ()` in the formula is equivalent to dimensionless learning, which can unify data with different dimensions under the same reference frame for combination or comparison, thereby avoiding evaluation bias caused by different data scales. Specifically:

[0060]

[0061] dist(v i ,w j ) and vel(v i ,w j Perform the same normalization operation.

[0062] The unmanned vehicle obtains its simulated trajectory based on a uniform motion model. Under the assumptions of this model, the magnitudes of the vehicle's linear velocity and angular velocity remain constant, and the change in the direction of the linear velocity is linearly related to time. To simplify the model and speed up computation, the velocity direction can be considered constant within small time intervals. Therefore, the uniform motion model can be discretized. t y t This represents the horizontal and vertical coordinates of the intelligent vehicle at time t. v represents the heading angle at time t. t w t Let t represent the velocity and angular velocity at time t, as shown in the following formula.

[0063]

[0064] Step 3: Establish a proximal policy optimization learning (PPO) model based on the actor-critic framework (e.g., Figure 5 As shown, a real-world application scenario of the driverless car is simulated as the learning environment for the model, and the state and actions in the model are determined based on the application scenario.

[0065] The Proximal Policy Optimization (PPO) algorithm works by optimizing the objective function. Add D to the middle KL The specific penalty items (p||q) are as follows:

[0066]

[0067] In the formula To obtain the objective function for importance-based policy learning by integrating over the parameter θ1, where θ is the policy parameter π, the better the policy, the better the objective function. The larger γ is, the more it is introduced as a parameter for the Monte Carlo approximation, U tFor the parameters in the policy gradient, π(a) t |s t ;θ1) is the target strategy, π(a t |s t ;θ2) represents the behavioral strategy, Let q be the mathematical expectation of the policy network, and β be a hyperparameter. The larger the difference between distributions q and p, the greater the expected value of D. KL The larger the (p||q) term, the better. The greater the punishment, the better; conversely, the less punishment, the better. KL The smaller the (p||q) term, the better. The smaller the penalty, the greater the goal of reinforcement learning. Therefore, it has a penalty clause. It can control behavior to remain within a certain range of similarity to the target strategy.

[0068] Autonomous vehicles seek the optimal path connecting the starting point and the destination in obstacle-prone environments. Therefore, the actual application scenario of autonomous vehicles serves as the learning environment for the model, which is the obstacle map.

[0069] In the model, the state 's' represents the environmental information perceived by the autonomous vehicle using sensors, and may also include its own position and motion state information. State 's' is the sole source of information for the autonomous vehicle's action decisions and a crucial basis for maximizing rewards; therefore, the quality of state 's' directly affects whether the algorithm converges, its convergence speed, and its final performance. State 's' can be understood as a high-dimensional vector of surrounding environmental information. The ultimate goal of the autonomous vehicle is to reach its destination via the optimal path; clearly, the vehicle's position and state, the distribution of surrounding obstacles, and the target point's location information will be the core basis for the autonomous vehicle's action decisions. To better suit practical application scenarios, the information reflected back from a 2-degree scan interval by the LiDAR can be used as the main part of state 's'. Furthermore, state 's' should also include the autonomous vehicle's speed 'v'. t angular velocity w t Heading angle and the current target point location information (x g t ,y g t ),like Figure 6 As shown. The specific method involves using the output of the policy network to replace the fixed weights of the evaluation function, constructing an adaptive evaluation function. Clearly, action a corresponds to the weights (α, η, γ) in the evaluation function; therefore, action a is defined as:

[0070] a=[μ1,σ1,μ2,σ2,μ3,σ3]

[0071] Where [μ1,σ1] are the mean and variance, used to describe the probability density function of the weight α:

[0072]

[0073] Similarly, [μ2,σ2] represents the mean and variance, used to describe the probability density function of weight η, and [μ3,σ3] represents the mean and variance, used to describe the probability density function of weight γ. Then, (α,η,γ) are randomly sampled according to their respective probability density functions, and the actions are mapped to the interval [-1,1] using the Tanh function.

[0074] Once the state s and action a are determined, the number of neurons in the input and output layers of the policy network π(a|s; θ) and the value network q(s, a; w) are also determined. A schematic diagram of the policy network and value network structure is shown below. Figure 7 , 8 As shown:

[0075] Step 4: Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main reward and the sub-objective reward; and determine the model parameters, including the size of the input layer, the output layer, the number of hidden layers and the number of neurons, to complete the instantiation of the DWA-PPO deep reinforcement learning model.

[0076] The reward function is the core component of the learning model. The reward obtained by the autonomous vehicle can be divided into main objective reward and sub-objective reward, depending on whether it triggers the main objective event.

[0077] Mainline Reward: The mainline reward can be understood as the settlement reward for the agent reaching the termination state. In this article, it is the reward R obtained by the autonomous vehicle navigating to the destination. mian goal Penalty and reward R when exceeding the maximum number of iterations mian out And the penalty and reward R when the autonomous vehicle collides with an obstacle. mian coll .

[0078] Sub-objective rewards: Rewards outside the main reward are called auxiliary rewards, and their main form is sub-objective rewards. Based on the practical application scenario of autonomous vehicle navigation planning in obstacle-prone environments, this paper analyzes the impact of factors such as local key points, environmental conditions, autonomous vehicle motion state, and the relative relationship between the autonomous vehicle and the target point on the main task of finding the optimal path for the autonomous vehicle, and proposes the following sub-objective rewards:

[0079] (1) Energy penalty reward R sub step :R sub step The existence of [something] can limit the energy consumption of autonomous vehicles while also helping them find the optimal path; E t For the t-th step, the autonomous vehicle travels at a speed v t Driving Δ tThe energy consumed in this process, after normalization, is defined as R. sub step for:

[0080]

[0081] (2) Distance change reward R sub dis During this process, the autonomous vehicle may temporarily move away from the finish line while avoiding obstacles, but globally it will always move closer to the finish line. Therefore, a reward, R, can be defined that is related to the distance between the autonomous vehicle's position and the target point. sub dis It should be a positive reward, and the greater the distance traveled towards the finish line, the higher R becomes. sub dis The larger.

[0082] (3) Obstacle distance bonus R sub obs :r t obs Defined as follows: when there are no obstacles within the safe distance of the autonomous vehicle and the autonomous vehicle brakes at maximum deceleration, avoiding collisions during the planning process is the primary prerequisite for ensuring driving safety. After normalization, R is defined as follows: sub obs for:

[0083]

[0084] (4) Azimuth bonus R sub head The goal of an autonomous vehicle is to reach its destination; therefore, in navigation, the closer it is to the target point, the better its heading angle. head A positive reward is defined as the autonomous vehicle receiving a reward only when its heading is very close to the optimal azimuth angle. After normalization, R is defined as... sub head for:

[0085]

[0086] In summary, the reward R of the autonomous vehicle at step t is... t The following formula is given: This is a sub-goal reward adjustment factor.

[0087]

[0088] The AC framework constructs a value network to approximate the action value in the policy gradient; therefore, the network architecture includes at least a value network and a policy network. Based on the value network loss function:

[0089]

[0090] The learning objectives of value networks are:

[0091]

[0092] It can be seen that its learning objectives include a portion of its own predictions. If the value network itself overestimates the action value Q(s,a), then the way the value network learns from itself will amplify this overestimation problem, and this overestimation is non-uniform, severely affecting network training. This phenomenon is called bootstrapping. To prevent bootstrapping in the value network, w is used... - Building a target value network The parameter structure of this network is the same as that of the value network, but the specific values ​​are different. It is used to calculate the TD error.

[0093]

[0094] The initial parameters of the target value network are consistent with those of the value network, where μ is a parameter, ensuring that the sum of the coefficients is 1. Subsequent updates will refer to the following formula:

[0095]

[0096] In summary, the network architecture of the DWA-PPO reinforcement learning model consists of three main parts: the policy network π(a|s;θ), the value network q(s,a;w), and the target value network q. T (s,a;w). The DWA-PPO reinforcement learning model is as follows: Figure 9 As shown.

[0097] In summary, the model construction includes an agent, an environment, a critic module, and an actor module. The critic module includes the value network error function L(w), the value network q(s,a;w), and the target value network q. T (s,a;w - The actor module includes the target network π(a|s; θ1), the behavior network π(a|s; θ2), and the policy network objective function. The initial training phase involves collecting the training set, as shown by the black line segment in the figure. At the initial moment of round 0, the autonomous vehicle observes state s0 from the environment using its perception and localization system. The behavior network π(a|s; θ2) receives s0 and outputs a Gaussian distribution π(A0|s0; θ1) about action A0. Then, it randomly selects action a0 from this probability distribution and transmits it to the autonomous vehicle, obtaining the evaluation function G0(v,w) of the DWA algorithm at the initial moment. This completes the evaluation of the simulated trajectory set of the DWA algorithm at the initial moment, and transmits the velocity and angular velocity commands of the optimal trajectory to the autonomous vehicle's motion control module to drive the vehicle's movement. At this point, the autonomous vehicle's position, orientation angle, and the distribution of surrounding obstacles change, and the environment transitions to state s1. The reward function also feeds back the changed information to the critic module, rewarding r0. When s1 is not the final state s... n If the current round proceeds to the next time step, otherwise the map and the autonomous vehicle's state are reset, and trajectory collection begins for the next round, continuing until all i rounds have been collected, ultimately yielding the training set.

[0098] χ=[χ0,χ1,…,χ i ]

[0099] χ0=[s0 0 ,a0 0 ,r0 0 ,…,s n-1 0 ,a n-1 0 ,r n-1 0 ,s n 0 ]

[0100] Step 5: Construct the PPO-ADWA algorithm. Using the established DWA-PPO deep reinforcement learning model, simulate the navigation planning of the autonomous vehicle in a randomly generated complex static obstacle environment to collect a training set for training the network model. Through repeated iterations, converge the model parameters that can output the corresponding weight parameters according to the changes in the distribution of surrounding obstacles, and complete the construction of the adaptive PPO-ADWA algorithm.

[0101] After obtaining the training set, the value network q(s,a;w) is updated using backpropagation with the error function L(w); the error function of the PPO algorithm is then used. Backpropagation updates π(a|s; θ1). Let the network q(s,a;w), q T (s,a;w - The current network parameters are w, π(a|s; θ1). now , θ now Repeat the following steps Z times to complete a generation update:

[0102] (1) Randomly select M from the shuffled training set χ I (Minimum batch size) states s N I .

[0103] (2) Use q T (s,a;w - Calculate state s N I K-step TD error MTD starting from N I :

[0104]

[0105] (3) Calculate the state s using the value network q(s,a;w). N I Estimation of action value at that time:

[0106] q N I =Q(s) N I ,a N I ;w now )

[0107] (4) Calculate L(w):

[0108]

[0109] (5) Calculation

[0110]

[0111] (6) Update the value network, strategy network, and target value network:

[0112]

[0113]

[0114]

[0115] Assume the parameter before the update is θ now The parameter θ is obtained after importance sampling and updating. new Assuming the parameter before the update is w now The parameter w is obtained after strategy learning and updating. new Assume that the parameter w was introduced before the update to prevent bootstrapping in the value network. - now After updating w with μ as the parameter to ensure the sum of the coefficients is 1, we obtain w. -new After completing Z updates, the parameters of the target network π(a|s; θ1) are assigned to the target network π(a|s; θ1), which is called the first generation update. Then the training set is cleared and the next generation update is started again until the model converges.

[0116] Figure 10 The graph shows the changes in the average score and arrival rate of the autonomous vehicle in each generation of the deep reinforcement learning environment during network training. As the model iterates and converges, the network model gradually learns a parameter network that can correctly guide the autonomous vehicle's path planning. The adaptive PPO-ADWA algorithm is then constructed.

[0117] Step 6: Demonstrate the adaptive adjustment capability of PPO-ADWA-based unmanned vehicle path planning through simulation comparison experiments;

[0118] To verify the self-adjusting capability of the autonomous vehicle path planning based on the PPO-ADWA algorithm, this section will verify its robustness in a randomly generated complex static obstacle environment. The simulation environment is as follows: Figure 11 As shown, the map size is 60m×60m, with green dots as the starting point and blue pentagrams as the ending point. Black geometric shapes represent obstacles, including regular polygons and circles. The size and number of obstacles are randomly generated within a certain range. The simulation results for 100 maps with different obstacle positions are shown in Table 1.

[0119] Table 1 Comparison of Simulation Results

[0120]

[0121] The arrival rate of the autonomous vehicle path planning results under PPO-ADWA was 84%, an improvement of 6 percentage points compared to the arrival rate under classic DWA; the average path length was 93.04m, and the path efficiency was improved by 5.00%; the average number of steps was 251.95, and the average cost per step was reduced by 4.85%. The results for classic DWA planning can be found here. Figure 12 The PPO-ADWA planning results are shown below. Figure 13 The curves showing the changes in weight parameters during the autonomous vehicle path planning process using the PPO-ADWA fusion strategy are as follows: Figure 14 As shown, it can be seen that the weight parameters generally maintain a numerical relationship of η>γ>α.

[0122] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. An adaptive path planning method for unmanned vehicles based on dynamic windowing and proximity strategy, characterized in that, Includes the following steps: Step 1: Construct an agent-environment interaction model, with the autonomous vehicle acting as the agent in deep reinforcement learning and the obstacle map as the environment; Step 2: Establish the DWA algorithm model, and determine the parameters including: speed range, angular velocity range, acceleration range, angular acceleration range, as well as the elements and evaluation functions of the DWA algorithm based on the Ackerman intelligent vehicle. Step 3: Establish a proximal policy optimization learning (PPO) model based on the actor-critic framework, simulate the actual application scenario of the autonomous vehicle as the learning environment of the model, and determine the state and action in the model according to the application scenario. Step 4: Construct the DWA-PPO deep reinforcement learning model, define the reward function including the main reward and the sub-objective reward; and determine the parameters of the DWA-PPO deep reinforcement learning model, including the size of the input layer, the output layer, the number of hidden layers and the number of neurons, and complete the instantiation of the DWA-PPO deep reinforcement learning model. Step 5: Construct the adaptive PPO-ADWA algorithm. Using the established DWA-PPO deep reinforcement learning model, simulate the navigation planning of the autonomous vehicle in a randomly generated complex static obstacle environment to collect a training set for training the DWA-PPO deep reinforcement learning model. Through repeated iterations, converge a model that can output corresponding weight parameters according to changes in the distribution of surrounding obstacles, thus completing the construction of the adaptive PPO-ADWA algorithm. The specific implementation is as follows: After obtaining the training set, use the error function Backpropagation updates the value network Let s, a, and w represent the state, action, and angular velocity, respectively; the error function of the proximal policy optimization algorithm is used. Backpropagation update , , For strategy Parameters; Define the value network Target Value Network Target network The current network parameters are as follows: , , Repeat the following steps Z times to complete a generation update: (1) Randomly select the smallest batch size from the shuffled training set χ. each state ; (2) Use Calculate the state K-step TD error starting from : (3) Using value networks Calculate the state Estimation of action value at that time: (4) Calculation : (5) Calculation : (6) Update the value network, strategy network, and target value network: Assuming the parameter before the update is The parameters are obtained after importance sampling and updating. Assuming the parameter before the update is The parameters are obtained after strategy learning and updating. Assuming that parameters were introduced before the update to prevent bootstrapping in the value network. ,go through As parameters, ensure the sum of coefficients is 1 after updating to obtain After completing Z updates, the target network will be... Parameters assigned to the target network This is recorded as one generation update. After that, the training set is cleared and the next generation update is started again until the model converges. The curves showing the changes in the average score and arrival rate of the autonomous vehicle in each generation in the deep reinforcement learning environment. As the model iterates and converges, the network model gradually learns the parameter network that can correctly guide the autonomous vehicle's path planning; the construction of the adaptive PPO-ADWA algorithm is completed. Step 6: Demonstrate the adaptive adjustment capability of unmanned vehicle path planning based on the adaptive PPO-ADWA algorithm through simulation comparison experiments.

2. The adaptive path planning method for unmanned vehicles based on dynamic window method and proximity strategy according to claim 1, characterized in that, Step one is implemented as follows: The intelligent agent is responsible for outputting action information and receiving rewards and states. The environment is the interactive object of the intelligent agent, and its interaction process includes the following three steps: (1) The agent is influenced by the environmental state Observed information , The state space is the set of possible values ​​for the environment states. The observation space is the set of values ​​for the agent's observations. (2) The agent is composed of known... Make appropriate decisions and decide on the actions to be taken in response to the environment. , It is the set of action values; (3) Environmental factors Impact, one's own state Transfer to And reward the intelligent agent. , It is the set of possible reward values; therefore, the discretized agent-environment interaction model is represented by the following sequence: When the state of the environment can be fully observed by the agent, then there is To simplify: 。 3. The adaptive path planning method for unmanned vehicles based on dynamic window method and proximity strategy according to claim 2, characterized in that, Step two is implemented as follows: The Direct Path Analysis (DWA) algorithm is a local path planning method that provides an intuitive understanding of the map environment in which an autonomous vehicle is located from a velocity space perspective. Its workflow is as follows: considering the constraints of various conditions on velocity and angular velocity at time t, it determines the velocity and angular velocity window that the autonomous vehicle can reach at time t. ; Discretize the velocity and angular velocity, and combine the discretized values; the autonomous vehicle traverses all combinations and simulates moving forward m times according to the given motion model. The duration is used to obtain the simulated trajectory set τ, which is a series of point sets; the evaluation function gives the score of all simulated trajectories in the simulated trajectory set τ, and the trajectory with the highest score is selected. The corresponding combination; the duration of time the autonomous vehicle is driven by this combination. The sampling process continues until time t+1 is reached; this cycle repeats until the endpoint is reached, where m is the number of sampling steps. The sampling interval; At time t, the velocity angular velocity window of the autonomous vehicle Constrained by its own hardware limitations and the surrounding environment, consider the following three constraints: (1) Limiting angular velocity constraints: (2) Acceleration-limited velocity and angular velocity constraints: (3) Speed ​​and angular velocity constraints for braking distance limitation: above, , For the limiting linear velocity, , The limiting angular velocity, , The current linear velocity and angular velocity are... , For the limit linear acceleration, , This is the limiting angular acceleration. For the combination of velocity and angular velocity The closest distance between the corresponding simulated trajectory and the obstacle; the velocity angular velocity window of the autonomous vehicle at time t. Represented as: The evaluation function consists of three sub-functions, which comprehensively consider three factors: the autonomous vehicle's speed, obstacle collision risk, and autonomous vehicle heading, as detailed below: in , Indicates the heading angle of the autonomous vehicle. The angle between the line connecting the autonomous vehicle and the target point and the positive x-axis; To simulate the Euclidean distance from the trajectory to the nearest obstacle, This indicates the magnitude of the linear velocity of the autonomous vehicle. , , There are three weighting coefficients; as shown above, the evaluation function is composed of sub-functions with different dimensions, and the normalization function in the formula... This is equivalent to dimensionless learning, which can unify data with different dimensions under the same reference frame for combination or comparison, avoiding evaluation bias caused by different data scales, as detailed below: and Perform the same normalization operation; The unmanned vehicle obtains its simulated trajectory based on a uniform motion model. Under the assumptions of this model, the magnitudes of the vehicle's linear velocity and angular velocity remain constant, and the change in the direction of the linear velocity is linearly related to time. To simplify the model and speed up computation, the velocity direction can be considered constant within small time intervals. Therefore, the uniform motion model is discretized. t y t This represents the horizontal and vertical coordinates of the intelligent vehicle at time t. v represents the heading angle at time t. t w t The velocity and angular velocity at time t are represented by the following formula: 。 4. The adaptive path planning method for unmanned vehicles based on dynamic window method and proximity strategy according to claim 3, characterized in that, Step three is implemented as follows: The approach of the near-end policy optimization algorithm is to optimize the objective function. Add to The specific penalties are as follows: In the formula For parameters The integral yields the objective function for policy learning based on importance sampling. For strategy The parameters, the better the strategy, the better the objective function. The larger, The parameters introduced for the Monte Carlo approximation, These are the parameters in the policy gradient. For the target strategy, For behavioral strategies, For the mathematical expectation of the policy network, As a hyperparameter, the greater the difference between distributions q and p, the better. The larger the item, The greater the punishment, the better. The smaller the item, The smaller the penalty, the greater the goal of reinforcement learning. Therefore, it has a penalty term. It can control behavior and target strategies within a predetermined similarity range; Autonomous vehicles search for the optimal path connecting the starting point and the destination in an obstacle environment. Therefore, the actual application scenario of the autonomous vehicle serves as the learning environment for the model, which is the obstacle map. In the model, state s represents the environmental information perceived by the autonomous vehicle using sensors, including its own position and motion status. The information reflected back from a complete scan by the LiDAR at 2-degree intervals is also included in state s. Additionally, state s also includes the autonomous vehicle's speed. angular velocity Heading angle and the current target point location information The specific method involves using the output of a policy network to replace the fixed weights of the evaluation function, constructing an adaptive evaluation function. Clearly, the action... Weights in the evaluation function Correspondingly, the action is defined. for: in The mean and variance are used to describe the weights. Similarly, the probability density function is: The mean and variance are used to describe the weights. The probability density function, The mean and variance are used to describe the weights. The probability density function is then used to determine the probability density function; subsequently, random sampling is performed according to the respective probability density functions. The action is then mapped to the interval [-1, 1] using the Tanh function; state With action Once determined, the number of neurons in the input and output layers of the policy network and value network are also determined.

5. The adaptive path planning method for unmanned vehicles based on dynamic window method and proximity strategy according to claim 4, characterized in that, Step four is implemented as follows: The reward function is a core component in learning the DWA-PPO deep reinforcement learning model. The reward obtained by the autonomous vehicle is divided into main objective reward and sub-objective reward, depending on whether it triggers the main objective event. Mainline Reward: The so-called mainline reward is the settlement reward for the agent reaching the termination state, that is, the reward obtained by the autonomous vehicle in navigating to the destination. Penalty and reward for exceeding the maximum number of iterations And the penalties and rewards when the autonomous vehicle collides with an obstacle. ; Sub-objective rewards: Rewards other than the main reward are called auxiliary rewards, which take the form of sub-objective rewards. Based on the practical application scenario of autonomous vehicle navigation and planning in obstacle-prone environments, this paper analyzes the impact of factors such as local key points, environmental conditions, autonomous vehicle motion state, and the relative relationship between the autonomous vehicle and the target point on the main task of finding the optimal path, and gives the following sub-objective rewards: (1) Energy penalty reward : The existence of this can limit the energy consumption of autonomous vehicles, while also helping them find the optimal path. For the t-th step, the autonomous vehicle has a speed of driving The energy consumed in this process, after normalization, is defined as follows: for: (2) Distance change reward Define a reward that is related to the distance between the autonomous vehicle's location and the target point. It should be a positive reward, and the greater the distance traveled towards the finish line, the better. The larger; (3) Obstacle distance reward : Defined as follows: when there are no obstacles within the safe distance of the autonomous vehicle and the autonomous vehicle brakes at maximum deceleration, avoiding collisions during the planning process is the primary prerequisite for ensuring driving safety. After normalization, the definition is... for: (4) Azimuth bonus The goal of an autonomous vehicle is to reach its destination, so navigation systems consider the heading angle of the vehicle to be better the closer it is to the target point. A positive reward is defined as the autonomous vehicle receiving a reward only when its heading is very close to the optimal azimuth angle. After normalization, the definition is... for: The reward for the autonomous vehicle at step t The following formula is given: This serves as a sub-target reward adjustment factor. The network architecture of the DWA-PPO reinforcement learning model includes at least a value network and a policy network; based on the value network loss function: The learning objective of value networks is: The learning objective of a value network includes a portion of its own prediction. To prevent bootstrapping in the value network, use Build a target value network The parameter structure of this target value network is the same as that of the value network, but the specific values ​​are different. It is used to calculate the TD error. The initial parameters of the target value network are consistent with those of the value network. For parameters, ensure the sum of coefficients is 1. Subsequent updates will refer to the following formula: The network architecture of the DWA-PPO reinforcement learning model consists of three main parts: the policy network. Value Network and target value network ; In summary, the DWA-PPO reinforcement learning model comprises agent, environment, critic module, and actor module; the critic module includes the value network error function. Value Network Target Value Network The actor module includes the target network. Behavioral networks Policy network objective function ; The initial phase of training involves collecting the training set: at the beginning of round 0, the autonomous vehicle uses its perception and localization system to observe the state from the environment. Behavioral networks take over Then output a message about the action. Gaussian distribution Then, actions are randomly selected from this probability distribution. The data is transmitted to the intelligent vehicle to obtain the evaluation function of the DWA algorithm at the initial time step. The system evaluates the simulated trajectory set generated by the DWA algorithm at the initial moment and transmits the velocity and angular velocity commands of the optimal trajectory to the autonomous vehicle's motion control module, driving the autonomous vehicle's movement. At this point, information including the autonomous vehicle's position, orientation angle, and the distribution of surrounding obstacles has changed, and the environment transitions to a new state. The reward function will also provide feedback to the commentator module based on the changed information. ;when Not in a terminated state If the current round proceeds to the next time step, otherwise the map and the autonomous vehicle's state are reset, and trajectory collection begins for the next round, continuing until all i rounds have been collected, ultimately yielding the training set. 。

Images