Efficient progressive strategy enhancement framework used for path planning and based on deep reinforcement learning

Abstract

The invention discloses an efficient progressive strategy enhancement (EPPE) framework based on deep reinforcement learning (DRL) and used for a path planning scene, and the framework assists a DRL algorithm and is used for solving dual challenges of training efficiency and combinatorial optimization faced by DRL in path planning. The framework combines the advantages of sparse rewards and process rewards, the sparse rewards help the intelligent agent to realize a global optimal strategy, and the process rewards provide real-time feedback, so that the intelligent agent can adjust the strategy in time. The strategy learning efficiency is remarkably improved, and the reward coupling problem caused by process rewards is effectively solved, so that the global optimization of the strategy is realized. The method can be applied to the research field of robot path planning, and plays an important role in the application fields of automatic driving, logistics distribution and the like.

Description

Technical Field

[0001] The present invention belongs to the field of path planning based on deep reinforcement learning, and specifically relates to an efficient progressive strategy enhancement framework based on deep reinforcement learning for the field of path planning. Background Art

[0002] In recent years, it has been found that the method based on deep reinforcement learning has shown great advantages and development potential in the path planning research of drones. However, there are two main problems in the study of path planning algorithms based on deep reinforcement learning:

[0003] (1) It is necessary to ensure that the agent maintains high learning efficiency in complex sequential decision-making environments;

[0004] (2) The global optimality of the path quality must be ensured, that is, the time it takes for the agent to reach the target and the distance it travels on the way to the target must be optimized.

[0005] Therefore, it is crucial to design a framework to improve the training efficiency of DRL methods while optimizing the combined metrics in path planning. Summary of the invention

[0006] The problem solved by this invention is to provide a method that combines the advantages of the two previously mentioned reward types: sparse rewards and process rewards. Sparse rewards are conducive to comprehensive optimization and have significant global convergence properties. Process rewards provide real-time feedback during the policy learning process and improve training efficiency. In addition, our work aims to achieve two goals: improve the efficiency of the DRL learning process while ensuring the optimization of the path.

[0007] The present invention is realized by the following technical solutions: To realize this vision, the present invention introduces an efficient progressive policy enhancement (EPPE) framework, which mainly consists of three models. First, the reward enhancement mechanism: We adopt the DWA (dynamic window method) algorithm and build a corresponding process reward mechanism based on it. This mechanism provides real-time feedback through process rewards as an enhancement mechanism for the entire reward system. As a process reward, this mechanism provides real-time feedback to the agent during the learning process, thereby promoting the immediate adjustment of the strategy. As a result, the training efficiency of traditional DRL is accelerated. Secondly, the incremental reward adjustment (IRA) model is designed: the reward enhancement mechanism can quickly help us obtain a successful strategy in the initial stage, but the resulting reward coupling effect may affect the quality and stability of the agent's subsequent strategy learning. After obtaining the initial strategy, the model will gradually increase the reward weights related to comprehensive optimization to promote the agent's learning to the global optimal strategy. Finally, the fine-tuning policy optimization (FPO) model is constructed: after successfully obtaining the initial strategy through the reward enhancement mechanism, the FPO model supports the IRA model by gradually reducing the initial and final learning rates at each stage. This model fine-tunes the strategy based on the previously learned strategy, which not only ensures the improvement of the previous strategy, but also prevents the strategy deviation caused by high learning rate. In addition, the adoption of learning rate decay strategy at specific stages helps the agent transition from early improvement of strategy quality to later stability improvement. The specific steps of the framework are:

[0008] 1. An efficient progressive policy enhancement framework based on deep reinforcement learning for path planning, the specific steps are:

[0009] Step 1: Construction of state space: From a modeling perspective, in the context of path planning problems, the state space can be divided into two parts:

[0010] S=[S 0 ,S 1 ]

[0011] S 1 = {O i |i=0,1,...n-1}

[0012] Where S 0 Represents the state information of the agent itself, S 1 Represents the status information of the obstacle.

[0013] Step 2: Construction of action space: In the framework of deep reinforcement learning, an action refers to the decision made by the agent in a specific state. This behavior is selected according to a certain strategy π, aiming to maximize the long-term cumulative reward.

[0014] Step 3: Construction of reward function: The reward function defines the learning goal and serves as a driving force in the learning process, guiding the DRL algorithm to distinguish between effective and ineffective behavior strategies.

[0015] Step 4: Setting of process rewards: We combine the evaluation function in the dynamic window method (DWA) and establish a reward enhancement mechanism that provides real-time feedback during complex path planning, enabling the agent to adjust its strategy in real time, thereby improving the learning efficiency of the DRL-based agent.

[0016] Step 5: Incremental reward adjustment model: This model adjusts the weight between sparse rewards and process rewards, that is, adjusts the balance between guidance rewards and rewards related to the combined optimization objective.

[0017] Step 6: Fine-tune the policy optimization model: In the process of deep reinforcement learning (DRL), the agent constantly balances between consolidating existing knowledge and quickly absorbing new information, including multiple training stages. In the early stages of the IRA model, the model efficiently helps to form an effective baseline policy; then, in the later stages of the model, the policy is gradually optimized based on the existing policy by gradually increasing the reward weight related to the combined optimization objective.

[0018] Compared with the prior art, the present invention has the following beneficial technical effects:

[0019] The present invention can be applied to obtain the optimal path in path planning tasks in complex environments. The framework-assisted deep reinforcement learning baseline algorithm studied can help the intelligent agent to learn efficiently in an unfamiliar environment, while making the strategy obtained by the intelligent agent theoretically globally optimal in terms of path quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the whole thought process of the present invention;

[0021] Figure 2 It is a simulation diagram comparing the training of the present invention in a static obstacle environment and a mixed (dynamic + static) obstacle environment;

[0022] Figure 3 It is a simulation diagram of the test path of the present invention in a static obstacle environment;

[0023] Figure 4 It is a simulation diagram of the test path of the present invention in a mixed (dynamic + static) obstacle environment;

[0024] Figure 5 It is the path quality test index of the present invention in a static obstacle environment;

[0025] Figure 6It is the path quality test index of the present invention in a mixed (dynamic + static) obstacle environment. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

[0028] This paper combines the advantages of the two types of rewards mentioned above: sparse rewards and process rewards. Sparse rewards are conducive to comprehensive optimization and have significant global convergence characteristics. Process rewards provide real-time feedback during the policy learning process and improve training efficiency. In addition, our work aims to achieve two goals: improve the efficiency of the DRL learning process while ensuring the optimization of the path. To achieve this vision, this paper introduces the Efficient Progressive Policy Enhancement (EPPE) framework, which mainly consists of three models. The first is the reward enhancement mechanism: we adopt the DWA (dynamic window method) algorithm and build a corresponding process reward mechanism based on it. This mechanism provides real-time feedback through process rewards as an enhancement mechanism for the entire reward system. As a process reward, this mechanism provides real-time feedback to the agent during the learning process, thereby promoting instant adjustment of the strategy. As a result, the training efficiency of traditional DRL is accelerated. Secondly, the incremental reward adjustment (IRA) model is designed: the reward enhancement mechanism can quickly help us obtain a successful strategy in the initial stage, but the resulting reward coupling effect may affect the quality and stability of the agent's subsequent strategy learning. After obtaining the initial policy, the model will gradually increase the reward weight associated with the comprehensive optimization to promote the agent's learning towards the global optimal policy. Finally, a fine-tuning policy optimization (FPO) model is constructed: after successfully obtaining the initial policy through the reward enhancement mechanism, the FPO model supports the IRA model by gradually reducing the initial and final learning rates at each stage. This model fine-tunes the strategy based on the previously learned policy, which not only ensures the improvement of the previous policy, but also prevents the policy deviation caused by high learning rates. In addition, the adoption of a learning rate decay strategy at specific stages helps the agent transition from early improvements in policy quality to later stability improvements.

[0029] Figure 1 The method used in this paper is presented, which combines the EPPE framework, the DRL method, and the path planning environment for learning. In the entire learning framework we proposed, first, in the DRL part, we collect state information from the environment, which is collected by the agent and compiled into a dynamic online dataset. Through the neural network, we calculate the mapping relationship between the state (state) S and the action (action) a, and continuously optimize the weights of these mappings during the training process. The optimization of the weight is based on the reward function R total, the optimal weight configuration corresponds to the maximum reward obtained. In addition, the state s, action a, and reward R are systematically stored in an experience replay mechanism to support online learning and policy optimization.

[0030] At the same time Figure 1 The EPPE framework in DRL consists of three parts: reward enhancement mechanism, IRA model and FPO model. The entire reward mechanism in DRL consists of two basic parts: basic reward, also known as sparse reward R DRL and the reward enhancement mechanism R enhance , the latter is a process reward mechanism. In addition, the weights of these two rewards are determined by the parameter a i and b i control.

[0031] The IRA and FPO models proposed at the same time are as follows Figure 1 As shown in the figure, it contains many small modules, which are the other parts of the EPPE framework. Each submodule is designed to work together. Their collaborative operation is achieved by gradually adjusting b i The value of is used to gradually optimize the strategy. In the entire FPO module, the learning rate α in a single stage E Affected by α start , α end and the number of training sets E, and changes with the number of training sets. In addition, α at different stages start and α end The value decreases gradually. Figure 1 The thick red arrow in the figure depicts the evolution direction of the entire framework, emphasizing the process of gradual development. The three parts of the EPPE framework can help improve the training efficiency of the DRL algorithm in path planning scenarios and its performance in comprehensive optimization challenges.

[0032] The framework described herein comprises the following specific steps:

[0033] 1. An efficient progressive policy enhancement framework based on deep reinforcement learning for path planning, the specific steps are:

[0034] Step 1: Construction of state space:

[0035] From a modeling perspective, in the context of path planning problems, the state space can be divided into two parts:

[0036] S=[S 0 ,S 1 ]

[0037] S 1 = {O i |i=0,1,...n-1}

[0038] Where S 0 Represents the state information of the agent itself, S1 represents the state information of the obstacle. More specifically, S 1 is composed of each obstacle O i S is a set of state information, and n represents the number of obstacles. 0 and S 1 It can be specifically expressed as:

[0039] S 0 =p,v,g,r a ,T,s,d g ,d min(b) ,d min(o) ,θ,

[0040]

[0041] where p = [p x ,p y ] represents the position of the agent, represents the position of the i-th obstacle, represents the relative position of the ith obstacle relative to the agent, g = [g x ,g y ] represents the position of the target, v = [v x ,v y ] contains the magnitude and direction of the agent’s velocity, Contains the magnitude and direction of the velocity of the i-th obstacle, represents the relative speed of the ith obstacle relative to the agent, T is the end time of a round, s represents the total distance traveled by the agent in a round, and d min(b) is the minimum distance from the agent to the boundary, is the minimum distance from the ith obstacle to the agent, d min(o) is the minimum distance from the agent to all obstacles, d g is the distance from the robot to the target, r a represents the influence radius of the agent, r i represents the influence radius of the ith obstacle, and θ represents the angle between the straight line connecting the current position of the agent and the target and the positive direction of the horizontal x-axis.

[0042] Step 2: Construction of action space:

[0043] In the framework of deep reinforcement learning, an action refers to the decision made by an agent in a specific state. This behavior is selected according to a certain strategy π, aiming to maximize the long-term cumulative reward. In our work, an action is defined as a change in the speed of the agent, including a change in the magnitude of the speed Δv and a direction adjustment Δφ. The action affects not only the upcoming state of the agent, but also the immediate reward it may receive. The action is to change the state of the entire space based on the dynamic equation of the agent in a two-dimensional environment, thereby achieving state transfer. The dynamic equation can be expressed as:

[0044]

[0045] [p x(t+1) ,p y(t+1) ,v t+1 ,φ t+1 ] represents the basic state information at the next moment, [p x(t) ,p y(t) ,v t ,φ t ] represents the current state information. Other state information can be obtained based on the basic state information and action space.

[0046] Step 3: Construction of reward function:

[0047] In deep reinforcement learning, the design of the reward function is crucial. It not only defines the learning goal, but also serves as a driving force in the learning process, guiding the DRL algorithm to distinguish between effective and ineffective behavior strategies. DRL It consists of three parts: R DRL(1) ,R DRL(2) and R DRL(3) , the specific definitions of the three are as follows:

[0048] When the agent successfully enters the target area, that is, the distance between the agent and the target is less than the predetermined threshold d min(g) =5, a reward R related to combination optimization will be given DRL(1) :

[0049]

[0050] The parameter T max represents the maximum number of steps specified in the simulation settings, Δt represents the time step of each step, and v max Indicates the maximum speed that can be achieved during the simulation.

[0051] When the agent collides with an obstacle or boundary, d min(o) <r i +r a or min(b) <r a, then set the reward function R DRL(2) :

[0052] R DRL(2) =-300

[0053] In order to optimize the time it takes for the agent to reach the target, the agent will receive a negative reward before reaching the target at each step. DRL(3) The definition is as follows:

[0054] R DRL(3) =-1

[0055] Basic Reward R DRL The overall definition is as follows:

[0056] R DRL =f1·R DRL(1) +f2·R DRL(2) +f3·R DRL(3)

[0057] where f1, f2 and f3 are weight parameters. DRL Under the influence of , the reward reaches its maximum value only when the agent successfully reaches the target location and the time and travel distance are both minimized. This means that the maximum reward corresponds to the scenario where the strategy achieves global convergence.

[0058] Step 4: Setting up process rewards

[0059] Sparse reward functions often fail to provide immediate feedback, which prevents agents from effectively learning complex strategies, especially when computing resources are limited, often leading to the failure of the entire DRL training process. To address this problem, we combined the evaluation function in the dynamic window method (DWA) to establish a reward enhancement mechanism. This mechanism provides real-time feedback during complex path planning, enabling the agent to adjust its strategy in real time, thereby improving the learning efficiency of DRL-based agents. The setting of the reward enhancement mechanism involves three aspects:

[0060] R head : The greater the angular deviation between the agent's current position and the target position relative to the velocity vector, the smaller the reward. This setting is intended to encourage the agent to move ideally in a straight line toward the target. head The definition is as follows:

[0061] R head =π-|φ-θ|

[0062] R dist : The reward provided increases as the distance between the agent and the nearest obstacle increases, thus promoting safer path planning. dist The definition is as follows:

[0063]

[0064] where d safe Defined as the safety distance, it determines the range of punishment when the agent enters the safety range. We set d safe =d min(o) +5. κ is a positive number that determines the severity of the penalty per unit distance when the agent enters the safe range. In this work, we extend R dist When the agent behaves in a way that brings it closer to the goal than in the previous time step, we set up a mechanism based on positive reward feedback R' dist , which aims to motivate the agent to effectively reduce the distance to the target. At the same time, we define the overall and distance-related reward function R dist(total) :

[0065] R′ dist(t) =d g(t-1) -d g(t) ,

[0066] R dist(total) =R′ dist +R dist ,

[0067] where d g(t-1) is the distance from the agent to the target at the previous moment, d g(t) is the distance from the agent to the target at the current moment.

[0068] R velocity : Aims to encourage the agent to navigate to the goal faster while complying with the dynamic constraints of the environment. velocity The definition is as follows:

[0069]

[0070] where v max is the defined maximum speed.

[0071] Our reward enhancement mechanism R enhance The definition is as follows:

[0072] R enhance =k1·R head +k2·R dist(total) +k3·R velocity ,

[0073] The overall reward composition R total Defined as:

[0074]

[0075] Among them, k1, k2, k3, a and b are weight parameters. We define the operation between two real numbers m and n as follows:

[0076]

[0077] Step 5: Incremental Reward Adjustment Model

[0078] Two types of rewards R DRL and R enhance Represents the two reward mechanisms proposed in Section 1. DRL The independent role of helps the agent explore and achieve the global optimal strategy during the learning process; however, since it cannot provide continuous real-time feedback throughout the learning process, it limits the learning efficiency of the agent. enhance Ability to provide real-time feedback, allowing the agent to adjust its strategy in real time.

[0079] In the process of setting the guidance reward mechanism, there is a direct or indirect coupling between the state variables and the state variables of the target optimization. This coupling means that the learning results under this reward system cannot guarantee global optimality. In addition, it may also lead to poor stability of the final learning strategy. In addition, in the agent path planning, setting the target reward for minimizing travel time and path length too high will significantly reduce the impact of the guidance reward in the entire reward framework, thereby greatly reducing its contribution to improving training efficiency.

[0080] The guidance reward mechanism helps to improve learning efficiency, but it is not conducive to the combinatorial optimization process and may have an adverse effect on the realization of the global optimal solution. Although sparse rewards can ensure that learning proceeds in the direction of the global optimal strategy, their efficiency is low due to the inability to provide real-time feedback. In order to address the challenges of training efficiency and combinatorial optimization, we should combine the advantages of the two rewards at different stages of the learning process, and propose an incremental reward adjustment model. This model adjusts the weight between sparse rewards and process rewards, that is, it adjusts the balance between guidance rewards and rewards related to the combinatorial optimization objective. In the initial stage of training, it prioritizes guidance rewards to improve the learning efficiency of the agent. After establishing a baseline successful strategy, it gradually increases the reward weight related to the combinatorial optimization objective (through weights b1, b2,..., b n , where b1, b2, ..., b n The value of b gradually increases. n+1 The condition b is the same as the previous condition b n When the combinatorial optimization objective has almost no change compared to the original one, we terminate the optimization process. It should be noted that in our work, the termination condition S th and T th The choice is based on time accuracy and space accuracy. Specifically, the time accuracy Tth Determined by the step size, and the spatial accuracy S th is set according to the product of the average speed and the time step. Through this model, we can not only promote efficient learning, but also help the agent achieve the global optimal solution.

[0081] Step 6: Fine-tune the strategy optimization model

[0082] In the process of deep reinforcement learning (DRL), the agent constantly balances between consolidating existing knowledge and quickly absorbing new information, including multiple training phases. In the early stages of the IRA model, the model efficiently helps to form an effective baseline policy. Subsequently, in the later stages of the model, the policy is gradually optimized based on the existing policy by gradually increasing the reward weight associated with the combined optimization objective.

[0083] The learning rate α plays a vital role in the DRL algorithm for policy improvement. A higher learning rate promotes rapid policy adjustment and helps to quickly integrate new knowledge, while a lower learning rate helps to consolidate knowledge and stabilize the policy. Therefore, in the process of the IRA model, appropriately setting the learning rate α is crucial to ensure efficient and stable policy transition.

[0084] As the training phase progresses, it is necessary to gradually reduce the learning rate α from one phase to the next. This approach not only prevents the policy π from drifting, but also allows further optimization based on the achievements of the previous phase. In each specific training phase, the learning rate α should be increased from a higher initial value α to a lower value. start Gradually decrease to a lower final value α end In the early stages, the focus is on learning new knowledge to prevent overfitting, promote policy improvement, and improve adaptability to dynamic environments. In contrast, in the later stages, the priority is to improve the stability of the policy.

[0085] In a specific stage, the decay of the learning rate is carried out according to a predetermined principle. In our work, we choose the learning rate decay model. Here, the initial learning rate α start , the final learning rate α end , the decay coefficient k and the number of training rounds E determine the learning rate of different rounds in a specific stage. This principle ensures that the learning rate is adaptively adjusted throughout the training process to maximize efficiency. The learning rate decay model is expressed as follows:

[0086] α(E)=α end +(α start -α end ) -kE ,

[0087] As we progress through the different stages, α start and α endAll maintain a downward trend to meet the needs of the entire learning process. In our work, the parameter E is determined by the number of training rounds. The parameter k is the key coefficient that affects the attenuation of the learning rate.

[0088] Figure 2 (a) and Figure 2 (c) shows the training curves of the correct application of the EPPE framework under the process reward based on a larger weight in the first stage in a static obstacle environment and a mixed obstacle environment, respectively. Figure 2 (b) and Figure 2 (d) shows the training curves in the static obstacle environment and the mixed obstacle environment without the correct application of the EPPE framework in the first stage. It can be seen that the correct use of the EPPE framework significantly accelerates the training efficiency of DRL.

Claims

1. An efficient progressive policy enhancement framework based on deep reinforcement learning for path planning, characterized in that: The specific steps are: Step 1: Construction of state space: From a modeling perspective, in the context of path planning problems, the state space can be divided into two parts: S=[S 0 ,S 1 ] S 1 ={O i |i=0,1,...n-1} Where S 0 Represents the state information of the agent itself, S 1 Represents the status information of the obstacle; Step 2: Construction of action space: In the framework of deep reinforcement learning, an action refers to the decision made by the agent in a specific state. This behavior is selected according to a certain strategy π, aiming to maximize the long-term cumulative reward; Step 3: Construction of reward function: The reward function defines the learning goal and serves as a driving force in the learning process, guiding the DRL algorithm to distinguish between effective and ineffective behavior strategies; Step 4: Setting of process rewards: We combine the evaluation function in the dynamic window method (DWA) to establish a reward enhancement mechanism, which provides real-time feedback during the complex path planning process, enabling the agent to adjust its strategy in real time, thereby improving the learning efficiency of the DRL-based agent; Step 5: Incremental reward adjustment model: This model adjusts the weight between sparse rewards and process rewards, that is, adjusts the balance between guidance rewards and rewards related to the combined optimization goal; Step 6: Fine-tune the policy optimization model: In the process of deep reinforcement learning (DRL), the agent constantly balances between consolidating existing knowledge and quickly absorbing new information, including multiple training stages; in the early stages of the IRA model, the model efficiently helps form an effective baseline policy; Subsequently, in the later stages of the model, policy optimization is performed step by step based on the existing policy by gradually increasing the reward weights associated with the combined optimization objective.

2. The efficient progressive policy enhancement framework based on deep reinforcement learning for path planning according to claim 1, characterized in that: The process performed can be described as: First, a Markov decision model is established, that is, the basic state space, action space, and basic reward function are constructed; then a process reward function is constructed to speed up the training efficiency of the deep reinforcement learning method. Finally, an incremental reward adjustment model and a fine-tuning strategy optimization model are used to balance the training efficiency and combinatorial optimization, helping DRL to train the optimal strategy, which is both efficient training to obtain the optimal strategy and the global optimal combinatorial optimization result.

3. The efficient progressive policy enhancement framework based on deep reinforcement learning for path planning according to claim 1, characterized in that: In the process of training path planning tasks using deep reinforcement learning methods, efficient training is used to obtain the optimal strategy while obtaining the globally optimal combined optimization result.

Images