Incremental reinforcement learning path planning method based on dynamic reward remodeling

Abstract

The invention provides an incremental reinforcement learning path planning method based on dynamic reward remodeling, which is used for solving the problems of training efficiency and composite optimization in path planning. According to the method, a rapid exploration random tree and an artificial potential field method are combined with reinforcement learning, a dynamic reward function is designed, real-time feedback is provided for an intelligent agent, and strategy learning is accelerated. The system specifically comprises a reward remodeling module which provides process rewards through a fast exploration random tree and an artificial potential field method; the sub-target course learning module is used for gradually increasing task complexity and optimizing sample utilization; the progressive reward adjustment module dynamically adjusts the reward weight and solves the reward coupling problem; and the strategy fine tuning module dynamically adjusts the learning rate and supports strategy optimization. Experimental results show that in static and dynamic path planning tasks, compared with a traditional method, the method has the advantages that the training efficiency and the composite optimization performance are remarkably improved, the reward coupling problem is effectively solved, and the intelligent agent can efficiently learn and converge to a global optimal strategy.

Description

Technical Field

[0001] This invention belongs to the field of path planning and reinforcement learning technology, specifically relating to an incremental reinforcement learning path planning method based on dynamic reward reshaping. Background Technology

[0002] Path planning is of great significance in fields such as industrial automation, autonomous driving, and robotic surgery. Traditional path planning methods are mainly divided into two categories: global planning and local planning. Global planning methods, such as Rapid Random Tree Exploration (RRT) and the A* algorithm, rely on prior environmental information. When facing dynamic environments, they are prone to getting trapped in local optima and have slow convergence speeds. This is because they often plan paths based on pre-defined environmental models, making it difficult to adapt to changes in the environment in real time, resulting in an inability to quickly find the globally optimal path in complex dynamic scenarios.

[0003] Local planning methods, such as the Artificial Potential Field (APF) method, can respond to dynamic obstacles in real time, but they suffer from problems such as unreachable targets and path oscillations. For example, when the target point is within the "gravitational field" of the obstacle and there is a local minimum, the agent may get stuck in a local optimum and fail to reach the target point. Furthermore, when approaching the target point, the path may oscillate back and forth, affecting the stability and efficiency of path planning.

[0004] Existing hybrid methods that combine global and local planning, while integrating the advantages of both to some extent, suffer from performance limitations due to the accuracy of environmental modeling. In dynamic environments, the rapid changes in environmental characteristics make accurate modeling difficult, thus restricting the performance improvement of hybrid methods and making it difficult to meet the needs of path planning in highly dynamic scenarios.

[0005] Path planning methods based on deep reinforcement learning (DRL) have shown certain advantages by optimizing policies through reward functions. However, traditional sparse reward mechanisms lack real-time feedback, making it difficult for the agent to quickly adjust its policy during training, resulting in low training efficiency. For example, when the agent moves far from the target point, the sparse reward mechanism cannot provide effective guidance, causing it to act blindly during exploration and wasting a lot of training time.

[0006] Furthermore, while introducing process rewards provides real-time feedback, it introduces reward coupling effects. The interactions between multiple process rewards and with sparse rewards disrupt the monotonicity of the objective function, making it difficult for the agent to obtain the globally optimal policy. For example, process rewards based on distance to the target point and process rewards based on path smoothness may conflict in certain states, causing the agent to become confused when optimizing its policy and unable to determine the correct course of action.

[0007] Existing research still has shortcomings. For example, the dynamic constraint reward mechanism proposed by Wu et al., while improving the efficiency of multi-robot collaboration, did not solve the problem of verifying the global convergence of the reward function; the adaptive reward mechanism designed by Bai et al., while enhancing navigation capabilities in complex environments, failed to effectively suppress the impact of reward coupling on policy stability. Therefore, there is an urgent need for a path planning mechanism that can balance training efficiency and global optimality to address the problems existing in current technologies. Summary of the Invention

[0008] The purpose of this invention is to provide an incremental reinforcement learning path planning method based on dynamic reward reshaping to address the problems of low training efficiency and difficulty in composite optimization in path planning. Specifically, it aims to improve the learning efficiency of the agent in path planning tasks by designing innovative modules and mechanisms, enabling it to quickly converge to the globally optimal policy. Simultaneously, it effectively solves the reward coupling problem, ensuring the global optimality of the policy, thereby improving the overall performance of path planning in complex environments and meeting the practical application needs of various fields such as robot navigation, autonomous driving, and logistics transportation.

[0009] The incremental reinforcement learning path planning method based on dynamic reward reshaping proposed in this invention is essentially a fusion of the dynamic reward reshaping concept and a progressive learning strategy. This method integrates Rapid Exploratory Random Tree (RRT) and Artificial Potential Field (APF) into the reward framework of Deep Reinforcement Learning (DRL), constructing a reward shaping module to provide real-time feedback to the agent and accelerate policy learning. Simultaneously, a Sub-Objective Course Learning (SCL) module is set up to gradually increase task complexity, optimize sample utilization, and help the agent quickly acquire an initial policy. Furthermore, a Progressive Reward Adjustment (PRA) module is designed to dynamically adjust the weights of process rewards and sparse rewards, solving the reward coupling problem and guiding the policy from the initial feasible stage to the globally optimal stage; the Policy Fine-tuning (PFT) module dynamically adjusts the learning rate to support the PRA module, ensuring a smooth transition and overall stability of multi-stage policy learning. Through the coordinated operation of these modules, the efficiency of path planning training and the performance of composite optimization are improved, enabling the agent to learn efficiently in complex environments and find the globally optimal path.

[0010] Reward Shaping Module: This module innovatively integrates Rapid Exploration Random Tree (RRT) and Artificial Potential Field (APF) into a deep reinforcement learning reward framework, aiming to provide real-time feedback to the agent and thus improve training efficiency. In static obstacle environments, it fully utilizes the characteristics of the RRT algorithm to design procedural rewards, guiding the agent to optimize its path. In mixed obstacle environments, it combines RRT and APF algorithms to enhance the agent's obstacle avoidance and target approach capabilities.

[0011] In a static obstacle environment, the process reward designed based on the RRT algorithm is... and Composition. Among them,

[0012] This formula operates based on the intersection of the line connecting the agent to the target and obstacles. Among these, Indicates in The current position of the agent is connected to the target line and horizontal distance. The angle between the positive axis and the axis; Indicates in At time +1, the next position predicted or actually reached by the agent is connected to the target line and horizontal plane. The angle between the positive axis and the axis. When When the line connecting the agent to the target intersects with the obstacle, this indicates that the agent is in a position to move towards the target. As a weighting parameter, its value affects the degree of reward or penalty. This encourages the agent to adjust its trajectory to avoid collisions; when This means that the line connecting the agent and the target is unobstructed. This encourages agents to prioritize the shortest, unobstructed path towards their goal. The reward function is used to adjust the reward function in this case.

[0013]

[0014] This formula is used to encourage agents to adopt smoother speed transitions and trajectory adjustments, avoiding the planning of overly sharp paths. The agent represents the intelligent agent in the first The steps taken This indicates predefined action restrictions.

[0015] in, These are weight parameters that can be fine-tuned during training to optimize the overall reward effect, thereby guiding the agent to learn and plan paths more efficiently in static obstacle environments.

[0016] In dynamic obstacle environments, process rewards are designed by combining RRT and APF algorithms. ,Depend on and Composition. Among them,

[0017] This formula directly improves the existing obstacle avoidance reward function based on the distance between the agent and the obstacle. When the agent is in a certain state, a reward of -300 is given to strongly incentivize it to avoid obstacles; when... At that time, through This function further adjusts the reward based on distance to enhance obstacle avoidance; when When the distance between the agent and the obstacle is too far, the reward is 0, indicating that the agent is far from the obstacle and no special reward is needed. This formula is used to encourage the agent to approach the goal by comparing the distance between the agent and the goal at the current moment and at the previous moment. ,and ,in, , These are weight parameters used to balance the influence of different reward functions. By reasonably adjusting these two parameters, the agent can better balance obstacle avoidance and goal approach behavior in dynamic obstacle environments, thus achieving more efficient path planning.

[0018] Sub-goal Curriculum Learning (SCL) Module: The core of the SCL module lies in optimizing sample utilization and improving the agent's learning efficiency by setting sub-goals and gradually increasing task complexity. During implementation, this module dynamically adjusts task complexity. c and performance threshold First, the initial task complexity is... The terminal performance threshold is This determines the difficulty of the initial training environment and the performance standards the agent needs to achieve. Based on this, a DRL environment is constructed. Provide the agent with training scenarios of specific complexity. Then, in the environment... Use the settings The DRL model is trained to allow the agent to learn policies in this environment. After training, the trained model is evaluated using random test cases, and the success rate is calculated. This is used to determine the learning performance of the agent under the current task complexity.

[0019] like This indicates that the agent performs well under the current difficulty level. At this point, increasing the task complexity is appropriate. At the same time, tighten the terminal threshold, so that This gradually increases the difficulty of the task, prompting the agent to learn more complex strategies to adapt to more demanding tasks. and This means that the task difficulty and performance requirements have reached the set limit. At this point, the loop ends, and the required strategy is output. Through this dynamic adjustment process, the SCL module enables the agent to start with simple tasks in the early stages of training, quickly accumulate experience, and converge to a suitable strategy. As training progresses, it gradually adapts to more complex tasks, effectively avoiding the problem of getting stuck in local optima due to facing high-difficulty tasks too early, thereby improving the learning efficiency of the agent.

[0020] The Progressive Reward Adjustment (PRA) module, based on the concept of dynamic reward reshaping, aims to adjust the weights of process rewards and sparse rewards to resolve the reward coupling problem and ensure a smooth transition of the agent's policy from the initial stage of acquiring feasible policies to the stage of pursuing the globally optimal policy. In a static environment, the total reward function is... In mixed obstacle scenarios, the total reward function is: .in, As an optimization weight parameter, it plays a crucial role in the dynamic reward reshaping process. In the early stages of learning, it is assigned... Larger values ​​result in process rewards. or In the reward framework or The middle element plays a dominant role, providing real-time feedback to the agent and helping it quickly generate initial feasible policies in unfamiliar environments. As training progresses, the middle element gradually decreases. The value of sparse reward The weight of the variable increases, and it plays a dominant role in the later stages of training, guiding the strategy towards global optimization.

[0021] The specific algorithm steps of the PRA module are as follows: First, initialize the optimization weight parameters. Next, a total reward framework is constructed based on the formula in the corresponding environment. ; after that, based on Train the DRL model; then evaluate the trained model using 1000 random test cases and calculate the average index. and Then, the list of preset weight coefficients is... Each of them (from (Start), repeat the steps of building the reward framework, training the model, and evaluating the model in sequence; when and When the strategy optimization has reached a certain convergence level, the loop stops and the output is released. , Through this process, the PRA module effectively utilizes the advantage of providing real-time feedback through process rewards, improving efficiency in the early stages of policy learning. At the same time, it gradually reduces the weight of process rewards, mitigating the impact of reward coupling effects and ensuring that the final policy can achieve global optimum.

[0022] Policy Fine-tuning (PFT) Module: The PFT module uses a learning rate decay model to dynamically adjust the learning rate, thereby supporting the PRA module and ensuring a smooth transition and overall stability in multi-stage policy learning. The learning rate decay model used is... ,in This represents the number of training rounds, reflecting the learning progress of the agent during the training process; As a key factor affecting learning rate decay, and The size of this value determines the rate at which the learning rate decays. During different training phases, this module gradually reduces the initial learning rate. and final learning rate In the early stages of training, a relatively high initial learning rate is set, which allows the agent to quickly explore a wide range of states and actions. Because the agent has limited knowledge of the environment and task at this stage, a higher learning rate allows it to try different strategies more quickly, accelerates the learning of new strategies and environmental features, avoids getting stuck in local optima, and better adapts to dynamically changing reward frameworks.

[0023] As training progresses, the learning rate gradually decreases. Within each specific training phase, the learning rate... The learning rate is gradually reduced from a high initial value to a lower terminal value. In the later stages of training, a smaller learning rate allows the agent to fine-tune the learned policy. At this point, the agent has accumulated some experience and needs to fine-tune the policy to enhance its stability and convergence. This dynamic adjustment of the learning rate at different stages, combined with the dynamic adjustment of the reward framework in the PRA module, enables the agent to effectively learn and optimize the policy under different reward weight allocations. When the PRA module adjusts the reward weights, the PFT module adjusts the learning rate accordingly, ensuring the smoothness and efficiency of the overall policy learning process. This allows the agent to fully explore new policies and stably optimize existing policies at different stages, improving overall learning efficiency and robustness.

[0024] Problem Modeling and Related Definitions: This invention models the path planning problem of an intelligent agent as a Markov Decision Process (MDP), using tuples... Define it. Among them, the state space... The action space represents the set of all possible states of an agent in the environment, the action space is the set of all actions the agent can take, and the state transition probabilities are... P The reward function describes the probability distribution of transitioning to other states after taking a specific action in a given state. Discount factor is used to measure the reward value obtained by the agent in each state-action pair. This is used to control the degree to which future rewards affect current decisions.

[0025] state space ,in, Used to describe the state information of the intelligent agent itself, specifically represented as In this expression, Used to identify the location of an agent in the environment; This indicates the magnitude and direction of the agent's velocity; For the target location; It is the radius of influence of the intelligent agent; The end time of an episode; It is the total distance the agent moves within one episode; The distance between the agent and the target; It is the minimum distance from the agent to the boundary of the environment; The minimum distance from the agent to all obstacles; It is the line connecting the agent's current position to the target and the horizontal axis. The included angle in the positive direction; This represents the intersection of the lines connecting the agent to the target and obstacles. The state information representing the obstacle is defined as follows: ,in , Indicates the position of the i-th obstacle. Representing the The speed and direction of each obstacle, For the first The relative position of an obstacle to the agent. It is the first The relative velocity of an obstacle with respect to the agent. For the first The radius of influence of an obstacle Indicates the first The minimum distance from an obstacle to an agent.

[0026] Action space Including the change in the magnitude of velocity and the change in velocity direction During the path planning process, It is used to adjust the speed of the intelligent agent's movement, enabling acceleration, deceleration, or uniform movement in different scenarios. For example, it can accelerate appropriately in an open area to shorten the time to reach the target, and decelerate when approaching obstacles or targets to ensure safe and accurate arrival. This is used to control the movement direction of the intelligent agent, enabling it to bypass obstacles and move towards the target direction based on environmental information. The state transition equation is:

[0027] Based on the actions in this action space This enabled intelligent agents to... Position and velocity at any moment Position and velocity transferred to the next moment This enables intelligent agents to move and plan paths within their environment.

[0028] reward function Rewards based on global planning Obstacle avoidance rewards Optimize time performance rewards And the process reward is composed of RRT and APF. It is used to optimize the overall performance of the path when the agent reaches the destination, and encourages the agent to complete the path planning in a better way by taking into account both time and distance. ,exist or To enhance path planning safety, a larger negative reward is given when the agent gets too close to an obstacle, prompting it to adjust its path to avoid a collision. To optimize time performance, a small negative reward is given at each step during path planning to incentivize the agent to reach the goal as quickly as possible and reduce path planning time. The total reward function in a static obstacle environment... Includes process rewards designed based on the RRT algorithm. Total reward function in mixed obstacle environment In addition to the rewards included in the static obstacle environment, a process reward based on the APF algorithm has been added. Each weight parameter (such as , , , (etc.) are used to adjust the weight of different reward items in order to balance the influence of different reward factors on the behavior of the agent, so that the agent can make reasonable decisions in different environments and achieve efficient path planning.

[0029] Application Method: Construct a DRL test environment containing static or mixed obstacles and set the corresponding parameters. When constructing a static obstacle environment, obstacles are divided into fixed linear static obstacles and randomly generated obstacles. Fixed linear static obstacles are defined by exact linear equations, such as... (Starting point (102.69, 103.36), Ending point (135.1, 76.47)) and (Start point (39.90, 98.76), End point (72.31, 71.96)), with an influence radius of 5, and a specific collision detection method is used to determine whether the agent collides with the obstacle. Specific principles are followed when randomly generating obstacles to avoid meaningless training and testing failures. In the mixed obstacle environment, in addition to the settings related to the static obstacles mentioned above, the speed of the dynamic obstacles is fixed at 5, the initial direction of motion is randomly determined, and the change in its speed direction at each time step is limited to a maximum of 5. Within the range of arc, this ensures that the movement of dynamic obstacles is controllable and realistically simulated, making the path planning training of the agent in this environment more in line with the needs of actual scenarios.

[0030] The DRL algorithm, implemented using the PyTorch framework, was used for policy training in this environment. During training, obstacle states and dynamic obstacle movement speeds were randomly initialized to introduce uncertainty and better simulate real-world scenarios. The DDPG algorithm was used as a baseline, and its hyperparameters were set as follows: initial learning rate (both Actor and Critic were set to 0.003), final learning rate (both Actor and Critic were set to 0.0003), discount factor to 0.99, replay buffer size to [value missing], batch size to 256, and soft update coefficient to 0.005. These parameter settings provide a stable foundation for DRL algorithm training, enabling the agent to learn policies effectively in different environments.

[0031] Evaluation metrics such as success rate (SR), failure rate (FR), time to reach the goal (T), total path length (D), and average speed (MS) are defined, and expectation and variance are introduced to evaluate the training performance of different DRL models. The same random seed is used during training and testing to ensure the repeatability and comparability of experimental results. Experiments verify the effectiveness of the reward shaping module, SCL module, PRA module, and PFT module, as well as the advantages of the IPPSL framework. The performance of the IPPSL-DRL method is compared with traditional methods (such as APF, RRT) and other DRL methods under different environments, comprehensively evaluating the effectiveness of the proposed method in path planning tasks from multiple dimensions, providing strong data support for practical applications.

[0032] Static obstacle environment: In a static obstacle environment, obstacles are meticulously divided into fixed linear static obstacles and randomly generated obstacles. Fixed linear static obstacles are defined by precise linear equations, such as... (Start point (102.69, 103.36), End point (135.1, 76.47)) and (Start point (39.90, 98.76), End point (72.31, 71.96)). The influence radius of this type of obstacle is set to 5, meaning that a collision is considered to have occurred when the distance between the agent and the linear obstacle is less than 5. A specific collision detection method is used to determine the state of the agent and the obstacle, as follows: If... and ,but ,like or ,but When randomly generating obstacles, specific principles are strictly followed to ensure that the distribution of generated obstacles is reasonable, avoid situations that lead to meaningless training and testing failures, and guarantee the effectiveness of training and testing.

[0033] Hybrid Obstacle Environment: In addition to static obstacle settings, the hybrid obstacle environment features special configurations for dynamic obstacles. The velocity of dynamic obstacles is fixed at 5, ensuring stability during simulation and facilitating the analysis and study of agent response strategies. Their initial direction of motion is randomly determined, increasing environmental uncertainty and complexity, better reflecting the unpredictable nature of dynamic obstacle movement in real-world scenarios. Furthermore, to ensure realistic simulation of dynamic obstacle movement, the change in velocity direction at each time step is limited to a maximum of radians. Within a certain range, it prevents dynamic obstacles from undergoing significant directional changes in a short period of time, ensuring that the path planning training of the agent in this environment can more realistically reflect the actual situation and improve the practicality and reliability of the training results.

[0034] Parameter Tuning - Reward Shaping Module: During training, specific parameter settings and dynamic adjustments are required for different modules. The reward shaping module involves multiple weight parameters, such as... In a static obstacle environment, For The reward and penalty system was adjusted based on the intersection of the agent's line with the target and obstacles. After multiple trials and optimizations, the final result was determined as follows: ; The reward function is used to adjust the non-intersecting condition, and its value is [value]. These parameters are fine-tuned during training to optimize performance. , Used to determine The weights of different reward items are assigned to balance their guiding role in agent behavior. In dynamic obstacle environments, Used for balance middle and Due to the influence of [the system], reasonable parameter settings can enable the agent to find a better balance between obstacle avoidance and target approach, thereby improving path planning performance.

[0035] Parameter Adjustment - Other Modules: In the SCL module, the terminal distance constraints are set to 20, 15, 10, and 5, and remain consistent in both static and mixed obstacle environments. In the static obstacle environment, the number of training rounds for the four stages are set to 2000, 1000, 1000, and 1000 respectively. In the mixed obstacle environment, considering the increased environmental complexity due to the presence of dynamic obstacles, the number of training rounds for the four stages are set to 2000, 1500, 1500, and 1000 respectively. This setting allows the agent to gradually adapt to tasks of varying difficulty. In the PRA module, a preset list of weight coefficients is set. And determine the weighting coefficients. The range of changes and adjustment rules, such as selecting values ​​from the list in a certain order at different training stages, and assigning them in the early stages of training. Larger values ​​initially lead to process rewards dominating the reward framework, gradually decreasing as training progresses. This value allows sparse rewards to play a dominant role in the later stages. A spatial threshold is also set. and time threshold In a static obstacle environment It is 3.2. The value is 0.8 in mixed obstacle environments. It is 6.4. A value of 0.8 is used to determine the convergence of the policy optimization and whether to stop weight adjustment. The PFT module employs a learning rate decay model. Set the initial learning rate according to different training stages. Final learning rate and attenuation coefficient In the multi-stage learning process involving static and mixed obstacle environments, different stages... , and Each has different values; for example, in a static obstacle environment, the four stages... , and The system gradually changes according to certain rules, and dynamically adjusts itself according to corresponding rules in mixed obstacle environments. This ensures that the agent can quickly learn new strategies at different stages, while also enabling the strategies to converge stably and work in conjunction with other modules to improve the overall learning effect.

[0036] The beneficial effects of this invention are as follows: This invention demonstrates significant effectiveness in improving training efficiency. The reward shaping module integrates RRT and APF into the DRL reward framework, providing real-time feedback to the agent, enabling it to quickly acquire feasible strategies in both static and mixed obstacle environments. In static obstacle environments, the process reward designed based on the RRT algorithm guides the agent to effectively avoid obstacles and approach the target, resulting in a substantial improvement in training efficiency compared to not using this module, and faster convergence to a stable and feasible strategy. In mixed obstacle environments, the combination of RRT and APF reward mechanisms further enhances the agent's obstacle avoidance and target approach capabilities, allowing the agent to obtain higher reward values ​​and more stable strategies with the same training resources. The SCL module, by gradually increasing task complexity and optimizing sample utilization, allows the agent to accumulate experience starting with simple tasks, avoiding blind exploration in complex tasks. Working in conjunction with the reward shaping module, it further improves policy learning efficiency. This synergy is particularly evident in dynamic obstacle environments; the PS-DRL algorithm (including the SCL and reward shaping modules) enables the agent to obtain a higher final reward value with the same number of training rounds compared to the P-DRL algorithm using only the reward shaping module. The PFT module dynamically adjusts the learning rate, enabling the agent to quickly explore new strategies in the early stages of training and then fine-tunes the strategies in the later stages. This ensures a smooth transition and overall stability in multi-stage strategy learning, accelerates the overall strategy learning process, and allows the agent to train higher-quality strategies with limited training resources.

[0037] This invention achieves significant results in resolving the reward coupling problem. The PRA module, based on dynamic reward reshaping, effectively adjusts the weights of process rewards and sparse rewards. In the early stages of learning, process rewards dominate, helping the agent quickly adapt to the environment and generate initial feasible strategies. As training progresses, the weight of sparse rewards gradually increases, guiding the strategy towards global optimum optimization. In this way, the PRA module successfully solves the reward coupling problem, preventing the strategy from getting trapped in local optima due to reward coupling effects. Experiments show that in static and mixed obstacle environments, after adopting the PRA module, the various performance indicators of the strategy are continuously optimized during the learning process, and the final strategy approaches global optimum. For example, in a static obstacle environment, after different stages of learning, the success rate of the strategy continuously improves, and the path length and time required to reach the target are continuously optimized; in a mixed obstacle environment, similarly good optimization results are achieved. This fully verifies the effectiveness of the PRA module in solving the reward coupling problem, ensuring that the agent can learn a globally optimal strategy.

[0038] This invention demonstrates excellent adaptability and significant advantages in various environments. In static obstacle environments, the IPPSL-DRL method achieves a 100% success rate. Compared to traditional APF and RRT methods, as well as the PS-DRL method which only includes the SCL and reward shaping modules, it outperforms in evaluation metrics such as success rate, expected distance, expected time, and variance of distance and time, indicating that its path planning is more efficient and the strategy is more stable. In mixed obstacle environments, the IPPSL-DRL method also performs excellently, with a success rate far exceeding that of traditional methods. It also outperforms in path length, time to reach the target, and other aspects, demonstrating better strategy stability. This shows that the path planning method of this invention can effectively cope with complex environments. Whether in static or dynamic obstacle scenarios, it can plan high-quality paths for the agent, meeting the requirements of accuracy, efficiency, and stability in practical applications, and possessing high practical value and broad application prospects. Attached Figure Description

[0039] Figure 1 This describes the basic learning mechanism of deep reinforcement learning in path planning tasks.

[0040] Figure 2 This paper describes the learning mechanism and process of deep reinforcement learning based on IPPSL in path planning tasks.

[0041] Figure 3 This is a geometric diagram illustrating the minimum distance between an agent and linear obstacles in a path planning task.

[0042] Figure 4 This is a comparison chart of training results with and without the reward shaping module.

[0043] Figure 5 This is a comparison chart of the training results of the PS-DRL algorithm (which includes the SCL module and the reward shaping module) and the P-DRL algorithm (which only uses the reward shaping module).

[0044] Figure 6 The training curves of the IPPSL framework at different stages in a static obstacle environment are shown.

[0045] Figure 7 The training curves of the IPPSL framework at different stages in a mixed obstacle environment are shown.

[0046] Figure 8 This is a comparison chart of the trajectory performance of the IPPSL-DRL method with the traditional APF, RRT, and PS-DRL methods in a static obstacle environment.

[0047] Figure 9 This is a comparison chart of the trajectory performance of the IPPSL-DRL method with the traditional APF, RRT, and PS-DRL methods in a mixed obstacle environment.

[0048] Figure 10 This image shows a comparison of the results of progressive training using the DRL algorithm combined with the IPPSL framework and training directly to the final stage in static and mixed obstacle environments. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0050] The incremental reinforcement learning path planning method proposed in this invention based on dynamic reward reshaping is essentially a fusion of the dynamic reward reshaping concept and progressive learning strategy. It consists of a reward shaping module, a sub-goal course learning (SCL) module, a progressive reward adjustment (PRA) module, and a policy fine-tuning (PFT) module working together to solve the problems of low training efficiency and difficulty in composite optimization in path planning.

[0051] refer to Figure 1 "The basic learning mechanism of deep reinforcement learning in path planning tasks" and Figure 2 The "Learning Mechanism and Process of Deep Reinforcement Learning Based on IPPSL in Path Planning Tasks" module integrates Rapid Exploratory Random Tree (RRT) and Artificial Potential Field (APF) into the deep reinforcement learning reward framework, providing real-time feedback to the agent and accelerating policy learning. The SCL module gradually increases task complexity by setting sub-objectives, optimizing sample utilization and helping the agent quickly acquire an initial policy. The PRA module, based on dynamic reward reshaping, adjusts the weights of process rewards and sparse rewards, resolving the reward coupling problem and guiding the policy from the initial feasible stage to the globally optimal stage. The PFT module uses a learning rate decay model to dynamically adjust the learning rate, supporting the PRA module and ensuring a smooth transition and overall stability in multi-stage policy learning.

[0052] This method demonstrates good adaptability in both static and mixed obstacle environments. In static obstacle environments, the modules cooperate to enable the agent to effectively avoid obstacles and approach the target. In mixed obstacle environments, facing dynamically changing obstacles, the modules work together to ensure that the agent plans a high-quality path, meeting the requirements of accuracy, efficiency, and stability in path planning in practical applications. It has high practical value and broad application prospects.

[0053] refer to Figure 1 "The basic learning mechanism of deep reinforcement learning in path planning tasks" and Figure 2The "Learning Mechanism and Process of Deep Reinforcement Learning Based on IPPSL in Path Planning Tasks" module integrates Rapid Exploratory Random Tree (RRT) and Artificial Potential Field (APF) into the deep reinforcement learning reward framework, providing real-time feedback to the agent and accelerating policy learning. The SCL module gradually increases task complexity by setting sub-objectives, optimizing sample utilization and helping the agent quickly acquire an initial policy. The PRA module, based on dynamic reward reshaping, adjusts the weights of process rewards and sparse rewards, resolving the reward coupling problem and guiding the policy from the initial feasible stage to the globally optimal stage. The PFT module uses a learning rate decay model to dynamically adjust the learning rate, supporting the PRA module and ensuring a smooth transition and overall stability in multi-stage policy learning.

[0054] This method demonstrates good adaptability in both static and mixed obstacle environments. In static obstacle environments, the modules cooperate to enable the agent to effectively avoid obstacles and approach the target. In mixed obstacle environments, facing dynamically changing obstacles, the modules work together to ensure that the agent plans a high-quality path, meeting the requirements of accuracy, efficiency, and stability in path planning in practical applications. It has high practical value and broad application prospects.

[0055] In static obstacle environments, the reward shaping module fully utilizes the characteristics of the RRT algorithm to design process rewards, thereby guiding the agent to optimize its path. Process Rewards and The composition, and the calculation formula therein, are as follows:

[0056] This formula operates based on the intersection of the line connecting the agent to the target and obstacles. Among these, Indicates in The current position of the agent is connected to the target line and horizontal distance. The angle between the positive axis and the axis; Indicates in At time +1, the next position predicted or actually reached by the agent is connected to the target line and horizontal plane. The angle between the positive axis and the axis. When When the line connecting the agent to the target intersects with the obstacle, this indicates that the agent is in a position to move towards the target. As a weighting parameter, its value affects the degree of reward or penalty. This encourages the agent to adjust its trajectory to avoid collisions; when This means that the line connecting the agent and the target is unobstructed. This encourages agents to prioritize the shortest, unobstructed path towards their goal. The reward function was adjusted for this situation. During actual training, it was determined through multiple trials and optimizations. =32, for optimal performance.

[0057] To encourage agents to adopt smoother speed transitions and trajectory adjustments, and to avoid planning overly sharp paths, a design was developed. Its calculation formula is ,in This represents the action taken by the agent in the first step. This represents predefined action restrictions. Combining these two reward items yields... The formula is , , These are weight parameters that can be fine-tuned during training to optimize the overall reward effect, thereby guiding the agent to learn and plan paths more efficiently in static obstacle environments.

[0058] In dynamic obstacle environments, the reward shaping module combines RRT and APF algorithms to design process rewards. It is made of and The formula for calculating the composition is:

[0059] This formula directly improves the existing obstacle avoidance reward function based on the distance between the agent and the obstacle. When the agent is in a certain state, a reward of -300 is given to strongly incentivize it to avoid obstacles; when... At that time, through This function further adjusts the reward based on distance to enhance obstacle avoidance; when When the distance between the agent and the obstacle is too great, the reward is 0, indicating that the agent is far enough away from the obstacle that no special reward is needed. The distance threshold was set to 20 in this study to ensure that the agent has enough reaction distance to deal with dynamic obstacles, maintain a safe distance when avoiding obstacles, and better achieve safe obstacle avoidance.

[0060] This formula is used to encourage the agent to approach the target by comparing the distance between the agent and the target at the current time and the previous time. This prompts the agent to move towards the target. , , These are weight parameters used to balance the influence of different reward functions. By reasonably adjusting these two parameters, the agent can better balance obstacle avoidance and goal approach behavior in dynamic obstacle environments, thus achieving more efficient path planning.

[0061] During training, the weight parameters of the reward shaping module 、 、 、 These play a crucial role. In static obstacle environments, For The reward and penalty functions are adjusted when the agent intersects with the target line and obstacles, and when there is no intersection. These functions are then fine-tuned during training to optimize performance. 、 Used to determine The weights of different reward items are adjusted to balance the guiding role of each reward on the agent's behavior.

[0062] In dynamic obstacle environments 、 Used for balance middle and Due to the influence of obstacles, reasonable parameter settings can enable the agent to find a better balance between obstacle avoidance and target approach, thus improving path planning performance. For example, after multiple experiments and optimizations, it was determined that in static obstacle environments, =32, in a dynamic obstacle environment, through continuous adjustment 、 The value of enables the agent to make reasonable decisions quickly when faced with complex dynamic obstacles, effectively avoid obstacles and reach the goal as soon as possible, thus achieving high efficiency and accuracy in path planning.

[0063] The SCL module initializes the task complexity and performance threshold to build the initial environment for the agent's learning. Specifically, it initializes the task complexity... Terminal performance threshold The settings of these two parameters determine the difficulty of the initial training environment and the performance standards that the agent needs to achieve. For example, in practical applications, The initial value can be set according to the basic difficulty of the specific task. If the task is relatively simple, a small initial value can be set so that the agent can learn gradually from easy to difficult. This is determined based on the expected initial performance of the agent, ensuring that the agent has a clear goal in the initial stage. Based on this, a DRL environment is constructed. This provides a training environment of specific complexity for the agent. In this environment, the agent begins policy learning, and its state space is... S The action space is A .

[0064] In the environment In, use the settings The DRL model is trained by having the agent continuously try different actions in the given environment, accumulating experience and optimizing the strategy. After training, the trained model is evaluated using random test cases. Specifically, a certain number of cases are randomly selected from a pre-defined set of test cases, and the agent executes the strategy in the scenarios represented by these cases, then the success rate is calculated. The success rate is calculated by dividing the number of times the agent successfully completes the task (e.g., successfully avoiding all obstacles and reaching the target point in a path planning task) by the total number of tests. This method allows for an objective assessment of the agent's learning performance under the current task complexity.

[0065] like This indicates that the agent is performing well at the current difficulty level and is able to complete the task relatively well. At this point, to further improve the agent's capabilities, it is necessary to increase the task complexity, i.e. At the same time, tighten the terminal threshold, so that For example, in path planning tasks, the complexity of the task can be increased by adding more obstacles or changing their distribution, while simultaneously reducing the acceptable error range for the agent to reach the target point. This encourages the agent to learn more complex strategies to adapt to more demanding tasks. This dynamic adjustment process allows the agent to gradually grow through increasingly challenging environments, avoiding learning stagnation caused by prolonged exposure to simple tasks.

[0066] like and This means that the task difficulty and performance requirements have reached the set limits. At this point, the agent has fully learned through training on increasingly difficult tasks and has the ability to cope with complex situations. The loop ends and the required strategy is output. Through this dynamic adjustment process, the SCL module enables the agent to start with simple tasks in the early stages of training, quickly accumulate experience, and converge to a suitable strategy. As training progresses, it gradually adapts to more complex tasks, effectively avoiding the problem of getting stuck in local optima due to facing high-difficulty tasks too early, thereby improving the agent's learning efficiency. In different application scenarios, such as robot navigation and autonomous driving, the SCL module can reasonably adjust the task complexity and performance thresholds according to the characteristics and requirements of specific tasks, helping the agent efficiently learn strategies that meet actual needs.

[0067] The PRA module constructs different total reward frameworks for static and mixed obstacle environments. In the static environment, the total reward function is: In mixed obstacle scenarios, the total reward function is: .in, As an optimization weight parameter, it plays a crucial role in the dynamic reward reshaping process. Numerous process reward functions (such as...) Variables and time in ) T ,distance s Factors such as these are coupled together, causing multiple reward functions to influence the policy trajectory, thus affecting the total reward function ( and Relative to optimization variables s and T It is not a strictly convex function. This could lead to a situation where the strategy is not globally optimal even when the reward value reaches its maximum.

[0068] In the early stages of learning, give Larger values ​​result in process rewards. or It dominates the reward framework. Taking a static obstacle environment as an example, at this time... The process reward designed based on the RRT algorithm provides the agent with rich real-time feedback, helping it quickly generate initial feasible strategies in unfamiliar environments. Based on this real-time feedback, the agent can continuously adjust its actions, gradually exploring a path that avoids obstacles and approaches the goal. As training progresses, the reward is gradually reduced... The value of sparse reward The weight of sparse rewards increases, playing a dominant role in later training stages. It is usually related to the global optimization goal of the task. For example, in a path planning task, it may be related to factors such as the overall length of the path and the time required to reach the goal, guiding the strategy to optimize in the direction of global optimum.

[0069] The specific algorithm steps of the PRA module are as follows: First, initialize the optimization weight parameters. Next, a total reward framework is constructed based on the formula in the corresponding environment. ; after that, based on Train the DRL model; then evaluate the trained model using 1000 random test cases and calculate the average index. and Here and These could be metrics related to path planning, such as success rate or average time required to reach the goal. Then, a pre-defined list of weighted coefficients is used. Each of them (from (Start), and repeat the steps of building the reward framework, training the model, and evaluating the model in sequence. When and When the strategy optimization has reached a certain convergence level, the loop stops and the output is released. , In practice, and are pre-set thresholds used to determine whether the strategy has converged to a relatively stable state close to the global optimum. For example, in a static obstacle environment, is determined through multiple trials. It is 3.2. It is 0.8; in mixed obstacle environments, It is 6.4. The value is 0.8. Through this process, the PRA module effectively utilizes the advantage of providing real-time feedback through process rewards, improving efficiency in the early stages of policy learning. At the same time, it gradually reduces the weight of process rewards, mitigating the impact of reward coupling effects and ensuring that the final policy can achieve global optimum.

[0070] Based on the experimental data, in a static obstacle environment, adjusting the reward weights through the PRA module continuously improves the success rate of the strategy as the training phase progresses, while optimizing the path length and the time required to reach the target. The same trend is observed in a mixed obstacle environment; with the help of the PRA module, the agent can better balance obstacle avoidance and target approach behaviors, finding a superior path planning strategy. This fully validates the effectiveness of the PRA module in solving the reward coupling problem, ensuring that the agent can learn the globally optimal strategy and providing a reliable guarantee for path planning tasks.

[0071] The PFT module employs a learning rate decay model to dynamically adjust the learning rate to support the PRA module, ensuring a smooth transition and overall stability during multi-stage policy learning. The learning rate decay model used is... , where represents the number of training rounds, which reflects the learning progress of the agent during the training process; As a key factor affecting learning rate decay, and The size of this value determines the rate at which the learning rate decays. During different training phases, this module gradually reduces the initial learning rate. and final learning rate .

[0072] In the early stages of training, a relatively high initial learning rate is set, allowing the agent to quickly explore a wide range of states and actions. Because the agent has limited knowledge of the environment and task at this stage, a higher learning rate allows it to try different strategies more quickly, accelerates the learning of new strategies and environmental features, avoids getting trapped in local optima, and better adapts to dynamically changing reward frameworks. For example, in the first phase of training in a static obstacle environment, a relatively high initial learning rate is set... =0.003, a relatively high learning rate, enables the agent to actively explore various possible paths, quickly accumulate experience, and find some preliminary feasible strategies when facing complex obstacle environments.

[0073] As training progresses, the learning rate gradually decreases. Within each specific training phase, the learning rate... The initial value is gradually decreased from a high value to a lower value. In the later stages of training, a smaller learning rate allows the agent to fine-tune the learned policy. At this point, the agent has accumulated some experience and needs fine-tuning of the policy to enhance its stability and convergence. Taking a mixed obstacle environment as an example, in the fourth stage of training, the learning rate is gradually decreased. By setting a small value, such as 0.000009, the agent can fine-tune the previously learned strategy using a smaller learning rate at this stage, further optimize path planning, improve the success rate of reaching the goal, and reduce path fluctuations to make the path more stable.

[0074] This dynamic adjustment of the learning rate at different stages, in conjunction with the dynamic adjustment of the reward framework in the PRA module, enables the agent to effectively learn and optimize the policy under different reward weight allocations. When the PRA module adjusts the reward weights, the PFT module adjusts the learning rate accordingly, ensuring the stability and efficiency of the overall policy learning process. When the PRA module adds sparse reward weights to guide the policy towards global optimum optimization, the PFT module reduces the learning rate, allowing the agent to adjust the policy more stably in this direction and avoiding policy instability caused by an excessively large learning rate.

[0075] In the multi-stage learning process in static obstacle environments and mixed obstacle environments, different stages and k Each has different values. In a static obstacle environment, the four stages... and k It changes gradually according to certain rules, such as the first stage. =0.003, =0.0003, k =0.002; In the fourth stage, =0.0005, =0.00001, k =0.008. In mixed obstacle environments, dynamic adjustments are also made according to the corresponding rules, such as in the first stage. =0.003, =0.0003, k =0.0005; Fourth stage =0.0005, =0.000009, k =0.008.

[0076] Through this dynamic adjustment, the PFT module ensures that the agent can quickly learn new policies at different stages while also enabling stable policy convergence. It also collaborates with other modules to improve overall learning performance. Experiments show that agents using the PFT module achieve faster policy convergence, higher quality final policies, and stronger adaptability and robustness in complex environments, fully validating the crucial role of the PFT module in path planning methods.

[0077] In the path planning method of this invention, the path planning problem of the agent is modeled as a Markov decision process (MDP), and the path planning is implemented through tuples. This modeling approach, once defined, demonstrates significant advantages in real-world applications. In scenarios such as robot navigation and autonomous driving, agents need to make decisions in complex and dynamically changing environments to plan the optimal path from the starting point to the destination. MDP provides an effective framework for solving these problems, enabling agents to make decisions based on the current state and continuously optimize their strategies based on environmental feedback. Taking a robot's material handling task in a warehouse as an example, the warehouse environment contains various shelves (which can be considered obstacles), and the robot needs to navigate through them and transport goods to a designated location. In this case, the state space... It includes information such as the robot's own position, velocity, target position, and the state of surrounding obstacles; motion space. This defines the actions the robot can take, such as accelerating, decelerating, and turning; and the state transition probabilities. Describes the probability of transitioning to another state after performing an action in the current state; reward function The robot is rewarded or penalized according to its behavior; for example, successfully avoiding obstacles and approaching the target will result in a positive reward, while colliding with obstacles will result in a negative reward; discount factor. It is used to control the degree to which future rewards affect current decisions, and it reflects the agent's trade-off between long-term and short-term goals.

[0078] In practical path planning, the state space Action space State transition probability Reward function and discount factor The specific definition and range of values ​​can be flexibly adjusted according to different environments. In a static obstacle environment, the position of the obstacle remains fixed, and the state space... In a mixed obstacle environment, the state information about obstacles is relatively stable; however, in a mixed obstacle environment, the addition of dynamic obstacles changes the state space. The state information of obstacles in the action space changes constantly over time, requiring the agent to perceive these changes and adjust its decisions more promptly. In different scenarios, the agent's action choices will vary. In open spaces, the agent may have more action options, such as significant acceleration and turning; while in narrow passages or areas with many obstacles, action choices will be more restricted to ensure safe passage. State transition probability. The probability of state transition when a robot performs the same action may differ depending on the environment. For example, the probability may vary on smooth and rough surfaces, depending on factors such as ground friction affecting the robot's motion. Reward Function The design needs to be adjusted according to the specific task objectives. In path planning tasks, in addition to considering the basic rewards of obstacle avoidance and reaching the goal, other reward items can be added according to actual needs, such as optimizing path length and reducing travel time. Discount factor The value of needs to be determined based on the nature of the task and the degree of dynamic change in the environment. If the environment changes rapidly, the agent should focus more on the current reward. The possible values ​​are relatively small; if the environment is relatively stable, the agent can focus more on long-term returns. Then the value can be relatively large.

[0079] When performing policy optimization and decision-making based on Multi-Deterministic Policy Gradient (MDP), the agent continuously interacts with the environment, selects actions based on the current state, and adjusts its policy based on reward feedback. During training, the agent utilizes historical experience and continuously updates its policy using optimization algorithms (such as the Deep Deterministic Policy Gradient (DDPG)) to maximize cumulative rewards. When faced with a new environmental state, the agent selects the optimal action based on the learned policy to achieve path planning. Simultaneously, as the environment dynamically changes, the parameters and structure of the MDP also need to be dynamically adjusted. For example, when new obstacles appear or their positions change, the state space needs to be re-evaluated. and state transition probability And adjust the reward function accordingly. This guides the agent to adapt to new environments, continuously optimizes path planning strategies, and ensures that path planning tasks can be completed efficiently in complex and ever-changing environments.

[0080] state space It plays a crucial role in path planning, comprehensively describing the state information of the agent and the environment, and providing a basis for the agent's decision-making. Among these, , Used to describe the agent's own state information, including the agent's position. ,speed Target location Radius of influence The end time of an episode The total distance traveled within an episode Distance to the target Minimum distance to the environmental boundary Minimum distance to all obstacles The line connecting the current position and the target, and the horizontal plane. Angle in the positive direction of the axis And the connections between the agent and the target, and the intersections with obstacles. In real-world path planning scenarios, the agent perceives this information in real time, determines its own state, and then makes reasonable decisions. For example, when the agent detects... A smaller value indicates the presence of an obstacle nearby, requiring immediate adjustment of speed and direction to avoid a collision; if... A larger value means that the agent is still some distance from the target and needs to continue moving towards it.

[0081] The state information representing the obstacle is defined as follows: ,in , Indicates the first i The location of the obstacle This represents its speed magnitude and direction. The relative position with respect to the agent. It is relative velocity. To affect the radius, This represents the minimum distance to the agent. In mixed obstacle environments, the agent needs to constantly monitor... The system uses information, especially the position and speed changes of dynamic obstacles, to adjust its own strategy in a timely manner. For example, in autonomous driving scenarios, the vehicle (intelligent agent) needs to plan a safe driving path based on information such as the position, speed, and driving direction of other vehicles (obstacles) around it.

[0082] Action space Including the change in the magnitude of velocity and the change in velocity direction These two parameters play a crucial role in different scenarios. During path planning, Used to adjust the agent's movement speed. In open areas, the agent can appropriately increase... Accelerate to shorten the time to reach the target; reduce speed when approaching obstacles or the target. Slowing down ensures safe and accurate arrival. For example, in warehouse material handling scenarios, robots can accelerate to improve handling efficiency when moving away from shelves and other obstacles; and reduce speed when approaching target shelves to avoid collisions with shelves or goods.

[0083] Change in velocity direction Used to control the movement direction of an intelligent agent, enabling it to bypass obstacles and move towards a target direction based on environmental information. The intelligent agent perceives the state space. The information in the middle is used to calculate the appropriate... Value. When encountering an obstacle, the agent adjusts its position based on the obstacle's location and its relative position to the obstacle. This achieves safe obstacle avoidance. State transition equation

[0084] Based on action space The actions within enable the intelligent agent to... Position and velocity at any moment Position and velocity transferred to the next moment This enables intelligent agents to move and plan paths within their environment.

[0085] reward function R Rewards based on global planning Obstacle avoidance rewards Optimize time performance rewards It also consists of process rewards based on RRT and APF, with each part of the reward playing a different role in path planning. This is used to optimize the overall path performance when the agent reaches its destination. By comprehensively considering time and distance, it encourages the agent to complete path planning in a more efficient manner. When the agent can reach the target in a shorter time and along a shorter path, Higher rewards will be given to guide the agent to seek more efficient path planning strategies.

[0086] ,exist or This enhances path planning safety. When the agent gets too close to an obstacle, a larger negative reward is given to encourage it to adjust its path to avoid a collision. To optimize time performance, a small negative reward is given at each step during path planning to incentivize the agent to reach the goal as quickly as possible and reduce path planning time. In static obstacle environments, the total reward function includes a process reward designed based on the RRT algorithm. Total reward function in mixed obstacle environment In addition to the rewards included in the static obstacle environment, a process reward based on the APF algorithm has been added. Each weight parameter (such as...) , , , (etc.) are used to adjust the weights of different reward items to balance the impact of different reward factors on the agent's behavior, enabling the agent to make reasonable decisions in different environments and achieve efficient path planning. For example, in a mixed obstacle environment, by reasonably adjusting... and The weights can help the agent find a better balance between obstacle avoidance and goal approach, improving the success rate and efficiency of path planning.

[0087] To comprehensively evaluate the performance of the incremental reinforcement learning path planning method based on dynamic reward reshaping proposed in this invention, it is necessary to construct a DRL test environment that includes static obstacles and mixed obstacles. The construction process requires defining key elements such as environment boundaries, setting obstacle types and quantities, and initializing agent positions and velocities.

[0088] In static obstacle environment construction, obstacles are divided into fixed linear static obstacles and randomly generated obstacles. Fixed linear static obstacles are defined by precise linear equations, such as... (Starting point (102.69, 103.36), Ending point (135.1, 76.47)) and (Start point (39.90, 98.76), End point (72.31, 71.96)). The influence radius of this type of obstacle is set to 5, meaning that when the distance between the agent and the linear obstacle is less than 5, a collision is considered to have occurred. A specific collision detection method is used to determine the state of the agent and the obstacle. If and , then ; if and ,but ,like or ,but When randomly generating obstacles, specific principles are followed to ensure that the distribution of generated obstacles is reasonable and to avoid situations that lead to meaningless failures in training and testing. For example, multiple obstacles should not be too concentrated or overlap with the target location to ensure the effectiveness of training and testing.

[0089] The hybrid obstacle environment adds dynamic obstacles to the existing static obstacle settings. The velocity of the dynamic obstacles is fixed at 5, ensuring stability during simulation and facilitating the analysis and study of the agent's response strategies. Their initial motion direction is randomly determined, increasing the uncertainty and complexity of the environment and better reflecting the unpredictable nature of dynamic obstacle motion in real-world scenarios. Furthermore, to ensure realistic simulation of the dynamic obstacle's motion, the change in its velocity direction at each time step is limited to a maximum of [value missing]. Within the range of the arc, it prevents dynamic obstacles from undergoing large-scale directional changes in a short period of time, ensuring that the path planning training of the agent in this environment can more realistically reflect the actual situation and improve the practicality and reliability of the training results.

[0090] In addition to setting obstacles, other environmental parameters also need to be determined. For example, if the environmental boundary is set to [0,175]×[0,175], the initial position, target position, initial velocity, and other parameters of the agent also need to be set reasonably according to the experimental requirements. In static obstacle environments and mixed obstacle environments, the settings of these parameters will be adjusted according to different experimental purposes and scenarios to comprehensively test the performance of the path planning method under various conditions.

[0091] This invention uses a DRL algorithm implemented in the PyTorch framework for policy training in a constructed test environment, selecting the DDPG algorithm as the baseline method. The DDPG algorithm is a deep reinforcement learning method based on the Actor-Critic (AC) framework. It utilizes the policy gradient method to optimize the optimal policy and consists of a Critic Network (CN) and an Actor Network (AN). The CN is responsible for learning the value function and evaluating the current state-action pair. (s,a) Under the current strategy π The expected return is lower; AN estimates the return by maximizing CN. Q The agent updates its policy parameters using the values ​​of these parameters, guiding it to choose actions that yield higher rewards. The DDPG algorithm is used in this study because it excels in handling continuous action spaces, effectively learns complex policies, and is suitable for the path planning task of this invention.

[0092] During training, to make the training more closely resemble real-world scenarios, obstacle states and dynamic obstacle movement speeds need to be randomly initialized. Specifically, for randomly generated obstacles in a static obstacle environment, their position, size, and other attributes are randomly determined while adhering to generation principles; in a mixed obstacle environment, the initial movement direction of dynamic obstacles is also randomly set. This random initialization increases the uncertainty of training, allowing the agent to learn under different initial conditions, thereby improving its adaptability to various environmental situations. Simultaneously, to ensure the repeatability and comparability of experimental results, the same random seed is used in both training and testing. This means that each experiment begins under the same initial conditions, eliminating the interference of random factors on the experimental results, making comparisons between different experiments more reliable, and enabling accurate evaluation of the impact of different modules and algorithms on path planning performance.

[0093] When setting the hyperparameters of the DDPG algorithm, both performance and stability were fully considered. An initial learning rate (both Actor and Critic set to 0.003) provides the agent with a large learning step size in the early stages of training, enabling it to quickly explore new policies and environmental features and avoid getting trapped in local optima. As training progresses, the final learning rate (both Actor and Critic set to 0.0003) gradually decreases, allowing the agent to fine-tune the policy in later stages, enhancing its stability and convergence. A discount factor of 0.99 indicates that the agent prioritizes future rewards and is willing to make appropriate decisions in the present to gain longer-term benefits. The replay buffer size is set to 1e6. A larger buffer can store more historical experience, allowing the agent to make fuller use of past information during training, reducing fluctuations during training, and improving learning efficiency. The batch size is set to 256. This value was determined based on multiple trials and experience, achieving a good balance between computational efficiency and training effect. The Soft Update Coefficient is set to 0.005 to control the update speed of the target network, avoiding overly rapid updates that could lead to training instability, while ensuring that the target network keeps up with changes in the current network.

[0094] To comprehensively evaluate the training performance of different DRL models, a series of evaluation metrics were defined, including success rate (SR), failure rate (FR), time to reach the target (T), total path length (D), and average speed (MS). Success rate (SR) represents the probability that the agent successfully reaches the target without collision, and is one of the key indicators for measuring the effectiveness of path planning methods. Failure rate (FR) is the probability that the agent fails to reach the target or collides, complementing success rate and reflecting the agent's performance more comprehensively. Time to reach the target (T) directly reflects the efficiency of path planning; a shorter time implies a more efficient path planning strategy. Total path length (D) reflects the actual distance traveled by the agent; a shorter path length usually indicates better path planning. Average speed (MS) is calculated by the ratio of total path length to time to reach the target, comprehensively considering both path length and time factors, and can more comprehensively evaluate the agent's motion efficiency during path planning.

[0095] When evaluating the training performance of different DRL models, the statistical concepts of expectation and variance are introduced. Expectation is used to measure the average performance of the trained policy under different initial conditions. A better expectation value indicates that the policy performs better on average. Variance is used to evaluate the stability of the policy. A smaller variance means that the policy performs more stably in different tests, with less fluctuation and higher reliability.

[0096] These evaluation metrics and statistical methods are used to comprehensively evaluate different DRL models during training and testing. The performance of different models under the same conditions is compared, such as the success rate, path length, and time to reach the target in static obstacle and mixed obstacle environments, between the IPPSL-DRL method and traditional methods (e.g., APF, RRT) and other DRL methods. This multi-dimensional analysis of the performance advantages and disadvantages of each model provides strong data support for further optimization of path planning methods, and verifies the effectiveness of each module in this invention (e.g., reward shaping module, SCL module, PRA module, PFT module) and the advantages of the IPPSL framework.

[0097] The experimental data and charts show that the reward shaping module is significantly effective in improving training efficiency. In a static obstacle environment, the training results with and without this module were compared (see...). Figure 4 Without the RRT-based process reward function, the agent struggles to obtain feasible policies after all training episodes, with average rewards consistently remaining in the negative range. However, by introducing the RRT-based process reward function, the agent eventually obtains a stable and feasible policy. The same applies to mixed obstacle environments; the introduction of the process reward mechanism significantly increases the final convergence reward value during training. Furthermore, combining the RRT and APF process reward functions, compared to using only the RRT-based process reward function, provides the agent with more real-time obstacle avoidance feedback, enabling the agent to obtain higher reward values ​​and more stable feasible policies with the same learning resources (see [link to relevant documentation]). Figure 4 (b) and related test results.

[0098] The SCL module also makes a significant contribution to improving training efficiency. In static and mixed obstacle environments (see...),... Figure 5 The PS-DRL algorithm, which integrates the SCL module and the reward shaping module, achieves a higher final reward value with the same number of training rounds compared to the P-DRL algorithm, which only uses the reward shaping module. This indicates that the combination of the two modules improves policy learning efficiency. Especially in dynamic environments such as mixed obstacle scenarios, the SCL module's effect on assisting the reward shaping module is more pronounced. Test results show that the PS-DRL method outperforms the R-DRL and DRL methods on various policy learning metrics (see Table 7), highlighting the auxiliary role of the SCL module in helping the agent acquire its initial policy.

[0099] The PRA module effectively solves the reward coupling problem. In static and mixed obstacle environments (see...),... Figure 6 , Figure 7(and related table data) As the learning phase progresses, the reward value rapidly converges to a stable value, and the convergence speed of policy learning significantly accelerates. In the static obstacle environment, most metrics optimize iteratively during the learning phase; in the mixed obstacle environment, all metrics optimize with increasing iterations, and ultimately the policies obtained in both environments approach the global optimum. This fully verifies that the PRA module successfully balances process rewards and sparse rewards by dynamically adjusting reward weights, avoiding the policy from getting trapped in local optima due to reward coupling effects, and ensuring that the agent learns the globally optimal policy.

[0100] The PFT module is crucial for ensuring a smooth transition and overall stability in policy learning. In static obstacle environments (see...), Figure 6 The use of the PFT module significantly accelerates the learning efficiency of IPPSL-DRL in the third and fourth stages, and the number of episodes required for policy convergence and the final policy value are significantly improved; in dynamic obstacle environments (see...), Figure 7 The PFT module improves policy learning efficiency and final results at all stages and effectively alleviates instability in the policy learning process (the learning curve oscillates significantly when the PFT module is not used), enabling the agent to train higher quality policies with limited training resources.

[0101] By comparing the experimental results under different module combinations, a significant synergistic effect between the modules is clearly visible. The reward shaping module provides real-time feedback to the agent, helping it quickly acquire initial feasible policies; the SCL module optimizes sample utilization, enabling the agent to gradually accumulate experience from simple tasks, accelerating the learning of the initial policy; the combination of the two further improves learning efficiency. The PRA module solves the reward coupling problem, guiding the policy towards the global optimum, while the PFT module supports the PRA module by dynamically adjusting the learning rate, ensuring a smooth transition and stability of multi-stage policy learning. These modules work together to improve the performance of the overall path planning method, enabling the agent to efficiently plan high-quality paths in complex environments.

[0102] The incremental reinforcement learning path planning method based on dynamic reward reshaping demonstrates strong adaptability and significant advantages in various environments. In static obstacle environments, the IPPSL-DRL method achieves a 100% success rate. It outperforms traditional APF and RRT methods, as well as the PS-DRL method which only includes SCL and reward shaping modules, in terms of success rate, expected distance, expected time, and variance of distance and time. This indicates that the method offers more efficient path planning and a more stable policy. In mixed obstacle environments, the IPPSL-DRL method also performs exceptionally well, with a success rate far exceeding traditional methods. It also outperforms traditional methods in terms of path length, time to reach the target, and policy stability. This method has high value and broad prospects in practical applications. In robot navigation, it can help robots plan paths quickly and safely, improving work efficiency, whether in complex indoor environments or dynamic outdoor scenarios. In autonomous driving scenarios, it can ensure the safety and efficiency of vehicle operation in the face of various road conditions and dynamic obstacles. In logistics and transportation, it can optimize transportation routes and reduce transportation time and costs. The path planning method of this invention effectively addresses complex environments, meets the stringent requirements for accuracy, efficiency, and stability in practical applications, provides strong technical support for the development of related fields, and is of great significance for promoting progress in the field of path planning.

[0103] It should be noted that any process or method description in the flowchart or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which the embodiments of this application pertain.

[0104] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0105] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0107] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An incremental reinforcement learning path planning method based on dynamic reward reshaping, characterized in that, The path planning problem is modeled as a Markov Decision Process (MDP), and efficient path planning and global strategy optimization are achieved through the synergistic effect of the following modules: Reward Shaping Module: The Rapid Exploration Random Tree (RRT) and Artificial Potential Field (APF) methods are integrated into the deep reinforcement learning reward framework. In static obstacle environments, the RRT algorithm is used to design process rewards to guide the agent in optimizing the path. In mixed obstacle environments, the RRT and APF algorithms are combined to enhance the agent's obstacle avoidance and target approach capabilities, laying a reward foundation for subsequent policy learning and optimization and initially improving training efficiency. The Sub-goal Course Learning SCL Module: Building upon the reward mechanism provided by the Reward Shaping Module, this module sets sub-goals and gradually increases the complexity of tasks from simple ones. It dynamically adjusts the task difficulty and performance thresholds and optimizes the learning objectives based on the agent's training success rate. The aim is to further improve the agent's efficiency in the early learning stage and accelerate the acquisition of preliminary feasible strategies. Progressive Reward Adjustment (PRA) Module: Based on the initial strategies and dynamic reward reshaping concepts learned from the above modules, this module dynamically adjusts the weights of process rewards and sparse rewards. In the early stages of learning, it focuses on using process rewards to drive rapid obstacle avoidance and local path generation, while in the later stages, it uses sparse rewards to drive global optimal path optimization. Policy Fine-tuning (PFT) Module: To support the multi-stage reward weight adjustment of the Progressive Reward Adjustment Module, the learning rate is dynamically adjusted through the learning rate decay model during the dynamic reward reshaping process. The initial and final learning rates are gradually reduced in different training stages, and the learning rate is gradually reduced in each stage to ensure a smooth transition and stable convergence of policy learning under the multi-stage reward framework, thereby further improving the overall learning efficiency and policy robustness.

2. The incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 1, characterized in that, The Markov Decision Process (MDP) consists of tuples definition: state space ,in ； in, Used to identify the location of an agent in the environment. This indicates the magnitude and direction of the agent's velocity. For the target location, It is the radius of influence of the intelligent agent. The end time of an episode. It is the total distance the agent moves within one episode. The distance between the agent and the target. It is the minimum distance from the agent to the boundary of the environment. The minimum distance from the agent to all obstacles. It is the line connecting the agent's current position to the target and the horizontal plane. The angle along the positive direction of the axis. Represents the intersection of the lines connecting the agent to the target and obstacles; Indicates the first The location of the obstacle Representing the The speed and direction of each obstacle, For the first The relative position of an obstacle to the agent. It is the first The relative velocity of an obstacle with respect to the agent. For the first The radius of influence of an obstacle Indicates the first The minimum distance from an obstacle to the agent; Action space Including the change in the magnitude of velocity and the change in velocity direction The state transition equation is ； reward function Rewards based on global planning Obstacle avoidance rewards Optimize time performance rewards And the process reward based on RRT and APF; among which Discount factor .

3. The incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 1, characterized in that, In the reward shaping module: In a static obstacle environment, a process reward based on the RRT algorithm is designed. and The composition and calculation formulas are as follows: in, Based on the intersection of the agent's line of sight with the target and obstacles, the agent is encouraged to adjust its trajectory or prioritize the shortest unobstructed path. This is used to encourage agents to adopt smoother speed transitions and trajectory adjustments, avoiding the planning of overly sharp paths; , , , These are weight parameters that are fine-tuned during training to optimize performance. In a dynamic obstacle environment, the process of designing a reward system that combines RRT and APF algorithms is described. Depend on and Composition, calculation formula is in, Based on the distance between the agent and obstacles, the existing obstacle avoidance reward function is directly improved to enhance obstacle avoidance performance; Used to encourage agents to approach the goal; For the set distance threshold, , These are weighting parameters used to balance the influence of different reward functions.

4. The incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 1, characterized in that, The SCL module dynamically adjusts task complexity in the following manner. and performance threshold Based on the success rate of the trained agent Optimize learning objectives: First, initialize the task complexity. Terminal performance threshold This determines the initial training environment difficulty and the performance standards the agent needs to achieve; next, the DRL environment is constructed. This provides the agent with training scenarios of specific complexity; then, in the environment... Use Train the DRL model so that the agent can learn policies in this environment; After training, the trained model is evaluated using random test cases, and the success rate is calculated. This is used to determine the learning performance of the agent under the current task complexity. like This increases the complexity of the task. At the same time, tighten terminal thresholds This gradually increases the difficulty of the task, prompting the agent to learn more complex strategies to adapt to more demanding tasks; if and Then the loop ends and the required strategy is output. At this point, the agent has learned a strategy that meets the final requirements through training on increasingly difficult tasks.

5. The incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 1, characterized in that, The PRA module adjusts the reward framework in the following way to solve the reward coupling problem, enabling the strategy to transition from the initial stage of acquiring feasible strategies to the stage of pursuing the globally optimal strategy: In a static environment, the total reward function is: For process rewards, For sparse rewards; In mixed obstacle scenarios, the total reward function is: Rewards for the process; The specific algorithm steps are as follows: First, initialize the optimization weight parameters. Next, a total reward framework is constructed based on the formula in the corresponding environment. ; after that, based on Train the DRL model; then evaluate the trained model using 1000 random test cases and calculate the average index. and Then, the list of preset weight coefficients is... Each of them ,from Begin by repeating the steps described above: building the reward framework, training the model, and evaluating the model; when and When the loop stops, output the value. , .

6. The incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 1, characterized in that, The PFT module employs a learning rate decay model to dynamically adjust the learning rate to support the PRA module, ensuring a smooth transition and overall stability in multi-stage policy learning, as detailed below: The learning rate decay model is as follows: ,in For the number of training rounds, As a key factor affecting learning rate decay, and ; Gradually reduce the initial learning rate at different training stages. and final learning rate Within each specific training phase, the learning rate The learning rate gradually decreases from a high initial value to a lower terminal value. In the early stages of training, a relatively high learning rate allows the agent to quickly explore a wide range of states and actions, accelerating the learning speed of new policies and environmental features, avoiding getting trapped in local optima, and adapting to dynamically changing reward frameworks. As training progresses, the learning rate gradually decreases, allowing the agent to finely adjust the learned policies, enhancing the stability and convergence of the policies. This, combined with the dynamic adjustment of the reward framework in the PRA module, enables the agent to effectively learn and optimize policies under different reward weight allocations, ensuring the smoothness and efficiency of the overall policy learning process.

7. An application method based on the incremental reinforcement learning path planning method based on dynamic reward reshaping as described in any one of claims 1-6, characterized in that... include: Construct a DRL test environment containing static or mixed obstacles and set the corresponding parameters; The DRL algorithm implemented using the PyTorch framework was used for policy training in this environment. During training, obstacle states and dynamic obstacle movement velocities were randomly initialized, and the hyperparameters were set using the DDPG algorithm as a baseline method. Success rate (SR), failure rate (FR), time to reach the target (T), total path length (D), and average speed (MS) were defined. Expectation and variance were introduced to evaluate the training performance of different DRL models. The same random seed was used during training and testing. Experiments were conducted to verify the effects of the reward shaping module, SCL module, PRA module, and PFT module, as well as the advantages of the IPPSL framework. The performance of the IPPSL-DRL method was compared with traditional methods and other DRL methods in different environments.

8. The application method of the incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 7, characterized in that: In the constructed static obstacle environment, obstacles are divided into fixed linear static obstacles and randomly generated obstacles; Fixed linear static obstacles are defined by linear equations, with an influence radius of 5. A specific collision detection method is used to determine whether the agent collides with the obstacle. like and ,but , like or ,but , when When a collision is considered to have occurred, , To influence the radius of the intelligent agent, specific principles are followed when randomly generating obstacles to avoid meaningless training and testing failures; In a mixed obstacle environment, in addition to the settings related to the static obstacles mentioned above, the speed of the dynamic obstacles is fixed at 5, the initial direction of motion is randomly determined, and the change in their speed direction at each time step is limited to a maximum of 5. Within the range of arc, this ensures that the movement of dynamic obstacles is controllable and realistically simulated, making the path planning training of the agent in this environment more in line with the needs of actual scenarios.

9. The application method of the incremental reinforcement learning path planning method based on dynamic reward reshaping according to claim 7, characterized in that: During training, specific parameters are set and dynamically adjusted for different modules. For the reward shaping module, set weight parameters. In a static obstacle environment, For The system adjusts the rewards and penalties when the agent intersects with the target or obstacles. The reward function is used to adjust the non-intersecting reward function, which is then fine-tuned during training to optimize performance and ultimately determined. , Used to determine The weight of different reward items; In dynamic obstacle environments Used for balance middle and The impact; In the SCL module, the terminal distance constraints are set to 20, 15, 10, and 5, and remain consistent in both static and mixed obstacle environments. In the static obstacle environment, the number of training rounds for the four stages are set to 2000, 1000, 1000, and 1000, respectively. In the mixed obstacle environment, considering the presence of dynamic obstacles, the number of training rounds for the four stages are set to 2000, 1500, 1500, and 1000, respectively. This setting allows the agent to gradually adapt to tasks of different difficulty levels. In the PRA module, set the preset weight coefficient list. And determine the weighting coefficients. The range of changes and adjustment rules, such as selecting values ​​from the list in a certain order at different training stages, and assigning them in the early stages of training. Larger values ​​initially lead to process rewards dominating the reward framework, gradually decreasing as training progresses. The value allows sparse rewards to play a dominant role in the later stages; at the same time, a spatial threshold is set. and time threshold In a static obstacle environment It is 3.

2. The value is 0.8 in mixed obstacle environments. It is 6.

4. The value is 0.8, which is used to judge the convergence of the strategy optimization and decide whether to stop weight adjustment; The PFT module employs a learning rate decay model. Set the initial learning rate according to different training stages. Final learning rate and attenuation coefficient In the multi-stage learning process of static obstacle environments and mixed obstacle environments, different stages... , and Each has different values; for example, in a static obstacle environment, the four stages... , and The system gradually changes according to certain rules, and dynamically adjusts itself according to corresponding rules in mixed obstacle environments. This ensures that the agent can quickly learn new strategies at different stages, while also enabling the strategies to converge stably and work in conjunction with other modules to improve the overall learning effect.

Images