Path planning method for indoor complex dynamic environment

Abstract

The invention relates to a path planning method for an indoor complex dynamic environment, and discloses the path planning method for the indoor complex dynamic environment, which introduces dynamic disturbance optimization, adaptive exploration rate adjustment and priority experience playback mechanisms to improve the response capability and strategy stability of an intelligent agent to the dynamic environment. Experiments in three typical dynamic scenes of families, restaurants and libraries show that APE-DPO is superior to representative reinforcement learning models DQN and DDQN and variants thereof in the field of path planning in the aspects of path step number, success rate and convergence rate, the step number of part of scenes is reduced by 6.9%-12.5%, and good robustness and practical application potential are shown.

Description

Technical Field

[0001] This invention relates to a path planning method for complex and dynamic indoor environments, and relates to the field of path planning in complex scenarios. Background Technology

[0002] Reinforcement learning, as a trial-and-error-based learning method, is widely used in fields such as automatic decision-making and control systems, demonstrating strong environmental adaptability. However, its slow convergence speed and poor policy stability in high-dimensional state spaces limit its practical application in complex tasks. In the field of path planning, especially in complex situations with frequent dynamic changes and densely distributed obstacles, traditional reinforcement learning methods often struggle to efficiently learn stable policies. The rise of deep reinforcement learning, by introducing deep neural networks, has effectively improved the ability to extract state features and optimize policies, providing a new solution to the path planning problem. Among them, DQN, as a representative algorithm of deep reinforcement learning, has achieved breakthrough results in multiple tasks, but it still faces the problems of Q-value overestimation and training instability in path planning, resulting in large fluctuations in path selection in dynamic environments and difficulty in consistently improving planning performance. To alleviate these problems, DDQN (Double DQN) introduces a target network mechanism to mitigate the Q-value overestimation problem to some extent, improving training stability and policy reliability, but its real-time response capability in variable environments remains insufficient.

[0003] In path planning, an increasing number of researchers are focusing on the application of the DDQN algorithm. Jiang et al. proposed an improved Dual Deep Q-Network (IDDQN) algorithm to address the slow convergence and low accuracy issues of DDQN in complex obstacle environments during deep reinforcement learning. IDDQN, by introducing a second-order temporal difference method and a binary tree data structure, achieved a 26.89% improvement in step convergence time and a 22.58% improvement in reward convergence time in simple environments, while effectively avoiding local optima in complex environments. Han et al. introduced expert knowledge and an optimized reward function, employing a dual-branch network structure. This method accelerated model convergence, improved training stability, and generated smooth, short-path, and collision-free navigation solutions. Despite these significant improvements, applications in complex environments may still face challenges in computational efficiency and training time. Zhang et al. proposed an improved Dual Deep Q-Network (DDQN) algorithm for complex environments to address the challenges of DQN in path planning, combining A* and the Rapid Expanded Random Number (RRT) strategy. By optimizing robot initialization and reward functions, the improved DDQN significantly accelerates the learning process, achieving effective obstacle avoidance and optimal path planning. Zhou et al. proposed an indoor blind-zone autonomous path planning method based on deep reinforcement learning, addressing the problem of low coverage caused by neglecting blind-zone exploration in previous studies. This method combines Dual Deep Q-Network (DDQN) and Priority Experience Replay (PER), introducing a Blind Spot and Blocking Area (BBA) mechanism, significantly improving exploration efficiency and training speed, reducing training time from over one hour to 36 minutes, and increasing coverage by 11.37%. Literature combining supervised learning and reinforcement learning techniques proposes a method for autonomous vehicle navigation in dynamic unknown environments, overcoming the limitations of traditional navigation systems. Using Faster-R-CNN for obstacle recognition and Dual Deep Q-Network (DDQN) for navigation decisions, this model demonstrates good efficiency and performance in a game environment. In summary, existing research has made various improvements to the DDQN algorithm in complex path planning tasks, including structural optimization, reward function design, and enhanced exploration mechanisms, achieving certain results and showing positive effects in convergence speed, path smoothness, and collision avoidance. However, when faced with complex and dynamic indoor scenarios where obstacles move frequently, the environment layout is highly irregular, and the real-time requirements for paths are high, existing methods still have shortcomings in terms of training convergence stability, policy generalization ability, and path planning efficiency. In particular, when obstacles change drastically, the algorithm is prone to getting stuck in local optima or policy failure. Summary of the Invention

[0004] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0005] Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0006] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0007] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0008] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0009] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0010] To improve the adaptability and path quality of algorithms in complex and dynamic indoor environments, this invention proposes an Adaptive Prioritized Experience DDQN Path Optimization (APE-DPO) method. This method optimizes training stability and planning performance in dynamic environments by comprehensively improving the reward function, the Self-Adaptive Perturbation Optimization (SAM) mechanism, and the dynamic greedy factor adjustment strategy, and combining prioritized experience replay to enhance learning efficiency. In typical dynamic indoor scenarios, comparative experiments with DQN, DDQN, and their variants demonstrate that APE-DPO exhibits superior performance in terms of path steps, success rate, and reward convergence.

[0011] A path planning method for complex dynamic indoor environments, characterized by the following steps:

[0012] Step S1: Initialize Q network parameters, priority experience replay pool, and greedy factor;

[0013] Step S2: Select an action based on the status;

[0014] Step S3: Store the experience in the replay pool;

[0015] Step S4: Sample and train from the replay pool according to priority;

[0016] Step S5: Adaptively optimize and update parameters;

[0017] Step S6: Synchronize the target network parameters every fixed number of steps;

[0018] Step S7: Calculate the reward and adjust the path;

[0019] Step S8: Dynamically adjust the greed factor;

[0020] Step S9: Output path and evaluation metrics.

[0021] In one possible embodiment, step SS10 is also included: the agent module performs an action, and the environment provides feedback.

[0022] In one possible embodiment, step S10 specifically includes:

[0023] The agent module employs a dual-depth Q-network structure, introducing a dual-Q network structure to enhance the system's adaptability in dynamic and complex environments: the evaluation network Q... eval Responsible for action selection, target network Q target Used for action value estimation, avoiding the Q-value overestimation problem caused by the same network. At each time step t, the agent uses Q... eval Choose the optimal action a from the current state s. best As shown in equation (1):

[0024]

[0025] After the action is executed, a simplified Q-value update strategy is adopted to directly calculate the future reward of the optimal action through the target network, thereby improving the update stability. The update formula is shown in Equation (2):

[0026] Q eval (s,a)←Q eval (s,a)+α(r+γQ target (s′,a best )-Q eval (s,a)) (2)

[0027] Where α is the learning rate, γ is the discount factor, r is the immediate reward, and the target network Q is...target Estimate the optimal action a in the next state s′. best The value of is used to construct the target Q value, as shown in formula (3):

[0028] Q target (s,a)=r+γ·Q target (s′,a best (3)

[0029] By evaluating the interaction and update mechanism between the network and the target network, the system achieved a more stable Q-value estimation effect in path planning; to ensure Q target To ensure stability, APE-DPO periodically delivers Q eval The parameters are copied to the target network, as shown in formula (4):

[0030] θ target ←θ eval (4).

[0031] In one possible embodiment, the priority sampling in the replay pool in step S4 specifically includes: through an adaptive priority experience replay mechanism, the agent can dynamically adjust the sampling priority of experience samples based on the temporal difference (TD) error of the experience, so that important experiences with high TD error are learned first.

[0032] In one possible implementation, the adaptive priority experience replay mechanism is as follows:

[0033] The TD error δ of the empirical samples (s,a,r,s′) generated in each round of interaction is calculated. t The deviation between the current valuation and the target valuation is measured, as shown in formula (5):

[0034]

[0035] Where, δ t The larger the value, the greater the influence of the experience on the agent's decision-making, and the higher its learning priority should be assigned; experience samples are categorized by δ. t Sort and assign sampling probabilities p i The system is guided to learn key experiences in a concentrated manner, thereby improving the convergence speed and policy quality, as shown in equation (6):

[0036]

[0037] High TD error samples have a higher sampling probability. Even if the error is close to zero, the system still assigns it the minimum positive value. In addition, a four-layer fully connected Q network is used to extract state features and estimate action value, providing expressive power and accuracy support for path planning. The relevant calculations are shown in Equation (7):

[0038] Q(s,a)=f(W4·ReLU(W3·ReLU(W2·ReLU(W1·s)))) (7)

[0039] In optimizing Q-value estimation, the mean squared error (MSE) is used as the loss function during training, as shown in equation (8):

[0040] loss = MSE(Q) target Q eval (8).

[0041] In one possible embodiment, a training optimization step is also included:

[0042] Step SS1: Perturb the parameters according to the gradient direction of the loss function to enhance robustness and diversity;

[0043] Step SS2: Perform standard gradient update.

[0044] In one possible embodiment, step SS1 perturbs the current parameter using formula (9):

[0045]

[0046] Where ρ is a hyperparameter that controls the perturbation amplitude. Larger perturbations can enhance the generalization ability and robustness of the model, while appropriate control can avoid excessive bias.

[0047] Step SS2: After completing the perturbation, restore the original parameters and perform standard gradient updates to achieve the minimization optimization of the objective.

[0048] In one possible embodiment, step S7 includes:

[0049] Step S71: Each action grants a base negative reward r step =-3, encouraging reaching the goal as quickly as possible;

[0050] Step S72: Dynamically adjust the reward based on the change in relative distance to the target point, as shown in the following formula (10):

[0051]

[0052] Where, d new d old These represent the distances from the current position and the previous position to the target, respectively, and are used to incentivize actions that move closer to the target.

[0053] In complex and dynamic indoor environments, path planning agents must simultaneously address the spatial constraints of static obstacles and the frequent changes in dynamic obstacles. This uncertainty often leads to slow policy convergence and significant path fluctuations in traditional DDQN when facing dynamic environments, especially as obstacle density increases, causing a significant decline in path quality. To address these challenges, this paper proposes an Adaptive Priority Experience DDQN path planning method (APE-DPO). This method integrates three key mechanisms on top of DDQN: priority experience replay, adaptive perturbation optimization (SAM), and a dynamic exploration rate adjustment strategy. Its overall architecture is as follows: Figure 1 As shown, these modules are designed to address typical problems such as "reuse of important experience", "unstable convergence of strategies under environmental disturbances", and "imbalance between exploration and utilization", aiming to enhance the model's responsiveness to dynamic changes and improve path coherence and planning stability.

[0054] The APE-DPO architecture integrates a multi-module collaborative mechanism to achieve efficient path planning in complex dynamic environments. The agent module employs a dual-deep Q-network structure to mitigate Q-value overestimation and improve the accuracy and stability of policy selection. Through an adaptive priority experience replay mechanism, the agent dynamically adjusts the sampling priority of experience samples based on the temporal difference (TD) error of the experience, prioritizing important experiences with high TD errors. This potentially improves policy optimization efficiency, shortens policy convergence time during training, and increases the likelihood of the planned path successfully reaching the target. The introduced adaptive perturbation optimization (SAM) policy enhances generalization and robustness to dynamic obstacle changes, helping to reduce collisions and policy instability during path planning. Simultaneously, the dynamic exploration rate adjustment mechanism allows the agent to maintain a high exploration frequency in areas of drastic environmental change, while gradually focusing on utilizing existing strategies in relatively stable areas. This mechanism design is expected to improve the coherence and overall efficiency of path planning.

[0055] Other aspects and advantages of the invention will be described in detail in the specific embodiments, and those skilled in the art will recognize from the following detailed description other aspects and advantages not expressly stated in this disclosure. Attached Figure Description

[0056] Figure 1 Overall architecture diagram of the APE-DPO path planning method;

[0057] Figure 2 : A schematic diagram of a complex and dynamic indoor environment in a home;

[0058] Figure 3 : Schematic diagram of the agent's path exploration and decision-making process;

[0059] Figure 4Design diagrams of 5 dynamic obstacles in a home environment;

[0060] Figure 5 : Trajectory diagram of 5 dynamic obstacles in the home;

[0061] Figure 6 Graph showing the number of steps per round for each algorithm on four dynamic obstacles in the library environment;

[0062] Figure 7 Success rate graph of each algorithm in each round for four dynamic obstacles in the library environment;

[0063] Figure 8 The graph shows the total reward for each algorithm in each round for the four dynamic obstacles in the library environment.

[0064] Figure 9 : Optimal path diagrams for each algorithm for four dynamic obstacles in the library environment. Detailed Implementation

[0065] In complex indoor environments, the challenges posed by dynamic changes and irregular layouts are increasing. Advances in path planning technology aim to provide more intelligent and efficient solutions to meet the needs of these environments. Traditional planning methods, such as A* algorithm, Dijkstra's algorithm, dynamic window method, and artificial potential field method, perform well in static environments. However, due to their reliance on static environment assumptions and prior information, they struggle to cope with the real-time path updates and decision-making requirements brought about by frequent obstacle movement and rapid environmental changes. Therefore, they often fail to meet the requirements of real-time performance and adaptability in complex dynamic scenarios. The rapid development of intelligent algorithms such as deep reinforcement learning has provided new opportunities for path planning. These methods can learn from the dynamic changes in the environment and the layout of obstacles, enabling the agent to make autonomous decisions and generate more efficient paths, thereby significantly improving the performance of path planning.

[0066] To visually illustrate the characteristics and challenges of complex and dynamic indoor environments, a home environment is used as an example for illustration. Figure 2 As shown in the diagram, the indoor space contains furniture, facilities, and multiple dynamically moving people. The intelligent agent needs to quickly plan an effective path in this complex and dynamic environment to ensure that it can safely and quickly reach the designated target location (marked as a star in the diagram).

[0067] To improve the adaptability and path quality of algorithms in complex and dynamic indoor environments, this paper proposes an Adaptive Prioritized Experience DDQN Path Optimization (APE-DPO) method. This method optimizes training stability and planning performance in dynamic environments by comprehensively improving the reward function, the Self-Adaptive Perturbation Optimization (SAM) mechanism, and the dynamic greedy factor adjustment strategy, and combining prioritized experience replay to enhance learning efficiency. In typical dynamic indoor scenarios, comparative experiments with DQN, DDQN, and their variants demonstrate that APE-DPO exhibits superior performance in terms of path steps, success rate, and reward convergence.

[0068] In complex and dynamic indoor environments, path planning agents must simultaneously address the spatial constraints of static obstacles and the frequent changes in dynamic obstacles. This uncertainty often leads to slow policy convergence and significant path fluctuations in traditional DDQN when facing dynamic environments, especially as obstacle density increases, causing a significant decline in path quality. To address these challenges, this paper proposes an Adaptive Priority Experience DDQN path planning method (APE-DPO). This method integrates three key mechanisms on top of DDQN: priority experience replay, adaptive perturbation optimization (SAM), and a dynamic exploration rate adjustment strategy. Its overall architecture is as follows: Figure 1 As shown, these modules are designed to address typical problems such as "reuse of important experience", "unstable convergence of strategies under environmental disturbances", and "imbalance between exploration and utilization", aiming to enhance the model's responsiveness to dynamic changes and improve path coherence and planning stability.

[0069] A path planning method for complex dynamic indoor environments, characterized by the following steps:

[0070] Step S1: Initialize Q network parameters, priority experience replay pool, and greedy factor;

[0071] Step S2: Select an action based on the status;

[0072] Step S3: Store the experience in the replay pool;

[0073] Step S4: Sample and train from the replay pool according to priority;

[0074] Step S5: Adaptively optimize and update parameters;

[0075] Step S6: Synchronize the target network parameters every fixed number of steps;

[0076] Step S7: Calculate the reward and adjust the path;

[0077] Step S8: Dynamically adjust the greed factor;

[0078] Step S9: Output path and evaluation metrics.

[0079] In one possible embodiment, step SS10 is also included: the agent module performs an action, and the environment provides feedback.

[0080] In one possible embodiment, step S10 specifically includes:

[0081] The agent module employs a dual-depth Q-network structure, introducing a dual-Q network structure to enhance the system's adaptability in dynamic and complex environments: the evaluation network Q... eval Responsible for action selection, target network Q target Used for action value estimation, avoiding the Q-value overestimation problem caused by the same network. At each time step t, the agent uses Q... eval Choose the optimal action a from the current state s. best As shown in equation (1):

[0082]

[0083] After the action is executed, a simplified Q-value update strategy is adopted to directly calculate the future reward of the optimal action through the target network, thereby improving the update stability. The update formula is shown in Equation (2):

[0084] Q eval (s,a)←Q eval (s,a)+a(r+γQ target (s′,a best )-Q eval (s,a)) (2)

[0085] Where α is the learning rate, γ is the discount factor, r is the immediate reward, and the target network Q is... target Estimate the optimal action a in the next state s′. best The value of is used to construct the target Q value, as shown in formula (3):

[0086] Q target (s,a)=r+γ·Q target (s′,a best (3)

[0087] By evaluating the interactive update mechanism between the network and the target network, the system achieved a more stable Q-value estimation performance in path planning, avoiding overestimation. To ensure Q... target To ensure stability, APE-DPO periodically delivers Q eval The parameters are copied to the target network, as shown in formula (4):

[0088] θ target ←θ eval (4).

[0089] In one possible embodiment, the priority sampling in the replay pool in step S4 specifically includes: through an adaptive priority experience replay mechanism, the agent can dynamically adjust the sampling priority of experience samples based on the temporal difference (TD) error of the experience, so that important experiences with high TD error are learned first.

[0090] In one possible implementation, the adaptive priority experience replay mechanism is as follows:

[0091] A TD error-driven priority sampling mechanism is introduced in the experience replay to improve learning efficiency. The TD error δ of the experience samples (s,a,r,s′) generated in each round of interaction is calculated. t The deviation between the current valuation and the target valuation is measured, as shown in formula (5):

[0092]

[0093] Where, δ t The larger the value, the greater the influence of the experience on the agent's decision-making, and the higher its learning priority should be assigned; experience samples are categorized by δ. t Sort and assign sampling probabilities p i The system is guided to learn key experiences in a concentrated manner, thereby improving the convergence speed and policy quality, as shown in equation (6):

[0094]

[0095] High TD error samples have a higher sampling probability. Even if the error is close to zero, the system still assigns it the minimum positive value to avoid ignoring valuable experience. This mechanism guides the agent to prioritize learning key samples that have a greater impact on the policy, improving the model's adaptability and convergence accuracy in dynamic and complex environments. In addition, a four-layer fully connected Q network is used to extract state features and estimate action values, providing expressive power and accuracy support for path planning. The relevant calculations are shown in Equation (7):

[0096] Q(s,a)=f(W4·ReLU(W3·ReLU(W2·ReLU(W1·s)))) (7)

[0097] In optimizing Q-value estimation, the mean squared error (MSE) is used as the loss function during training, as shown in equation (8):

[0098] loss = MSE(Q) target Q eval (8).

[0099] In one possible embodiment, a training optimization step is also included:

[0100] Step SS1: Perturb the parameters according to the gradient direction of the loss function to enhance robustness and diversity;

[0101] Step SS2: Perform standard gradient update.

[0102] In one possible embodiment, step SS1 perturbs the current parameter using formula (9):

[0103]

[0104] Where ρ is a hyperparameter that controls the perturbation amplitude. Larger perturbations can enhance the generalization ability and robustness of the model, while appropriate control can avoid excessive bias.

[0105] Step SS2: After completing the perturbation, restore the original parameters and perform standard gradient updates to achieve the minimization optimization of the objective.

[0106] Specifically, an adaptive perturbation optimization (SAM) strategy is introduced to improve the model's generalization ability in dynamic environments. This strategy performs two optimization steps in each training round: the first step perturbs the parameters according to the gradient direction of the loss function to enhance robustness and diversity; the second step performs standard gradient updates. SAM is particularly suitable for dynamic scenarios with frequently changing paths, helping to improve the stability and convergence performance of the system under complex tasks.

[0107] In the SAM strategy, the first step is to perturb the current parameters using the following formula:

[0108]

[0109] Here, ρ is a hyperparameter that controls the magnitude of the perturbation. A larger perturbation can enhance the model's generalization ability and robustness, while appropriate control can avoid excessive shift. After the perturbation is completed, SAM restores the original parameters and performs standard gradient updates in the second step, achieving minimization optimization of the objective. This two-step strategy ensures that the model remains stable when dealing with perturbation changes, thereby achieving a balance between exploration and convergence in dynamic environments and improving the APE-DPO method's adaptability and path planning accuracy in complex environments.

[0110] In one possible embodiment, to achieve a balance between exploration and exploitation during training, a dynamic exploration rate adjustment mechanism based on training progress is designed, employing an exponential decay strategy to update ∈, as shown in the following equation:

[0111] ∈=∈ min +(∈ max -∈ min )·e -k·t

[0112] Where, ∈ max and ∈ minHere, k represents the initial and minimum exploration rates, k is the decay factor, and t represents the number of training steps. This strategy maintains a high exploration rate in the early stages of training to gain diverse experiences, and then gradually reduces it to focus on the optimal strategy, thereby improving path planning efficiency and strategy stability. Combined with the training metric monitoring of the output module, APE-DPO can flexibly adjust according to task progress, achieving rapid response and accurate decision-making.

[0113] In one possible embodiment, step S7 includes:

[0114] Step S71: Each action grants a base negative reward r step =-3, encouraging reaching the goal as quickly as possible;

[0115] Step S72: Dynamically adjust the reward based on the change in relative distance to the target point, as shown in the following formula (10):

[0116]

[0117] Where, d new d old These represent the distances from the current position and the previous position to the target, respectively, and are used to incentivize actions that move closer to the target.

[0118] Specifically, this algorithm designs a multi-dimensional reward mechanism to improve the diversity of path exploration and the planning effect. When the agent is at a boundary position or repeatedly stops, it is given a slight negative reward r. border =-1, r stay Gradually decreasing. A base reward of r is given upon successfully reaching the goal. goal =300, and record the number of steps in this round as a reference. If the path is shorter in subsequent rounds, update the reference number of steps; otherwise, apply formula (11) to the difference and impose a penalty:

[0119] r goal =300-(steps) current -steps ref )×5(11)

[0120] To encourage exploration, a penalty of r is applied if a duplicate location is visited. repeat =-5, visit a new location and get a +2 bonus; a maximum step count of 100 steps is set, exceeding this limit without reaching the target will terminate the run and award r. fail = -300. In addition, to avoid collisions with obstacles, the system sets a collision penalty r. collision = -100, to ensure path safety.

[0121] To visually demonstrate the path exploration and decision-making mechanisms of intelligent agents in complex and dynamic environments, Figure 3Six typical scenarios are illustrated: (a) the agent begins to move and receives a basic negative reward; (b) and (c) the agent collides with static and dynamic obstacles respectively, the round ends, and a larger negative reward is given; (d) the optimal path to successfully reach the goal; (e) demonstrating the best choice among multiple paths, with the solid path being superior to the dashed path; and (f) the scenario where there is no collision during exploration but the number of steps exceeds the limit, the round ends, and a negative reward is given. These typical scenarios provide an intuitive background for the mechanism design of subsequent algorithms in path selection strategies, collision handling logic, and reward function construction, helping to understand the decision-making basis and optimization direction of the APE-DPO method in complex dynamic environments.

[0122] In some specific embodiments, the following steps are included: 1. Parameter settings, 2. Simulation environment, and 3. Experimental results and analysis.

[0123] The parameter settings include: The table below lists the main hyperparameters used in this study and their set values. Parameter selection referenced representative path planning studies in the field of deep reinforcement learning (such as DQN, DDQN, and their improved algorithms), with most values ​​being common values, and were initially optimized through experiments. The listed parameters include learning rate, discount factor, and exploration rate control terms, aiming to reasonably control the model's learning process and policy exploration behavior.

[0124] Table 1: Hyperparameter Configuration Table

[0125]

[0126] The simulation environment includes a 20x20 grid map designed to simulate three complex indoor dynamic scenarios: a home, a restaurant, and a library. Static obstacles are placed in each scenario, while 3 to 5 dynamic obstacles are added to increase the complexity of path planning. The agent must find the optimal path to the target point under the interference of these obstacles. To further evaluate the agent's adaptability to dynamic obstacles, the movement trajectory of the dynamic obstacles is shown. Figure 4-5 As shown in the diagram: blue represents the starting position, green represents the target position, black represents static obstacles, and red represents dynamic obstacles.

[0127] The experimental results and analysis include: To comprehensively evaluate the performance of the proposed Adaptive Priority Experience DDQN path planning method (APE-DPO) in complex dynamic environments, experiments were conducted in three different scenarios: home, restaurant, and library, with 3 to 5 dynamic obstacles set up in each scenario. To further analyze the algorithm's performance, this study compared the innovative algorithm with traditional DQN, DDQN, and DDQN combined with Priority Experience Replay (PER) algorithms. The following is a detailed analysis of the experimental results in the library environment with four dynamic obstacles.

[0128] Figure 6 The graph shows the number of steps per round for each algorithm in a library environment with four dynamic obstacles (from left to right: DQN, DDQN, DDQN combined with priority experience replay, and APE-DPO algorithm). Figure 6 It can be observed that DQN and DDQN exhibit significant fluctuations in the number of steps during the early stages of training, and while DDQN+PER shows some improvement, it remains unstable. In contrast, APE-DPO rapidly compresses the average number of steps in the early stages and stabilizes earlier, indicating that this method possesses strong path optimization capabilities and policy convergence speed in the early stages of learning.

[0129] Figure 7 The success rate graphs for each round of the algorithms for four dynamic obstacles in the library environment are shown (from left to right: DQN, DDQN, DDQN combined with priority experience replay, and APE-DPO algorithm). Furthermore, Figure 7 The training success rate trends of the four algorithms were compared under the same scenario. Overall, DQN and DDQN still showed significant fluctuations in the mid-to-late stages, while DDQN+PER showed some improvement in success rate, but overall remained highly volatile. APE-DPO, on the other hand, exhibited a steady upward trend, indicating stronger task completion capabilities and policy robustness in dynamic environments.

[0130] Figure 8 The graphs show the total reward for each algorithm in each round for four dynamic obstacles in a library environment (from left to right: DQN, DDQN, DDQN combined with priority experience replay, and APE-DPO algorithm). Figure 8 The study demonstrates the changes in total reward for each round of training for four algorithms in a library environment. DQN and DDQN exhibit significant negative reward fluctuations in the early stages of training. While DDQN+PER shows some improvement, its reward convergence speed is slow. In contrast, APE-DPO quickly reverses the negative reward trend in the early stages and maintains a stable increase in the mid-to-late stages, indicating stronger policy adaptability and path optimization potential in dynamic environments.

[0131] Figure 9 The optimal path graphs for each algorithm are shown for four dynamic obstacles in the library environment (from left to right: DQN, DDQN, DDQN combined with priority experience replay, and APE-DPO algorithm). Figure 9 The results show a comparison of typical optimal paths generated by four algorithms in a library environment. As can be seen from the image results, the paths generated by DQN and DDQN have the risk of detours or collisions. While DDQN+PER partially optimizes the path, it still appears disjointed. In contrast, the path generated by APE-DPO is more compact and efficient, effectively avoiding collisions even with complex obstacle layouts, demonstrating better path planning performance.

[0132] The analysis of the above indicators shows that APE-DPO demonstrates superior performance across multiple dimensions, including path steps, success rate, and reward convergence, in dynamic obstacle scenarios. This validates the effectiveness of its introduced mechanisms (prioritized experience replay, adaptive perturbation, and dynamic exploration rate control) in improving training stability and path quality.

[0133] Table 2: Comparison of experimental results of various algorithms in different scenarios

[0134]

[0135]

[0136] Experimental results show that APE-DPO outperforms existing algorithms in path optimization, convergence stability, and reward benefits. In terms of steps, it reduces the number of steps by 6.9%, up to 11.11%, and 12.5% ​​in home, restaurant, and library scenarios, respectively, compared to other methods. Although the success rate is slightly lower in some scenarios, APE-DPO exhibits better convergence and stability in the later stages of training. Traditional DQN and DDQN did not achieve effective convergence in the experiments due to the lack of a novel reward function; therefore, their maximum total reward is not listed.

[0137] This paper proposes an Adaptive Priority Experience-Based DDQN path planning method (APE-DPO). By optimizing experience replay, adaptive perturbation (SAM), dynamic exploration rate adjustment, and reward function design, it effectively improves the efficiency and stability of path planning in complex dynamic environments. Experiments show that it outperforms traditional algorithms in multiple scenarios, exhibiting better convergence and success rate.

[0138] It should be understood that although the invention has been specifically disclosed through preferred embodiments and optional features, modifications, improvements, and variations can be made to the invention disclosed herein by those skilled in the art, and such modifications, improvements, and variations are considered to be within the scope of the invention. The materials, methods, and embodiments provided herein are representative and exemplary of preferred embodiments and are not intended to be limiting of the scope of the invention.

Claims

1. A path planning method for complex dynamic indoor environments, characterized in that... Includes the following steps: Step S1: Initialize Q network parameters, priority experience replay pool, and greedy factor; Step S2: Select an action based on the status; Step S3: Store the experience in the replay pool; Step S4: Sample and train from the replay pool according to priority; Step S5: Adaptively optimize and update parameters; Step S6: Synchronize the target network parameters every fixed number of steps; Step S7: Calculate the reward and adjust the path; Step S8: Dynamically adjust the greed factor; Step S9: Output path and evaluation metrics.

2. The path planning method for complex dynamic indoor environments according to claim 1, characterized in that... It also includes step SS10: the agent module executes actions and provides environmental feedback.

3. The path planning method for complex dynamic indoor environments according to claim 2, characterized in that... Step S10 specifically includes: The agent module employs a dual-depth Q-network structure, introducing a dual-Q network structure to enhance the system's adaptability in dynamic and complex environments: the evaluation network Q... eval Responsible for action selection, target network Q target Used for action value estimation, avoiding the Q-value overestimation problem caused by the same network. At each time step t, the agent uses Q... eval Choose the optimal action a from the current state s. best As shown in equation (1): After the action is executed, a simplified Q-value update strategy is adopted to directly calculate the future reward of the optimal action through the target network, thereby improving the update stability. The update formula is shown in Equation (2): Q eval (s,a)←Q eval (s,a)+α(r+γQ target (s′,a best )-Q eval (s,a)) (2) Where α is the learning rate, γ is the discount factor, r is the immediate reward, and the target network Q is... target Estimate the optimal action a in the next state s′. best The value of is used to construct the target Q value, as shown in formula (3): Q target (s,a)=r+γ·Q target (s′,a best ) (3) By evaluating the interaction update mechanism between the network and the target network, the system achieved a more stable Q-value estimation effect in path planning; APE-DPO periodically updates the Q-value. eval The parameters are copied to the target network, as shown in formula (4): i target ←θ eval (4)。 4. The path planning method for complex dynamic indoor environments according to claim 1 or 2, characterized in that... The priority sampling in the replay pool in step S4 specifically includes: through the adaptive priority experience replay mechanism, the agent can dynamically adjust the sampling priority of experience samples according to the temporal difference (TD) error of the experience, so that important experiences with high TD error are learned first.

5. The path planning method for complex dynamic indoor environments according to claim 4, characterized in that... The adaptive priority experience replay mechanism is as follows: The TD error δ of the empirical samples (s,a,r,s′) generated in each round of interaction is calculated. t The deviation between the current valuation and the target valuation is measured, as shown in formula (5): Where, δ t The larger the value, the greater the influence of the experience on the agent's decision-making, and the higher its learning priority should be assigned; experience samples are categorized by δ. t Sort and assign sampling probabilities p i The system is guided to learn key experiences in a concentrated manner, thereby improving the convergence speed and strategy quality, as shown in equation (6): High TD error samples have a higher sampling probability. Even if the error is close to zero, the system still assigns it the minimum positive value. In addition, a four-layer fully connected Q network is used to extract state features and estimate action value, providing expressive power and accuracy support for path planning. The relevant calculations are shown in Equation (7): Q(s,a)=f(W4·ReLU(W3·ReLU(W2·ReLU(W1·s)))) (7) In optimizing Q-value estimation, the mean squared error (MSE) is used as the loss function during training, as shown in equation (8): loss=MSE(Q target ,Q eval ) (8)。 6. The path planning method for complex dynamic indoor environments according to claim 1, characterized in that... It also includes training optimization steps: Step SS1: Perturb the parameters according to the gradient direction of the loss function to enhance robustness and diversity; Step SS2: Perform standard gradient update.

7. The path planning method for complex dynamic indoor environments according to claim 6, characterized in that... Step SS1 perturbs the current parameters using formula (9): Where ρ is a hyperparameter that controls the perturbation amplitude. Larger perturbations can enhance the generalization ability and robustness of the model, while appropriate control can avoid excessive bias. Step SS2: After completing the perturbation, restore the original parameters and perform standard gradient updates to achieve the minimization optimization of the objective.

8. The path planning method for complex dynamic indoor environments according to claim 1, characterized in that... Step S7 includes: Step S71: Each action grants a base negative reward r step =-3, encouraging reaching the goal as quickly as possible; Step S72: Dynamically adjust the reward based on the change in relative distance to the target point, as shown in the following formula (10): Where, d new d old These represent the distances from the current position and the previous position to the target, respectively, and are used to incentivize actions that move closer to the target.

Images