Autonomous robot path planning method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of robot path planning, discloses an autonomous robot path planning method based on deep reinforcement learning, and aims at the problem that an agent is difficult to fully sense the environment in a complex dynamic environment, the State space of SAC is expanded and adjusted to enhance comprehensive sensing of an unknown environment and more flexible omnibearing obstacle avoidance. A heuristic reward function is designed, dynamic adjustment of reward values is achieved, and therefore the learning process of the robot is optimized. In order to solve the problems that the calculation complexity of an algorithm is increased and the convergence speed is reduced in a complex environment, the sample utilization rate is optimized in combination with a prior experience playback (PER) technology and a SumTree data structure. The weight of a high-value sample is increased through the PER technology, so that an intelligent agent can learn an effective strategy more quickly in the training process; and a SumTree data structure is introduced, so that the PER efficiency is further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot path planning, and in particular relates to a path planning method for an autonomous mobile robot based on deep reinforcement learning. Background Art

[0002] In recent years, Autonomous Mobile Robot (AMR) has attracted extensive attention in many fields such as logistics, warehousing, and home services. Path planning is the core issue of AMR autonomy in unknown environments. Its task is to find the optimal collision-free path from the starting point to the end point in an environment with obstacles. This path must meet several criteria: it should be as smooth, short, and efficient as possible. Traditional path planning methods, such as graph search, sampling, and geometric methods, have long been used for the task of mobile robot path planning. Although these methods can effectively find the optimal path in known and structured environments, they have poor adaptability when facing unstructured, dynamic, or complex environments. In particular, when dealing with increased environmental complexity, the computational complexity will increase significantly, affecting real-time performance. To address these limitations, bionic algorithms optimize path planning by simulating the behavior of organisms in nature, such as particle swarm optimization and ant colony optimization. However, these algorithms still have problems such as weak generalization ability and easy to fall into local optimality.

[0003] The above methods have a common constraint that mobile robots have poor adaptability when facing complex, unstructured or dynamic environments. With the increasing demand for complex environments, deep reinforcement learning (DRL) has gradually become an important tool for solving path planning problems. Unlike traditional algorithms, DRL does not rely on environmental models, but learns optimal strategies by continuously interacting with the environment, which is particularly suitable for autonomous navigation tasks in unstructured, dynamic or complex environments. However, existing value-based DRL algorithms (such as DQN) are computationally inefficient when dealing with continuous state spaces and are easily limited by the curse of dimensionality. Although policy gradient algorithms (such as DDPG and TD3) solve the problem of continuous action spaces, their insufficient exploration and the risk of convergence to suboptimal strategies still exist in complex environments. To address this problem, the maximum entropy deep reinforcement learning algorithm (SAC) enhances exploration capabilities and reduces the possibility of premature convergence to suboptimal strategies by maximizing the expected return while maximizing the entropy of the policy. On this basis, researchers have proposed a variety of improvement methods. For example, the long short-term memory network (LSTM) is introduced into SAC to utilize historical states to improve decision-making capabilities, or the Bi-LSTM network is used to retain and utilize past and future state information. Although these methods have improved the performance of SAC to a certain extent, LSTM and Bi-LSTM are not selective enough for information, and all information will be processed together. This will result in the inability of Bi-LSTM to effectively focus on the most relevant features when facing complex and changing environments, affecting the accuracy of path planning.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] Existing value-based DRL algorithms (such as DQN) are computationally inefficient when dealing with continuous state spaces and are easily limited by the curse of dimensionality. Although policy gradient algorithms (such as DDPG and TD3) solve the problem of continuous action spaces, their risk of insufficient exploration and convergence to suboptimal policies still exists in complex environments. Summary of the invention

[0006] In response to the problems existing in the prior art, the present invention provides a path planning method for an autonomous mobile robot based on deep reinforcement learning.

[0007] The present invention is implemented as follows: a path planning method for an autonomous mobile robot based on deep reinforcement learning comprises:

[0008] Step 1: Expand and adjust the SAC state space to fully perceive the unknown environment and flexibly avoid obstacles in all directions; dynamically adjust the reward value through the heuristic reward function;

[0009] Step 2: Combine the Prioritized Experience Replay (PER) technique with the SumTree data structure to optimize sample utilization. The PER technique increases the weight of high-value samples, allowing the agent to learn effective strategies during training.

[0010] Step 3: Dynamically adjust attention allocation through a gating network and a transformation network to improve the agent's selection accuracy for key states and actions and strengthen attention to key environmental features.

[0011] Furthermore, the method for expanding and adjusting the State space of SAC is as follows:

[0012] 1) State Space introduces four key metrics: G , D O , A O and M A ; where D G represents the Euclidean distance between the Mobile Robot and the Goal point, D O represents the Euclidean distance between the Mobile Robot and the obstacle, which is crucial for collision avoidance and ensuring safe navigation. O When the distance is less than 0.5m, the agent will determine it as a potential collision risk; A O Indicates the angle from the nearest obstacle, M A Indicates the angle the robot needs to turn from its current position to the target point;

[0013] 2)Action Space modeling.

[0014] Further, the Action Space modeling: In many previous studies, most of the work simplified and restricted the movement of mobile robots to eight fixed directions (a1, a2, ..., a8).

[0015] Furthermore, the reward function can be expressed as

[0016]

[0017] R0 represents the target reward function, R1 is the yaw angle reward function, R2 is the distance reward function, R3 and R4 are the shortest distance obstacle penalty function and collision penalty function respectively; together they constitute a multi-dimensional, dynamically adjusted reward mechanism; more specifically, R0 is set to the maximum positive reward; to motivate the mobile robot to reach the target point until the target distance D G On the other hand, R4 is set to the maximum negative reward to impose a penalty when the robot collides;

[0018] The yaw angle reward function R1 calculates the reward value based on the angle between the current forward direction of the mobile robot and the target point position; where α (α∈[-π,π]) is the radian representation of the angle between the forward direction of the robot and the target position; the formula is expressed as:

[0019]

[0020] The reason why 3 / π is chosen as the constant in the reward function is to control the yaw angle and ensure that the mobile robot moves towards the target direction. λ1 is used to balance the size of the R1 reward and ensure that the robot gets enough incentive when it approaches the target direction.

[0021] The distance reward function R2 is used to evaluate the distance between the mobile robot and the target position. The expression of this function is:

[0022]

[0023] Among them, D K is calculated by calculating the robot's position (x k ,y k ) and the target position (x Goal ,y Goal ), λ2 is a positive constant value used to control the reward value of the distance between time k and time k+1 relative to the target point; -1 is the size of the balance reward;

[0024] The obstacle reward function R3 aims to reduce the risk of the mobile robot colliding with the edge of obstacles during movement, especially considering the width of the robot; the function is designed as follows:

[0025]

[0026] Here, λ3 represents negative feedback, which is used to impose penalties on the mobile robot when it approaches an obstacle; the minimum safe distance between the robot and the obstacle is set as Min obs (Min obs =0.2), with the obstacle center as the center and the radius as Min obs The area is defined as a safety restricted area; when the mobile robot enters this area, it will receive negative feedback λ3 to ensure that the robot maintains a safe distance from obstacles.

[0027] Furthermore, the priority experience playback PER:

[0028] |δ t |=|R t +γminQ π (s t+1 , a′)-Q(s,a)|

[0029] The SumTree data structure is used to implement prioritized experience replay.

[0030] Furthermore, the SumTree improves the efficiency of experience sampling by storing all experience samples on leaf nodes and accumulating the priority sum of child nodes on internal nodes; when sampling, the total priority sum is divided into n intervals, where n = sum(p) / batchsize; ensuring that the experience in each interval has a chance to be selected; in addition, in order to prevent the network from overfitting, experience is extracted probabilistically to ensure that even experience with TD-errors of 0 can be extracted; the priority value of each experience is defined as:

[0031]

[0032] where δ i =(ξ i +ε ) a , ε is a small constant that ensures that all experiences have priority, and a is a tuning parameter that controls the degree of influence of TD error on the priority.

[0033] Another object of the present invention is to provide an autonomous mobile robot path planning system based on deep reinforcement learning, comprising:

[0034] The expansion module is used to expand and adjust the state space of SAC, fully perceive the unknown environment and flexibly avoid obstacles in all directions; dynamically adjust the reward value through the heuristic reward function;

[0035] The optimization module is used to optimize sample utilization by combining the Prioritized Experience Replay (PER) technology and the SumTree data structure. The PER technology increases the weight of high-value samples, allowing the agent to learn effective strategies during training.

[0036] The dynamic adjustment module is used to dynamically adjust attention allocation through a gating network and a transformation network to improve the agent's selection accuracy for key states and actions and strengthen attention to key environmental features.

[0037] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the autonomous mobile robot path planning method based on deep reinforcement learning.

[0038] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the autonomous mobile robot path planning method based on deep reinforcement learning.

[0039] Another object of the present invention is to provide an information data processing terminal, which is used to implement the autonomous mobile robot path planning system based on deep reinforcement learning.

[0040] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0041] First, in view of the technical problems existing in the above-mentioned prior art, some creative technical effects are brought about after solving the problems. The specific description is as follows:

[0042] 1) In order to solve the problem that the agent has difficulty in fully perceiving the environment in a complex dynamic environment, the State space of SAC is expanded and adjusted to enhance the comprehensive perception of the unknown environment and more flexible all-round obstacle avoidance; a heuristic reward function is designed to achieve dynamic adjustment of the reward value, thereby optimizing the robot's learning process.

[0043] 2) In order to solve the problem of increased algorithm computational complexity and slow convergence speed in complex environments, we combined the Prioritized Experience Replay (PER) technology and the SumTree data structure to optimize sample utilization. The PER technology increases the weight of high-value samples, allowing the agent to learn effective strategies faster during training; the introduction of the SumTree data structure further improves the efficiency of PER.

[0044] 3) The agent only uses the FC layer to linearly fit the environment state, resulting in a weak model expression ability. An improved gated attention mechanism is introduced to dynamically adjust the attention allocation through a gated network and a transformation network to improve the agent's selection accuracy for key states and actions. It strengthens the focus on key environmental features.

[0045] 4) In the training and generalization test experiments, the proposed method is verified to be capable and effective in avoiding obstacles in complex, dynamic and unstructured environments. The experimental results show that the GAP_SAC algorithm not only improves the success rate of path planning, but also shows good adaptability and robustness in the face of unknown and dynamic environments.

[0046] Second, the expected benefits and commercial value after the transformation of the technical solution of the present invention are as follows: In recent years, with the increasing demand for automation in the fields of logistics and home services, Autonomous Mobile Robot (AMR) robots have received widespread attention. The AMR path planning algorithm of the present invention aims to optimize the robot's autonomous navigation capabilities in unknown environments, and can efficiently and safely find the best collision-free path from the starting point to the destination. Expected benefits include significantly reducing operating costs, reducing dependence on manpower, improving work efficiency and response speed, and thus improving customer satisfaction. In addition, with the growing market demand for automated solutions, the algorithm has broad market potential and application scenarios, and can bring higher returns on investment to enterprises. By improving the competitiveness of AMR, this algorithm will enable enterprises to gain an advantage in the fierce market and achieve sustainable business value. Overall, the path planning algorithm not only improves the performance of AMR, but also creates new economic growth points for enterprises.

[0047] Whether the technical solution of the present invention overcomes technical bias: The GatedAttention and PERbased SAC (GAP_SAC) path planning algorithm of the present invention effectively overcomes the limitations of traditional technologies in complex dynamic environments. By expanding the state space and designing a heuristic reward function, the algorithm enhances the agent's comprehensive perception of unknown environments and solves the problem of insufficient expression ability of traditional fully connected layer models. In addition, combined with the Prioritized Experience Replay (PER) technology and the SumTree data structure, the sample utilization rate is optimized, the convergence speed is improved, and it is ensured that the intelligent agent can quickly learn effective strategies. The introduced gated attention mechanism further improves the selection accuracy of key states and actions, allowing the agent to flexibly avoid obstacles in complex environments. These innovations significantly improve the adaptability and robustness of the algorithm, overcome technical bias, and make the path planning of AMR in dynamic and unstructured environments more efficient and reliable.

[0048] Third, the present invention solves multiple industrial application problems in existing autonomous mobile robot path planning technologies. First, the existing technologies have poor robustness and flexibility in path planning in dynamic and complex environments, and cannot effectively respond to sudden obstacles or environmental changes. The present invention enhances the mobile robot's comprehensive perception of the environment by expanding and adjusting the state space of the deep reinforcement learning SAC algorithm, especially significantly improving performance in obstacle avoidance and dynamic path adjustment, thereby solving the path planning problem in complex environments.

[0049] Secondly, the traditional path planning method has deficiencies in sample utilization and learning efficiency, which makes it difficult for the robot to obtain sufficient effective experience within a limited time. The present invention effectively improves the utilization of high-value experience and enhances the efficiency of model training by introducing the Prioritized Experience Replay (PER) technology and SumTree data structure. The robot can learn effective strategies faster, significantly shorten the model training time, and improve the efficiency of task execution.

[0050] In addition, traditional path planning technology is not accurate enough in selecting key states and actions, and it is easy to ignore important features in the environment, resulting in low quality of path planning. This invention dynamically adjusts attention allocation by designing a gating network and a transformation network to ensure that the robot can accurately identify key states and actions, thereby making more accurate decisions in complex environments. This improvement greatly improves the planning accuracy and decision-making ability of robots in industrial applications.

[0051] Finally, the present invention has achieved significant technological progress in industrial applications, especially in the fields of intelligent warehousing and intelligent manufacturing. Through the application of deep reinforcement learning technology, mobile robots can complete path planning tasks autonomously, flexibly and efficiently, reducing dependence on human intervention and improving the automation level and safety of the system. This not only improves the adaptability of robots in actual industrial environments, but also greatly improves overall production and operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a method for autonomous mobile robot path planning based on deep reinforcement learning provided by an embodiment of the present invention.

[0053] Figure 2 This is a system structure diagram of an autonomous mobile robot path planning method based on deep reinforcement learning provided in an embodiment of the present invention.

[0054] Figure 3 It is an overall framework diagram of the path planning method for an autonomous mobile robot based on deep reinforcement learning provided in an embodiment of the present invention.

[0055] Figure 4 The SAC algorithm provided in the embodiment of the present invention uses MLP as the basic network graph.

[0056] Figure 5 It is a specific implementation flow chart of the gated attention mechanism provided by an embodiment of the present invention.

[0057] Figure 6 This is a simulation effect diagram of the gated attention mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] The following are two examples of industrial applications of autonomous mobile robot path planning methods based on deep reinforcement learning:

[0060] 1. Path planning for autonomous mobile robots in warehouse automation systems

[0061] In modern intelligent warehousing, autonomous mobile robots (AMRs) are widely used in automated material handling and commodity sorting. Through the method of the present invention, AMRs in warehouses can perform efficient path planning in complex and dynamically changing environments, avoid collisions and complete tasks quickly. By introducing an extended state space and a dynamically adjusted reward mechanism, the robot can flexibly perceive obstacles in the environment and adjust the path in real time to avoid obstacles, thereby improving the work efficiency and safety of the warehousing system. Combined with the priority experience playback technology and the SumTree data structure, the robot can continuously learn and optimize the path planning strategy, adapt to different task requirements and environmental changes, and improve the intelligence level of the entire warehousing system.

[0062] 2. Path planning for autonomous handling robots in smart factories

[0063] In smart manufacturing factories, autonomous handling robots are used for the handling of materials and products on production lines. The environment inside the factory is complex, including static equipment and dynamic personnel flow, and the accuracy and safety of path planning are crucial. The method of the present invention is applied to smart factories. The autonomous handling robot can accurately select key states and actions through deep reinforcement learning algorithms and gated network technology to improve the efficiency and accuracy of path planning. The robot can quickly avoid obstacles in a dynamic environment and reach the target point by the optimal path to ensure efficient completion of the handling task. This technology not only reduces manual intervention and improves the level of automation, but also reduces the risk of collision during the handling process, and improves the operational safety and production efficiency of the factory.

[0064] like Figure 1 As shown, an autonomous mobile robot path planning method based on deep reinforcement learning provided by an embodiment of the present invention includes the following steps:

[0065] S101, expand and adjust the SAC state space, fully perceive the unknown environment and flexibly avoid obstacles in all directions; dynamically adjust the reward value through the heuristic reward function;

[0066] S102, combines the Prioritized Experience Replay PER technology and the SumTree data structure to optimize sample utilization; the PER technology increases the weight of high-value samples, allowing the agent to learn effective strategies during training;

[0067] S103 dynamically adjusts attention allocation through a gating network and a transformation network to improve the agent's selection accuracy for key states and actions; and strengthens attention to key environmental features.

[0068] The method for expanding and adjusting the State space of SAC provided in the embodiment of the present invention is as follows:

[0069] 1) State Space introduces four key metrics: G , D O , A O and M A ; where D G represents the Euclidean distance between the Mobile Robot and the Goal point, D O represents the Euclidean distance between the Mobile Robot and the obstacle, which is crucial for collision avoidance and ensuring safe navigation. O When the distance is less than 0.5m, the agent will determine it as a potential collision risk; A O Indicates the angle from the nearest obstacle, M A Indicates the angle the robot needs to turn from its current position to the target point;

[0070] 2)Action Space modeling.

[0071] Further, the Action Space modeling: In many previous studies, most of the work simplified and restricted the movement of mobile robots to eight fixed directions (a1, a2, ..., a8).

[0072] Furthermore, the reward function can be expressed as

[0073]

[0074] R0 represents the target reward function, R1 is the yaw angle reward function, R2 is the distance reward function, R3 and R4 are the shortest distance obstacle penalty function and collision penalty function respectively; together they constitute a multi-dimensional, dynamically adjusted reward mechanism; more specifically, R0 is set to the maximum positive reward; to motivate the mobile robot to reach the target point until the target distance D G On the other hand, R4 is set to the maximum negative reward to impose a penalty when the robot collides;

[0075] The yaw angle reward function R1 calculates the reward value based on the angle between the current forward direction of the mobile robot and the target point position; where α (α∈[-π,π]) is the radian representation of the angle between the forward direction of the robot and the target position; the formula is expressed as:

[0076]

[0077] The reason why 3 / π is chosen as the constant in the reward function is to control the yaw angle and ensure that the mobile robot moves towards the target direction. λ1 is used to balance the size of the R1 reward and ensure that the robot gets enough incentive when it approaches the target direction.

[0078] The distance reward function R2 is used to evaluate the distance between the mobile robot and the target position. The expression of this function is:

[0079]

[0080] Among them, D K is calculated by calculating the robot's position (x k ,y k ) and the target position (x Goal ,y Goal ), λ2 is a positive constant value used to control the reward value of the distance between time k and time k+1 relative to the target point; -1 is the size of the balance reward;

[0081] The obstacle reward function R3 aims to reduce the risk of the mobile robot colliding with the edge of obstacles during movement, especially considering the width of the robot; the function is designed as follows:

[0082]

[0083] Here, λ3 represents negative feedback, which is used to impose penalties on the mobile robot when it approaches an obstacle; the minimum safe distance between the robot and the obstacle is set as Min obs (Min obs =0.2), with the obstacle center as the center and the radius as Min obs The area is defined as a safety restricted area; when the mobile robot enters this area, it will receive negative feedback λ3 to ensure that the robot maintains a safe distance from obstacles.

[0084] The preferred experience playback PER provided by the embodiment of the present invention is:

[0085] |δ t |=|R t +γminQ π (s t+1 , a′)-Q(s,a)|

[0086] The SumTree data structure is used to implement prioritized experience replay.

[0087] The SumTree provided in the embodiment of the present invention improves the efficiency of experience sampling by storing all experience samples on leaf nodes and accumulating the priority sum of child nodes on internal nodes; when sampling, the total priority sum is divided into n intervals, where n=sum(p) / batchsize; it is ensured that the experience in each interval has a chance to be selected; in addition, in order to prevent network overfitting, experience is extracted with probability to ensure that even experience with TD-errors of 0 can be extracted; the priority value of each experience is defined as:

[0088]

[0089] where δ i =(ξ i +ε ) a , ε is a small constant that ensures that all experiences have priority, and a is a tuning parameter that controls the degree of influence of TD error on the priority.

[0090] like Figure 2 As shown, an autonomous mobile robot path planning system based on deep reinforcement learning provided by an embodiment of the present invention includes:

[0091] The expansion module is used to expand and adjust the state space of SAC, fully perceive the unknown environment and flexibly avoid obstacles in all directions; dynamically adjust the reward value through the heuristic reward function;

[0092] The optimization module is used to optimize sample utilization by combining the Prioritized Experience Replay (PER) technology and the SumTree data structure. The PER technology increases the weight of high-value samples, allowing the agent to learn effective strategies during training.

[0093] The dynamic adjustment module is used to dynamically adjust attention allocation through a gating network and a transformation network to improve the agent's selection accuracy for key states and actions and strengthen attention to key environmental features.

[0094] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the autonomous mobile robot path planning method based on deep reinforcement learning.

[0095] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the autonomous mobile robot path planning method based on deep reinforcement learning.

[0096] Another object of the present invention is to provide an information data processing terminal, which is used to implement the autonomous mobile robot path planning system based on deep reinforcement learning.

[0097] The present invention is specifically implemented:

[0098] In general, path planning in unknown dynamic environments is a model-free reinforcement learning (RL) problem. In complex and dynamic environments, only using fully connected layers to linearly fit the state will limit the model's expressiveness, prolong training time and affect convergence. Although the SAC algorithm balances exploration and utilization through the maximum entropy strategy, improves global search capabilities and path planning success rate, it still suffers from problems such as premature convergence, information loss and difficulty in coping with frequent state changes in complex environments. Therefore, the SAC algorithm is improved and combined with the PER strategy and gated attention mechanism. A path planning method of Gated Attention andPERbased SAC (GAP_SAC) is proposed. The overall framework of the method is as follows: Figure 3 .

[0099] The main contributions of the present invention are summarized as follows:

[0100] 1) In order to solve the problem that the agent has difficulty in fully perceiving the environment in a complex dynamic environment, the State space of SAC is expanded and adjusted to enhance the comprehensive perception of the unknown environment and more flexible all-round obstacle avoidance; a heuristic reward function is designed to achieve dynamic adjustment of the reward value, thereby optimizing the robot's learning process.

[0101] 2) In order to solve the problem of increased algorithm computational complexity and slow convergence speed in complex environments, we combined the Prioritized Experience Replay (PER) technology and the SumTree data structure to optimize sample utilization. The PER technology increases the weight of high-value samples, allowing the agent to learn effective strategies faster during training; the introduction of the SumTree data structure further improves the efficiency of PER.

[0102] 3) The agent only uses the FC layer to linearly fit the environment state, resulting in a weak model expression ability. An improved gated attention mechanism is introduced to dynamically adjust the attention allocation through a gated network and a transformation network to improve the agent's selection accuracy for key states and actions. It strengthens the focus on key environmental features.

[0103] In the training and generalization test experiments, the proposed method is verified to be capable and effective in avoiding obstacles in complex, dynamic and unstructured environments. The experimental results show that the GAP_SAC algorithm not only improves the success rate of path planning, but also shows good adaptability and robustness in the face of unknown and dynamic environments.

[0104] 1) State Space introduces four key metrics: G , D O , A O and M A . Where D G Represents the Euclidean distance between the Mobile Robot and the Goal point, which is an important indicator to measure whether the robot is close to the goal point. O represents the Euclidean distance between the Mobile Robot and the obstacle, which is crucial for collision avoidance and ensuring safe navigation. O When the distance is less than 0.5m, the agent will determine it as a potential collision risk. O Indicates the angle from the nearest obstacle, M A Indicates the angle that the robot needs to turn from its current position to the target point. The introduction of these two ensures that the robot can accurately avoid obstacles and maintain a high degree of alertness and adaptability in a dynamic and unpredictable environment. In summary, D G , D O , A O and M A Introducing it into the state space enables the mobile robot to obtain surrounding environment information more intelligently, thereby improving the ability of the mobile robot to perform path planning tasks.

[0105] 2) Action Space Modeling: In many previous studies, most of the work simplified and restricted the movement of mobile robots to eight fixed directions (a1, a2, ..., a8). Although this simplified motion model is effective in some scenarios, its applicability and performance may be limited in narrow and dynamic environments. Therefore, the angular velocity and linear velocity are controlled to achieve motion states from starting, stopping to linear motion and full-range steering. The robot's adaptability and navigation flexibility in complex environments are significantly improved. As shown in the figure. Compared with fixed-direction movement, omnidirectional drive provides higher maneuverability and positioning accuracy, especially in narrow or dynamically changing environments. On the basis of expanding the robot's range of motion, its performance and efficiency in performing complex path planning tasks are enhanced.

[0106] 3) Reward Function The reward function is crucial in reinforcement learning (RL) algorithms and directly affects the learning efficiency and performance of the agent. In previous studies, the immediate reward is usually set to a positive value only when the agent reaches the target state, set to a negative value when it collides with an obstacle, and set to zero in all non-terminal states. Although this design is intuitive, it often leads to sparse rewards during training, which increases the difficulty of learning and convergence time. In order to overcome this problem, a more sophisticated and dynamic immediate reward mechanism is designed by drawing on the ideas of heuristic search methods. The value of the immediate reward is closely related to the distance and yaw angle between the current state of the agent and the target state. This design not only improves the richness and continuity of the reward, but also helps the agent perceive the environment more accurately, thereby making more effective decisions. The compound reward function can be expressed as

[0107]

[0108] In the set reward function, R0 represents the target reward function, R1 is the yaw angle reward function, R2 is the distance reward function, R3 and R4 are the shortest distance obstacle penalty function and collision penalty function respectively. Together they form a multi-dimensional, dynamically adjusted reward mechanism. More specifically, R0 is set to the maximum positive reward. This is to motivate the mobile robot to reach the target point until the target distance D G is reduced to 0. On the other hand, R4 is set to the maximum negative reward to impose a penalty when the robot collides.

[0109] The yaw angle reward function R1 calculates the reward value based on the angle between the current forward direction of the mobile robot and the target point position. Among them, α (α∈[-π,π]) is the radian representation of the angle between the robot's forward direction and the target position. The formula is expressed as:

[0110]

[0111] choose The constant in the reward function is to control the yaw angle to ensure that the mobile robot moves towards the target direction. λ1 is used to balance the size of the R1 reward to ensure that the robot receives enough incentives when it approaches the target direction.

[0112] The distance reward function R2 is used to evaluate the distance between the mobile robot and the target position. The expression of this function is:

[0113]

[0114] Among them, D K is calculated by calculating the robot's position (x k ,y k ) and the target position (x Goal,y Goal ), λ2 is a positive constant value used to control the reward value of the distance between time k and time k+1 relative to the target point. -1 is the size of the balanced reward. By dynamically adjusting the reward value, the blindness of the mobile robot in the process of environmental search can be effectively reduced.

[0115] In order to comprehensively evaluate the navigation performance of the mobile robot, the R1 yaw angle reward function is multiplied by the R2 distance reward function. This multiplication method can determine the positive or negative reward value based on the relative relationship between the robot's forward direction and the target position. When the robot's yaw angle is far away from the direction of the target position, the robot will receive a small negative reward even if the distance is close. Therefore, the product of R1 and R2 can not only determine the positive or negative reward, but also adjust the size of the reward according to the distance. This design strategy not only improves the accuracy and flexibility of the reward mechanism, but also helps the robot learn and adapt more effectively in complex environments.

[0116] The obstacle reward function R3 aims to reduce the risk of the mobile robot colliding with the edge of obstacles during movement, especially considering the width of the robot. The function is designed as follows:

[0117]

[0118] Here, λ3 represents negative feedback, which is used to impose penalties on the mobile robot when it approaches an obstacle. The minimum safe distance between the robot and the obstacle is set as Min obs (Min obs =0.2), with the obstacle center as the center and the radius as Min obs The area is defined as a safe restricted area. When the mobile robot enters this area, it will receive negative feedback λ3 to ensure that the robot maintains a safe distance from obstacles. The details are shown in the figure below. However, determining the appropriate negative feedback λ3 is another tricky problem. Too much negative feedback may excessively affect the robot's obstacle avoidance strategy, resulting in its inability to effectively navigate in narrow or dynamic environments. On the contrary, too little negative feedback may not be enough to prevent the robot from directly colliding with obstacles. Therefore, accurately adjusting the value of λ3 is crucial to achieving efficient obstacle avoidance behavior. According to multiple rounds of experimental studies, it was found that when λ3=R0 / 10e 2 (where e is a natural number), the algorithm exhibits the best obstacle avoidance performance.

[0119] 4) PrioritizedExperienceReply When a mobile robot explores an unknown environment, it often encounters a variety of uncertainties. The experience replay mechanism stores and reuses the agent's historical interaction data, including state, action, reward, and success, recorded as (s t ,a,r,st+1 ,d) to improve learning efficiency. However, traditional uniform sampling experience replay may not be enough to make full use of previous learning resources. In order to solve this problem. This study uses prioritized experience replay. Unlike uniform sampling, prioritized experience replay adds weight priority to the stored experience, so that more valuable experience is replayed more frequently. Prioritized experience replay usually gives priority to those very successful or extremely failed experience samples, which are of great significance to policy optimization and can accelerate the learning process of the agent. In reinforcement learning, TDerror indicates the degree to which the agent needs to learn. In the SAC algorithm, TD error refers to the difference between the true Qvalue and the estimated Qvalue, and its calculation formula is as follows. A larger TD error indicates that there is a large deviation between the current Q value estimate and the target value, which requires more attention and updating. By giving priority to these high TD error experiences, the agent can learn beneficial strategies faster and effectively avoid unfavorable actions.

[0120] |δ t |=|R t +γminQ π (s t+1 , a′)-Q(s,a)|

[0121] The SumTree data structure is used to implement prioritized experience replay. SumTree improves the efficiency of experience sampling by storing all experience samples on leaf nodes and accumulating the priority sum of child nodes on internal nodes. When sampling, the total priority sum is divided into n intervals, where n = sum(p) / batchsize. It is ensured that the experience in each interval has a chance to be selected. In addition, in order to prevent the network from overfitting, experience is extracted probabilistically to ensure that even experience with TD-errors of 0 can be extracted. The priority value of each experience is defined as:

[0122]

[0123] where δ i =(ξ i +ε ) a , ε is a small constant that ensures that all experiences have priority, and a is a tuning parameter that controls the degree of influence of TD error on priority. In this way, high-value experience samples can be used more effectively, accelerating the learning process of the agent and improving the overall performance of path planning.

[0124] 5) The SAC algorithm uses MLP as the basic network, such as Figure 4As shown. However, in dynamic and complex environments, when the state information in the complex environment is extracted for training through the policy network of the SAC reinforcement learning algorithm, the state information of the environment will change in real time. Since the structure of the MLP network is relatively simple, it only contains a fully connected layer (FC) and an activation function, so it can only perform a linear fit on the state of the environment. This makes the model less expressive. Using MLP as the action network of the SAC agent will limit the performance of the agent. For example, the training time will be longer, the performance after convergence is not ideal, and it is difficult to avoid collisions in complex and dynamic environments.

[0125] Gated Attention Mechanism was first used in the field of image processing

[31] . By introducing the Gated Channel Transformation (GCT) module, the gating mechanism and the normalization operation were combined to improve the performance of visual recognition tasks. Inspired by this, this paper modifies the Gated Attention Mechanism and integrates it into the network of the SAC algorithm. The attention mechanism can effectively extract key state information in complex environments, enabling the agent to more accurately identify and predict obstacles, the location of target points, and the orientation of the robot during path planning. The gating mechanism dynamically adjusts the flow of input features to ensure that the agent can flexibly adjust its strategy in different environments, thereby maintaining efficient decision-making capabilities in the face of ever-changing environments. Combining the two mechanisms and adding them to the SAC network helps optimize the training process and improve the performance of the agent.

[0126] In the action network of the SAC algorithm, dynamic environmental information changes frequently and key information needs to be extracted. Some state information has a significant impact on the reward results, so the action network needs to perform special processing on this information and give it more opportunities to express. The attention mechanism can extract key state information and improve the learning efficiency and decision reliability of the agent. During the long training process, the attention mechanism ensures that key information is highlighted. The gating mechanism can dynamically adjust the flow of input features according to the current environmental state and reduce attention to unimportant information. Combining the two can ensure that the intelligent agent can flexibly adjust its strategy in different environments to adapt to changes in the environment. As a result, the output results can integrate state information in any complex dynamic environment.

[0127] The update formula of the attention weight coefficient is shown in Eq.

[0128]

[0129] At the same time, in order to make the algorithm maintain better learning efficiency, the learning rate is set to 0.0008 at the beginning of the training process. Then the adaptive moment estimation (Adam) algorithm is used for optimization to achieve adaptive update of the learning rate. The Adam algorithm combines momentum to improve the stochastic gradient descent performance of the neural network and combines it with the method to prevent gradient explosion, which further enhances the prediction ability of the SAC algorithm. The parameter update formula of the Adam algorithm is as follows, where α is the learning rate, and are the first-order moment estimate and the second-order moment estimate of the gradient, and ε is a small constant to prevent division by zero. In this way, the stability and efficiency of the algorithm during training are ensured, and the learning effect and prediction ability of the SAC algorithm in complex dynamic environments are significantly improved.

[0130]

[0131] In addition, for the update of the target network, a soft update strategy is adopted. Specifically, the parameter update formula of the target network is as follows. target are the parameters of the target network, θ sourse is the source network parameter, and τ is the soft update coefficient. In this way, the target network parameters gradually approach the source network parameters, achieving smooth parameter updates and avoiding the instability caused by drastic parameter changes. This maintains higher stability and robustness in a dynamic environment and improves the performance of path planning.

[0132] θ target ←τθ source +(1-τ)θ target In short, the SAC algorithm combined with the gated attention mechanism ensures that key information will not be lost during long-term training, and the gate mechanism dynamically adjusts the flow of input features to reduce attention to unimportant information, so that the output results can integrate state information in any complex dynamic environment. This method not only improves the learning effect of the model, but also greatly improves the performance and decision-making ability of the intelligent agent.

[0133] like Figure 5 The specific implementation process of the gated attention mechanism is as follows: The input data of the network mainly includes two parts: 1) the environmental information received by the laser radar mounted on the mobile robot, 2) and the distance D between the current robot and the target point G , the distance D between the current robot and the obstacle O , the angle A between the current robot and the obstacle O , and M AThe robot's forward direction. This information is expanded and input into the FC layer to obtain the QKV vector. In the Attention module, QK is dot-producted to calculate the attention score, and then the score is scaled by the square root of the K dimension and the softmax function is used to obtain the attention weight. The attention weight is multiplied by the V vector to obtain the attention output. The attention weight is then applied to the transformed layer. The attention output is linearly transformed and normalized to improve the representation ability of the model, stabilize the training process of the model, speed up the convergence speed and enhance the generalization ability of the model. In the gated Gate layer, the changed features are weighted and reorganized, and the sigmoid function is used to compress the range to the range of (0,1). Finally, the transformed output is multiplied by the gated signal to obtain the final gated attention output. This mechanism effectively integrates state information from any complex dynamic environment, significantly improving the learning effect of the model and the performance of the intelligent agent.

[0134] Figure 6 As shown in the figure, the green one is the path planned by the GAP_SAC algorithm, the pink one is the path planned by the GA_SAC algorithm, the red one is the path planned by the P_SAC algorithm, the blue one is the path planned by the SAC algorithm, and the yellow one is the path planned by the TD3 algorithm. The black arrow is the path of the simulated moving obstacle.

[0135] The first simulation experiment was conducted in an unknown dynamic environment map. The robot's task is to find the optimal collision-free path from the starting point to the target point under the guidance of the reinforcement learning algorithm. The figure shows the planning trajectories of the five algorithms after 1200 training iterations at different target points. All algorithms successfully generated collision-free paths and completed the path planning task. In the Difficult scenario, the green path generated by our method has the lowest distance and number of steps, showing a clear advantage. The local curvature in the middle section shows that it can flexibly adjust the path to avoid collision when dynamic obstacles approach. Thanks to the design of our heuristic function R1, the algorithm can not only effectively avoid obstacles, but also quickly correct the path to obtain the maximum reward by minimizing the angle of the robot toward the target point. This shows the algorithm's ability to balance collision avoidance and path optimization. In contrast, the path generated by the GA_SAC algorithm performs moderately. The orange path generated by the P_SAC algorithm is the most conservative, avoiding major obstacles. Although the path is relatively straight, it has the longest detour distance. In the Normal and Easy scenarios, there are fewer obstacles, and the P_SAC algorithm performs well in terms of execution time and number of steps, which is consistent with its tendency to avoid obstacles and linearize the path area in complex scenarios. TABLE II Comparison of main evaluation indexes in training scenarios.

[0136]

[0137] As shown in Table 2, after 1200 rounds of training, in the Easy and Normal categories, the five algorithms can generate collision-free paths with a success rate of more than 65% in unknown environments. However, in the Complicated category, due to the lack of optimization of the PER strategy and gated attention mechanism. The SAC and TD3 algorithms performed poorly in environmental perception, with a success rate of less than 50%, so they were not included in the statistics. By comparing the indicators of the GAP_SAC algorithm and other algorithms in three categories, it can be concluded that the GAP_SAC algorithm performs well in execution success rate, execution time, path length and number of steps compared with other algorithms, especially in path planning success rate, which is nearly 30% higher than the SAC algorithm on average. In addition, the minimum collision distance D0 of GAP_SAC is 0.226m, 0.259m and 0.216m, all within a reasonable range, ensuring a safe distance without affecting path optimization. Based on the reward function assumption, when D0 < 0.2, it will be punished by obstacles. This shows that the gated attention mechanism of GAP_SAC effectively balances obstacle avoidance and optimal path selection. In terms of execution path length and number of steps, the P_SAC algorithm performs better in the Easy and Normal categories because these two types of scenes have short paths and few obstacles, and do not require enhanced feature extraction. Although the network complexity of GAP_SAC increases and the execution time is disadvantageous, it still performs better in global search, learning efficiency and convergence in complex scenes.

[0138] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0139] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A path planning method for an autonomous mobile robot based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Expand and adjust the SAC state space to fully perceive the unknown environment and flexibly avoid obstacles in all directions; dynamically adjust the reward value through the heuristic reward function; Step 2: Combine the Prioritized Experience Replay (PER) technique with the SumTree data structure to optimize sample utilization. The PER technique increases the weight of high-value samples, allowing the agent to learn effective strategies during training. Step 3: Dynamically adjust attention allocation through a gating network and a transformation network to improve the agent's selection accuracy for key states and actions and strengthen attention to key environmental features.

2. The method for autonomous robot path planning based on deep reinforcement learning as claimed in claim 1, characterized in that: The method for expanding and adjusting the State space of SAC is as follows: 1) State Space introduces four key metrics: G , D O , A O and M A ; where D G represents the Euclidean distance between the Mobile Robot and the Goal point, D O represents the Euclidean distance between the Mobile Robot and the obstacle, which is crucial for collision avoidance and ensuring safe navigation. O When the distance is less than 0.5m, the agent will determine it as a potential collision risk; A O Indicates the angle from the nearest obstacle, M A Indicates the angle the robot needs to turn from its current position to the target point; 2) Action Space modeling.

3. The method for autonomous robot path planning based on deep reinforcement learning as claimed in claim 2, characterized in that: Action Space Modeling: In many previous studies, most of the work simplified and restricted the movement of mobile robots to eight fixed directions (a1, a2, ..., a8).

4. The method for autonomous mobile robot path planning based on deep reinforcement learning as claimed in claim 1, characterized in that: The reward function can be expressed as R0 represents the target reward function, R1 is the yaw angle reward function, R2 is the distance reward function, R3 and R4 are the shortest distance obstacle penalty function and collision penalty function respectively; together they constitute a multi-dimensional, dynamically adjusted reward mechanism; more specifically, R0 is set to the maximum positive reward; to motivate the mobile robot to reach the target point until the target distance D G On the other hand, R4 is set to the maximum negative reward to impose a penalty when the robot collides; The yaw angle reward function R1 calculates the reward value based on the angle between the current forward direction of the mobile robot and the target point position; where α (α∈[-π,π]) is the radian representation of the angle between the forward direction of the robot and the target position; the formula is expressed as: The reason why 3 / π is chosen as the constant in the reward function is to control the yaw angle and ensure that the mobile robot moves towards the target direction. λ1 is used to balance the size of the R1 reward and ensure that the robot gets enough incentive when it approaches the target direction. The distance reward function R2 is used to evaluate the distance between the mobile robot and the target position. The expression of this function is: Among them, D K is calculated by calculating the robot's position (x k ,y k ) and the target position (x Goal ,y Goal ), λ2 is a positive constant value used to control the reward value of the distance between time k and time k+1 relative to the target point; -1 is the size of the balance reward; The obstacle reward function R3 aims to reduce the risk of the mobile robot colliding with the edge of obstacles during movement, especially considering the width of the robot; the function is designed as follows: Here, λ3 represents negative feedback, which is used to impose penalties on the mobile robot when it approaches an obstacle; the minimum safe distance between the robot and the obstacle is set as Min obs (Min obs =0.2), with the obstacle center as the center and the radius as Min obs The area is defined as a safety restricted area; when the mobile robot enters this area, it will receive negative feedback λ3 to ensure that the robot maintains a safe distance from obstacles.

5. The method for autonomous robot path planning based on deep reinforcement learning as claimed in claim 1, characterized in that: The priority experience playback PER: |δ t |=|R t +γminQ π (s t+1 ,a′)-Q(s,a)| The SumTree data structure is used to implement prioritized experience replay.

6. The method for autonomous mobile robot path planning based on deep reinforcement learning as claimed in claim 5, characterized in that: The SumTree improves the efficiency of experience sampling by storing all experience samples on leaf nodes and accumulating the priority sum of child nodes on internal nodes. When sampling, the total priority sum is divided into n intervals, where n = sum(p) / batchsize. It ensures that the experience in each interval has a chance to be selected. In addition, in order to prevent the network from overfitting, experience is extracted probabilistically to ensure that even experience with TD-errors of 0 can be extracted. The priority value of each experience is defined as: where δ i =(ξ i +ε ) a , ε is a small constant that ensures that all experiences have priority, and a is a tuning parameter that controls the degree of influence of TD error on the priority.

7. A deep reinforcement learning-based autonomous mobile robot path planning system implementing the deep reinforcement learning-based autonomous mobile robot path planning method as described in any one of claims 1 to 6, characterized in that: The autonomous mobile robot path planning system based on deep reinforcement learning includes: The expansion module is used to expand and adjust the state space of SAC, fully perceive the unknown environment and flexibly avoid obstacles in all directions; dynamically adjust the reward value through the heuristic reward function; The optimization module is used to optimize sample utilization by combining the Prioritized Experience Replay (PER) technology and the SumTree data structure. The PER technology increases the weight of high-value samples, allowing the agent to learn effective strategies during training. The dynamic adjustment module is used to dynamically adjust attention allocation through a gating network and a transformation network to improve the agent's selection accuracy for key states and actions and strengthen attention to key environmental features.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the autonomous mobile robot path planning method based on deep reinforcement learning as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the autonomous mobile robot path planning method based on deep reinforcement learning as described in any one of claims 1 to 6.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the autonomous mobile robot path planning system based on deep reinforcement learning as described in claim 7.

Images