Mobile robot hierarchical path planning method and system based on deep reinforcement learning

Abstract

The invention provides a mobile robot hierarchical path planning method and system based on deep reinforcement learning, and the method comprises the steps: obtaining a two-dimensional map of a static environment where a mobile robot is located, planning a global path from a current position to a target point based on the map, and extracting sub-target points with key information from the global path according to the generated global path; performing local planning based on the global path and the sub-target points, including: constructing a comprehensive reward function, introducing a space attention mechanism for processing, and identifying a key area in the environment; and obtaining an optimal path through a path smoothing method according to a local planning result.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robot path planning, and particularly relates to a hierarchical path planning method and system for mobile robots based on deep reinforcement learning. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, autonomous mobile robots have the advantages of high efficiency, flexibility, intelligence, and autonomy, and can adapt to various complex environments and task requirements.

[0004] In the indoor field, such as mobile robots for accompanying the elderly, the mobile robot needs to autonomously plan a path according to the environmental layout to provide safe and convenient navigation services for the elderly. Mobile robot path planning refers to the robot automatically designing a shortest, safest, and collision-free path from the starting point to the ending point based on its own perception ability, while minimizing the time consumption as much as possible.

[0005] The requirements for the path planning of mobile robots for accompanying the elderly are different from those of ordinary indoor or outdoor, or inspection robots, mainly because:

[0006] The environment in which the elderly live is usually dynamic and complex, and there may be moving obstacles (such as furniture, pedestrians, pets, etc.), as well as sudden changes (such as temporarily placed items). Most of the existing robot path planning algorithms are based on the assumption of a static environment and are difficult to adapt to the dynamically changing environment in real time, resulting in easy collisions or path failures in complex scenarios.

[0007] Path planning can generally be divided into two categories: global path planning and local path planning. Global path planning, also known as offline path planning, calculates the optimal path from the starting point to the target by constructing a global map and using a search algorithm, and is applicable to situations where the environment is known and stable. However, with the dynamic change of the environment, especially when new obstacles appear, traditional global path planning often fails. In contrast, local path planning, also known as online path planning, is more adaptable to unknown or partially known environments, can sense surrounding obstacles in real time and dynamically adjust the path. Although local path planning has high flexibility and can respond to changes in real time in complex and dynamic environments, the path generated by it is usually only a local optimal solution and cannot guarantee the global optimum. Even in some cases, the target may not be reachable. Therefore, how to combine the advantages of global and local path planning is an important issue in current path planning research.

[0008] In addition, the means of combining global and local path planning in the prior art include: using the improved global path planning A* algorithm to plan the global path on the grid map; the improved global path planning A* algorithm is an algorithm that integrates node angle information and node distance information into the cost evaluation function of the global path planning A* algorithm; adopting a smoothing strategy to smooth the global path to obtain a smoothed global path; using a fusion algorithm to select the target of local path planning from the smoothed global path, so that the planned path is close to the smoothed global path, and completing the fusion of global path planning and local path planning.

[0009] Or: combining with a pre-constructed topological map, initially obtaining a global path, which is processed by cubic spline interpolation so that the smoothness of the path meets the vehicle dynamics requirements. Subsequently, referring to the global path, integrating high-precision positioning information and real-time obstacle detection results, using the DWA algorithm considering the global information weight during local path planning to complete real-time local dynamic path planning and decision-making. When the local planning decision algorithm finds no feasible route, update the topological map relationship and re-plan the global reference route so that the local planning algorithm can re-find the local optimal path.

[0010] It can be seen that traditional path planning methods usually combine global planning and local planning. First, determine the general route through global path planning, and then adjust the path through local planning during the actual execution process.

[0011] However, this method has some limitations when facing a dynamic environment with moving obstacles. Frequent re-planning will increase the computational load, resulting in a decrease in path efficiency, and may cause oscillations and detours, affecting the stable operation of the robot in a dynamic environment.

[0012] In recent years, with the development of deep learning, the online path planning problem has been significantly improved, promoting the rapid development of this field. Deep reinforcement learning (DRL), as one of the important technologies, can be mainly divided into value-based and policy-based methods. Policy-based methods directly learn the optimal policy to guide decision-making, and can handle high-dimensional action spaces. However, their disadvantage is that the learning process is usually slow and unstable, and it takes a long training time to converge. In contrast, value-based methods select the optimal action by learning the value function of actions. Their main advantage is relatively high computational efficiency, and they are usually more stable and converge faster during the training process.

[0013] Path planning methods based on deep learning can learn effective navigation strategies through continuous trial and error and feedback.

[0014] For example, the existing technical solutions for path planning based on deep learning include: Step 1: Construct a global map of the carrying area of the carrying robot Figure 3dimensional coordinate system to obtain globally Figure 3 coordinates of the traversable area in the dimensional coordinate system; Step 2: Obtain a training sample set; Step 3: Construct a global static path planning model for the transport robot; Step 4: Input the start and end coordinates in the transport task into the global static path planning model based on the fuzzy neural network to obtain the optimal planned path of the corresponding transport robot.

[0015] However, this method also faces some challenges. Since the reward signals are usually sparsely distributed throughout the environment, it is difficult for the mobile robot to obtain sufficient feedback at each stage to optimize its behavior, resulting in a decrease in the learning rate. On the other hand, in long-distance navigation, the target is often far away, and the exploration of the agent may become inefficient, making it difficult to find the correct target, further resulting in low training efficiency. Finally, due to the lack of guidance of global information, it is easy to fall into local optima. Summary of the Invention

[0016] To overcome the above deficiencies of the prior art, the present invention provides a hierarchical path planning method for mobile robots based on deep reinforcement learning, which is a hierarchical path planning algorithm to solve the problem of performing path planning tasks in an unknown environment.

[0017] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0018] In the first aspect, a hierarchical path planning method for mobile robots based on deep reinforcement learning is disclosed, including:

[0019] Obtain a two-dimensional map of the static environment where the mobile robot is located, plan a global path from the current position to the target point based on the map, and extract sub-goal points with key information from the generated global path;

[0020] Perform local planning based on the global path and sub-goal points, including: constructing a comprehensive reward function and introducing a spatial attention mechanism for processing to identify key areas in the environment;

[0021] Obtain the optimal path through the path smoothing method for the local planning result.

[0022] As a further technical solution, after planning the global path from the current position to the target point, a density function is introduced to evaluate the static obstacle distribution in the global area.

[0023] As a further technical solution, during the process of local planning, the three-dimensional environment where the mobile robot is located is modeled as an MDP, and the MDP is represented as a quadruple M = [S, A, P, R], where S represents the state space composed of the current position of the agent, the target position, and the distribution information of obstacles, A represents the actions executed by the agent, P represents the probability of transferring to the next state s' after selecting action a in state s, and R represents the immediate reward obtained by the agent after executing a certain action.

[0024] As a further technical solution, the three-dimensional environment information of the mobile robot is mapped to a two-dimensional grid, and the two-dimensional grid adopts a three-channel representation method: the first channel represents the self-position, that is, the position of the agent in the grid; the second channel represents the obstacle position, which is used to mark the area occupied by obstacles in the grid; the third channel is the sub-goal position of the global planning module, that is, the sub-goal position obtained from the global path planning module.

[0025] As a further technical solution, for the global path map, a weighted spatial attention map is generated by weighting the spatial importance of different regions.

[0026] As a further technical solution, the constructed comprehensive reward function is:

[0027] R = r1 + r2 + r3 + r4 + r5

[0028] Where r1 is the target reward function, r2 is the key point reward function, r3 is the obstacle, r4 is the boundary reward function, and r5 is the step reward function.

[0029] In the second aspect, a hierarchical path planning system for a mobile robot based on deep reinforcement learning is disclosed, including:

[0030] A global path planning module, configured to: obtain a two-dimensional map of the static environment where the mobile robot is located, plan a global path from the current position to the target point based on this map, and extract sub-goal points with key information from the generated global path;

[0031] A local planning module, configured to: perform local planning based on the global path and sub-goal points, including: constructing a comprehensive reward function, and introducing a spatial attention mechanism for processing to identify key regions in the environment;

[0032] Obtain the optimal path through the path smoothing method for the local planning result.

[0033] The above one or more technical solutions have the following beneficial effects:

[0034] The technical solution of the present invention proposes a hierarchical path planning method based on deep reinforcement learning, which solves the local optimum problem by adding a global planning module and simplifies the complex planning tasks. In addition, an attention mechanism based on global information is added, and spatial weighting is performed through global information to help the agent better learn key features.

[0035] The technical solution of the present invention designs a new comprehensive reward function. The addition of intensive rewards and global information enables the model to converge quickly and the training to be stable.

[0036] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. Brief Description of the Drawings

[0037] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0038] Figure 1 It is a schematic diagram of the overall path planning of the embodiment of the present invention;

[0039] Figure 2 It is a diagram of the key point positions of the embodiment of the present invention;

[0040] Figure 3 It is a state space diagram of the embodiment of the present invention;

[0041] Figure 4 It is an action space diagram designed for the embodiment of the present invention;

[0042] Figure 5 It is a neural network structure diagram of the current network and the target network. Detailed Description of the Specific Embodiments

[0043] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further explanations of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0045] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0046] In order to better understand the innovation of the technical solution of the present application, the traditional path planning method and the learning-based method are respectively explained.

[0047] Traditional path planning methods mainly include classical algorithms and bionic algorithms. Representative methods in classical algorithms include the A* algorithm, the Dynamic Window Approach (DWA), the Artificial Potential Field (APF) method, and the Rapidly-exploring Random Tree (RRT). Sampling-based algorithms are suitable for high-dimensional, complex, and dynamic environments and can handle complex obstacles, but the paths may not be optimal; graph-based algorithms are suitable for low-dimensional and static environments and can provide accurate optimal paths, but have high computational complexity in high-dimensional spaces. Bionic algorithms are algorithms designed inspired by the behaviors of organisms in nature, such as the ant colony algorithm, the genetic algorithm, the Particle Swarm Optimization (PSO) algorithm, etc. They solve optimization problems by simulating the behaviors or evolutionary processes of organisms. Compared with classical algorithms, bionic algorithms have less computational effort and have obvious advantages in dealing with complex environments. At the same time, bionic algorithms are prone to falling into local optimal solutions, it is difficult to guarantee global optimality, and they are relatively strongly dependent on the environment. Among these algorithms, some can be used for both global planning and local optimization. Many studies have attempted to combine global and local planning, using global algorithms to ensure the long-term optimality of paths, and at the same time making quick responses and fine-tuning in dynamic environments through local algorithms, so as to improve the adaptability and efficiency of the system. By taking the key nodes in global path planning as sub-goal points and combining the A* algorithm for global path planning and the DWA algorithm for local dynamic adjustment, it is possible to respond to changes in dynamic environments while ensuring global optimality. A hybrid technique has been proposed by using the A* algorithm as the global path planner and integrating the artificial potential field method as the local or reactive planner. In the work of these people, although the combination of global planning and local planning can improve the robot's navigation ability and effectively avoid dynamic obstacles, it still faces challenges such as path conflicts, large computational amounts for path evaluation and selection, insufficient handling of dynamic obstacles, and target deviation.

[0048] Learning-based path planning methods have received extensive attention in the field of robot navigation in recent years, especially in applications in complex and dynamic environments. Different from traditional model-based path planning methods, learning-based path planning methods obtain environmental features, obstacle information, and optimal decision-making strategies from a large amount of data through learning. These methods usually include technologies such as reinforcement learning and deep learning. Through the interaction between the agent and the environment, the path selection strategy is continuously optimized to achieve more flexible and efficient navigation. In the field of deep reinforcement learning, some work has been done by predecessors. Aiming at the problem of too slow convergence speed, a method of initializing the Q-table by adding prior knowledge is proposed. Based on DDQN, a comprehensive reward function is designed, and an adaptive greedy behavior selection strategy is proposed to optimize the trade-off between exploration and exploitation of the deep neural network. A new method based on deep distributed reinforcement learning is used, and a reward function provided by heuristic knowledge is introduced to accelerate the learning process of the agent in complex environments and improve decision-making efficiency. To alleviate the local optimality problem and the problems of scarce rewards and slow training speed in reinforcement learning algorithms, maximum entropy and HER algorithms are added to the SAC algorithm.

[0049] When facing complex path planning tasks, improving based on a single algorithm alone may have limited effects, while better results can often be achieved by combining different methods. The integration of multiple strategies can more effectively address the challenges in the task and improve the overall learning efficiency and performance. A globally guided reinforcement learning method is introduced. By inputting global information into the network model, it guides the agent to select the optimal global action, and a reward function is used to guide the agent to approach the global path point. However, the lack of an objective reward function may cause slow convergence. A hierarchical reinforcement learning method is proposed, which divides local planning into a high-level policy selector and multiple low-level policies. The high level is responsible for outputting options, and the low-level policies are responsible for generating actions for interacting with the environment. During the learning process of the low-level policies, by maximizing entropy and reducing the similarity between options, the diversity and independence of the behavior strategy are achieved. It is proposed to use the optimized A* algorithm to select several key points, and the SFVO algorithm and evaluation function are used to determine the specific actions of the agent. According to the evaluation results, decide whether to re-execute the A* algorithm or update the key point sequence.

[0050] When combining traditional optimization methods with deep learning, it can effectively guide the agent to perform more efficient training and complete complex tasks. However, improper selection of key points and the design of the comprehensive reward function often affect the learning efficiency, and may even cause the agent to overly focus on local goals and ignore the achievement of global goals. Therefore, how to correctly select representative key points and design a reasonable comprehensive reward function for them has become the key to improving learning efficiency and the convergence speed of the agent. Therefore, we propose a new method that selects key points through a density function and combines a comprehensive reward mechanism to help the agent converge in a shorter time, thereby achieving better learning results.

[0051] Embodiment 1

[0052] This embodiment discloses a hierarchical path planning method for a mobile robot based on deep reinforcement learning, including: first, establishing a 3D point cloud map through visual SLAM, then projecting it onto a two-dimensional grid map, and obtaining global path points using a global planning module. The above information is used for the training and input of the deep reinforcement learning model. Among them, the global planning module selects key points through a density function, and the local planning module receives key point information and local observation information and outputs the action Q value.

[0053] As Figure 1 shown, the path planning algorithm of this embodiment realizes navigation in a complex environment through a hierarchical strategy. The data corresponding to the environment is an indoor two-dimensional grid map, and the A* algorithm is the path planning algorithm. A path is pre-planned from the starting point to the ending point, and the output is the planned path points. The density function is used to select key points. The input is the global path points and obstacles, and the output is the filtered global path points (key points). The observation input is the two-dimensional grid map within the local field of view, generated by lidar scanning. The attention input is the global path points and the observation, and the output is the observation with spatial weighting at the global path points. The processing process is spatial feature weighting. The action is the moving direction of the mobile robot, and the direction with the largest Q value is selected. The new observation is the new observation for reaching the next state.

[0054] The specific steps include:

[0055] Step 1: The global planning module efficiently determines the global path point information using the A* algorithm, and this information is then transmitted as key input to the local planning module.

[0056] Step 2: The deep reinforcement learning model of the local planning module uses the attention mechanism to accurately focus on the global path point information in the environment, and at the same time processes sequential information with the long short-term memory network to effectively cope with the dynamic changes of the environment.

[0057] Step 3: In the training phase, the deep reinforcement learning model continuously optimizes its own parameters based on the high-level path point information and the reward feedback obtained through interacting with the environment, so as to output the optimal action strategy.

[0058] In Step 1, the global planning module mainly provides the key points in the global path, which is divided into two steps.

[0059] First, the mobile robot obtains a point cloud map through SLAM scanning. The point cloud map is projected into a two-dimensional grid map, which is used to reflect the distribution information of obstacles in the environment.

[0060] The global path from the current position to the target point is planned through the A* algorithm. This path takes into account the static obstacle information and provides prior information for the agent to avoid aimless exploration during environment initialization.

[0061] Then, according to the generated global path, sub-goal points with key information are extracted from it to reduce the use of redundant path points, thereby optimizing the path planning and improving the learning efficiency.

[0062] Under the two-dimensional map model of the known static environment, that is, the two-dimensional grid map, the A* algorithm can quickly calculate a global path from the starting point to the ending point.

[0063] However, in the absence of environmental prior information, simply following the global path may lead to collisions with obstacles. To solve this problem, a density function is introduced to evaluate the static obstacle distribution in the global area. After the evaluation, valuable key points are selected from the global path points and used to set positive rewards in the subsequent reward function. Suppose a point in the path is denotes the position coordinate of one of the global path points, and the obstacle set is O = {o1, o2, …, o n}, and the obstacle denotes the position coordinate of the obstacle in the two-dimensional grid map, the number of obstacles is |O|, and the distance of each obstacle can be calculated by the Euclidean distance formula:

[0064]

[0065] where d(p i , o j ) is the Euclidean distance from any point p i in the path to the obstacle o j . Next, it is judged whether there is an obstacle within the given radius r:

[0066]

[0067] Among them, r is reasonably set to 2 according to the size of the environment. Therefore, the midpoint p of the path i The density ρ(p i ) of the obstacles around

[0068]

[0069] This represents the number of obstacles around the path point p i The calculation method is the sum of all obstacles whose distance to p i is less than or equal to the radius r.

[0070] For each point p in the path i ∈Path, calculate its obstacle density ρ(p i ), and compare it with the threshold θ. If ρ(p i , r) ≥ θ. θ is also set to 2 considering the size of the environment. The formula for sub-goal selection is expressed as:

[0071]

[0072] As Figure 2 shown, this function takes the distribution density of obstacles in the global area as the core, and evaluates the potential value of each path point by the obstacle density around each point on the global path. In areas with high obstacle density, these areas usually contain complex obstacles, and the agent needs to pay special attention to avoid collisions in these places. Therefore, it is very valuable to select areas with high obstacle density as key points. Set a threshold according to the size of the two-dimensional environment. If it is greater, it is determined as a key point. By using these high-density areas as key points, it can help the agent better avoid obstacles during path planning and reduce the probability of accidental collisions. These key points actually play a guiding role, and the agent will give priority to these points to ensure that it can pass through complex obstacle areas smoothly. Among them, Figure 2 is the key point position map.

[0073] In the open area with low obstacle density, compared with the area with dense obstacles, there are fewer obstacles, and the agent can move relatively freely. In this area, the generation of path points is usually more random because the distribution of obstacles is sparse, the movement space of the agent is large, and the complexity of path planning is low. Therefore, the path points generated in these open areas can, to a certain extent, ignore the influence of obstacles and choose paths more flexibly. In this way, combining the global path and local density information not only improves the safety of navigation but also increases the flexibility of path planning.

[0074] Step 2: Local Planning Module. The intelligent agent path planning problem is first reformulated as a Markov Decision Process (MDP) model, where the MDP mainly consists of a state space, an action space, and a reward function. To further improve the path planning effect, a comprehensive reward function is proposed, and a spatial attention mechanism is introduced to more accurately identify key regions in the environment. Finally, the optimal path is obtained through a path smoothing method.

[0075] Problem Definition: Consider the path planning task of an intelligent agent on a two-dimensional grid map. The goal of the intelligent agent is to reach the target position from the starting point, avoiding obstacles on the path and minimizing the path length as much as possible. The entire environment corresponding two-dimensional grid map can be modeled as a Markov Decision Process MDP for model training, and the optimal policy is found by maximizing the cumulative reward. The MDP can be represented as a quadruple M = [S, A, P, R], where S represents the state space composed of information such as the current position of the intelligent agent, the target position, and the distribution of obstacles, A represents the actions executed by the agent, P represents the probability of transferring to the next state s′ after selecting action a in state s, and R represents the immediate reward obtained by the intelligent agent after executing a certain action.

[0076] State Space: Mapping information in a three-dimensional environment to a two-dimensional grid is a common spatial representation method, aiming to simplify calculations and analysis. The two-dimensional grid can provide a more efficient state space representation. The three-dimensional environment is the point cloud information of the entire indoor environment, and the local observation is the information of the current observation, which is a part of the entire environment.

[0077] To effectively process this spatial information, the two-dimensional grid adopts a three-channel representation. As the input of the deep reinforcement learning model, it contains its own position, environmental information, and global path point information. The first channel represents its own position, that is, the position of the intelligent agent in the grid; the second channel represents the obstacle position, which is used to mark the areas occupied by obstacles in the grid to help the planning algorithm avoid collisions; the third channel is the sub-goal position of the global planning module, that is, the sub-goal position obtained from the global path planning module. These sub-goal points serve as a guide for navigation, driving the system to gradually move from the starting point to the end point. Through this hierarchical structure, not only can the spatial characteristics of the environment be retained, but also the three-dimensional space can be simplified into a more manageable two-dimensional representation. The size of the local field of view (FOV) is H×W, where its own position corresponds to the central position of the local field of view. After being processed by the LSTM network, the time step is Nt, and the composition form of the input state is Nt×3×H×W, where Nt is the time step, 3 is the 3 channels, and H and W respectively represent the size of each channel. The state is as Figure 3 shown. The left side is the model input state, and the right side is the input state of each time step.

[0078] Since the scale of the grid is much smaller than the turning radius of the mobile robot, the action space of the agent can be simplified and represented as 8 basic actions. The A* algorithm also uses the same 8 directions, as Figure 4 shown, numbered from 0 - 7. Specifically, this set of 8 basic actions A = {up, down, left, right, upper left, lower left, upper right, lower right}, covering the possible movement directions of the robot in the two-dimensional grid. Assume the current position of the robot is (x, y), then each action can be represented as an increment (Δx, Δy) to the current position, where Δx and Δy represent the increment values in the horizontal and vertical directions respectively. Since in the discrete environment, the movement unit is 1 cell. Then the new position (x ′ , y ′ ) of the robot after executing this action can be calculated by the following formula:

[0079] (x ′ , y ′ ) = (x + Δx, y + Δy).

[0080] Network structure: Due to the dynamic changes of obstacles in the environment, traditional neural network structures often cannot respond to these changes in a timely manner. Therefore, LSTM is needed to extract information in the time dimension to help the robot predict the change trend of obstacles. The input of LSTM is the features after spatial attention processing of 3 time-step information. LSTM is a special type of recurrent neural network (RNN) that processes time series data. The output is given to the fully connected layer.

[0081] Meanwhile, considering that the input data has a spatial structure, a spatial attention weighting method combined with the global path map is proposed. The network structure diagram is as Figure 5 shown. In this method, the global path map is input into the spatial attention module, and by weighting the spatial importance of different regions, a weighted spatial attention map is generated. This map then acts on the state input of the local planning module, improving the accuracy and efficiency of local planning. The three-channel features of local observations are used as input, and by combining time series information and spatial attention weighting, it can better handle the uncertainty in the dynamic environment and improve the robustness of path planning.

[0082] First, assume the input feature map is represented as where C is the number of channels, H is the height, and W is the width. The input feature map is local observation information, containing three channels. A simplified form of the spatial attention mechanism is used, where the weights of certain regions are enhanced under specific conditions. First, define A as the attention weight, and its elements are set according to the value of X:

[0083]

[0084] where i and j represent the coordinates of each element position in the global map. Finally, perform an element-wise dot product on A and X, then the result R can be expressed as:

[0085] R(i,j) = A(i,j) × X(i,j)

[0086] Among them, the result R is the input to the LSTM network after spatial weighting.

[0087] Regarding the comprehensive reward function: Traditional reward functions are often defined by setting a positive reward for reaching the target point, a negative reward for hitting an obstacle, and zero for other situations. This way of setting rewards may lead to the problem of sparse rewards, that is, during the training process, the agent can only receive meaningful rewards in very few time steps, making the learning process very slow and difficult to effectively explore the environment.

[0088] To solve this problem, a comprehensive reward function is introduced. This function not only retains the original positive and negative reward settings but also introduces continuous reward signals based on factors such as the distance between the agent and the target and the state change, making the reward space smoother. In addition, key-point rewards provided by the global planning module are added to improve the performance and convergence speed of the model. The comprehensive reward function can be expressed as:

[0089] R = r1 + r2 + r3 + r4 + r5

[0090] where r1 is the target reward function, r2 is the key-point reward function, r3 is the obstacle, r4 is the boundary reward function, and r5 is the step reward function.

[0091] Let P prev be the previous position of the agent, P new be the current position of the agent, and G be the target position. Then calculate the distances between the previous position and the current position and the target position through the Manhattan distance. The formula is as follows:

[0092] d prev = Manhattan(P prev ,G) = |x prev -x goal | + |y prev -y goal |

[0093] d new = Manhattan(P new ,G) = |x new -x goal | + |y new -y goal |

[0094] x previs the abscissa of the previous position, x new is the abscissa of the current position, x goal is the abscissa of the target position. y is the ordinate.

[0095] r q The expression is as follows. Rewards are set according to the relative position relationship between the agent and the end point, and specifically divided into two cases. First, when the agent reaches the end point, a relatively large positive reward is given; second, within a specific area near the end point, a relatively small positive reward is given. Such a reward design aims to motivate the agent to maintain the momentum to move forward when approaching the end point. By providing small positive rewards when approaching the target area, the agent can obtain continuous feedback, reduce the possibility of deviating from the target path, and at the same time enhance its approach to the target.

[0096]

[0097] The expression of r2 is as follows. Through the global information provided by the global planning module, the agent can obtain important key points and gradually approach the end point along the optimal path. Positive rewards are set for each key point to motivate the agent to act according to the global path planning, thereby reducing ineffective exploration and improving learning efficiency.

[0098]

[0099] Here, (x, y) represents the current position coordinates, (x s , y s ) is the position of the sub-goal point.

[0100] r3 is the boundary penalty function, which aims to encourage the agent to conduct reasonable exploration in the simulation environment and prevent collisions with walls or boundary areas, thereby avoiding the agent staying near the boundary or showing unreasonable behaviors. The formula is given by:

[0101]

[0102] r4, as the obstacle reward function, once the agent collides with an obstacle in the environment, a relatively large negative reward will be immediately given to the agent, so as to strongly warn the agent to avoid obstacles and prompt it to give priority to obstacle avoidance strategies in the path planning process. The formula is given by:

[0103]

[0104] Here, (x, y) represents the current position coordinates, x O , y O is the position of the obstacle.

[0105] r5 is the distance reward function. The step reward function is used to penalize the redundant steps of the agent and encourage it to reduce unnecessary movements. By reducing the number of steps, the agent can find the shortest path and thus complete the task more efficiently. The formula is given by:

[0106]

[0107] where n is the number of steps taken by the agent. If the agent does not move, the reward is 0.

[0108] Local planning model training: The entire path planning process is shown in Algorithm 1. The local path planning of the robot is formulated as an MDP problem. Secondly, the agent perceives the environmental information through sensors and selects either a completely random behavior or the optimal behavior under the current network policy according to the ε-greedy strategy. Initially, the learning rate is set to 0.001 and is dynamically adjusted during training through the Adaptive Moment Estimation algorithm (Adam) to optimize the convergence speed and stability of the model. As the training progresses, the greedy factor ε gradually decays, that is, by setting a decay factor, the agent increasingly exploits behaviors and reduces random exploration. If the agent reaches the target point, the termination reward is calculated according to the target achievement situation and the current episode ends. As the training progresses, the greedy factor ε continuously decays, thereby gradually reducing the exploration frequency and turning to a more efficient exploitation strategy, ultimately achieving optimal path planning, as shown in Table 1.

[0109] Table 1

[0110]

[0111]

[0112] Path smoothing: The output actions of the H-DDQN algorithm are discrete, while in actual situations, the turning operations of the agent are continuous. To better make the planned path more in line with the actual requirements of the agent, Bezier curves need to be used to smooth the planned path because it has the advantages of low programming difficulty, low computational cost, and strong trajectory continuity. Therefore, Bezier curves are introduced into the proposed algorithm framework to achieve end-to-end smooth path planning. Therefore, multiple second-order Bezier curves are concatenated to smooth the turning points of the path, and the calculation expression is formed as:

[0113]

[0114] The global planning information is used as the input to the local planning deep reinforcement learning model. The local planning outputs discrete actions and then performs smoothing processing.

[0115] Embodiment 2

[0116] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0117] Embodiment III

[0118] The purpose of this embodiment is to provide a computer-readable storage medium.

[0119] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above method are executed.

[0120] Embodiment IV

[0121] The purpose of this embodiment is to provide a hierarchical path planning system for a mobile robot based on deep reinforcement learning, including:

[0122] A global path planning module, configured to: obtain a two-dimensional map of the static environment where the mobile robot is located, plan a global path from the current position to the target point based on the map, and extract sub-target points with key information from the generated global path;

[0123] A local planning module, configured to: perform local planning based on the global path and sub-target points, including: constructing a comprehensive reward function and introducing a spatial attention mechanism for processing to identify key regions in the environment;

[0124] Obtain an optimal path through a path smoothing method for the local planning result.

[0125] Embodiment V

[0126] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any one of the above embodiments.

[0127] The steps involved in the devices of the above embodiments correspond to those of Method Embodiment I. For specific implementation manners, reference may be made to the relevant description part of Embodiment I. The term "computer-readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0128] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0129] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A mobile robot hierarchical path planning method based on deep reinforcement learning, characterized by: include: Obtain a two-dimensional map of the static environment in which the mobile robot is located, plan a global path from the current position to the target point based on the map, and extract sub-target points with key information from the generated global path; Perform local planning based on the global path and sub-target points, including: constructing a comprehensive reward function and introducing a spatial attention mechanism to process and identify key areas in the environment; The optimal path is obtained based on the local planning results through path smoothing method.

2. The hierarchical path planning method for a mobile robot based on deep reinforcement learning as described in claim 1 is characterized in that after planning the global path from the current position to the target point, a density function is introduced to evaluate the static obstacle distribution in the global area.

3. The mobile robot hierarchical path planning method based on deep reinforcement learning as claimed in claim 1, characterized in that: In the local planning process, the three-dimensional environment in which the mobile robot is located is modeled as an MDP, and the MDP is represented by a four-tuple M=[S, A, P, R], where S represents the state space composed of the current position of the agent, the target position, and the distribution information of obstacles, A represents the action performed by the agent, P represents the probability of transferring to the next state s′ after selecting action a in state s, and R represents the immediate reward obtained by the agent after performing a certain action.

4. The mobile robot hierarchical path planning method based on deep reinforcement learning as claimed in claim 1, characterized in that: The three-dimensional environment information of the mobile robot is mapped to a two-dimensional grid, which is represented by three channels: the first channel represents its own position, that is, the position of the agent in the grid; the second channel represents the obstacle position, which is used to mark the area occupied by obstacles in the grid; the third channel is the sub-target position of the global planning module, that is, the sub-target position obtained from the global path planning module.

5. The mobile robot hierarchical path planning method based on deep reinforcement learning as claimed in claim 1, characterized in that: For the global path map, a weighted spatial attention map is generated by weighting the spatial importance of different regions.

6. The mobile robot hierarchical path planning method based on deep reinforcement learning as claimed in claim 1, characterized in that: The constructed comprehensive reward function is: R＝r1+r2+r3+r4+r5 Where r1 is the target reward function, r2 is the key point reward function, r3 is the obstacle, r4 is the boundary reward function, and r5 is the step reward function.

7. A mobile robot hierarchical path planning system based on deep reinforcement learning, characterized by: include: The global path planning module is configured to: obtain a two-dimensional map of the static environment in which the mobile robot is located, plan a global path from the current position to the target point based on the map, and extract sub-target points with key information from the generated global path; The local planning module is configured to: perform local planning based on the global path and sub-target points, including: constructing a comprehensive reward function and introducing a spatial attention mechanism for processing to identify key areas in the environment; The optimal path is obtained based on the local planning results through path smoothing method.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are performed.

Images