Global-local collaborative trajectory planning method based on deep reinforcement learning

Abstract

The invention discloses a global-local collaborative trajectory planning method based on deep reinforcement learning, and the method comprises the following steps: starting to move, and randomly selecting whether to carry out the re-planning or not and the dynamic weight parameter of dwa when the number of forward steps is less than 200; and after the step number is greater than 200, the S is transmitted into the event network to select the current action. And the state S'of the next step is further obtained in the environment after the action is selected. And calculating an R value according to S and S ', and storing [S, A, R, S'] into a memory bank. [S, A, R, S '] is extracted from a memory bank for training, and learning is carried out once every five steps. According to the method, an autonomous strategy space is constructed through deep reinforcement learning, the static environment hypothesis of a traditional A * algorithm is broken through, multi-target collaborative optimization is carried out, the global re-planning decision-making capability for coping with huge unknown obstacles is formed, and the path smoothness and the environment dynamic adaptability are remarkably improved. Experimental results show that the path smoothness around the large obstacle is improved, and the path length is shortened.

Description

Technical Field

[0001] The present invention relates to a global-local collaborative trajectory planning method based on deep reinforcement learning, and belongs to the field of robotics and artificial intelligence technology. Background Art

[0002] With the continuous development of robotics and artificial intelligence technologies, path planning has become a highly sought-after research area in the field of mobile robotics. The methods for evaluating the quality of paths generated by path planning algorithms vary depending on the type of robot and the application domain. Some methods focus on planning efficiency, i.e., finding the shortest path within a limited time; others on safety, i.e., minimizing the probability of collisions with obstacles; and still others on energy consumption, i.e., focusing on the algorithm's computational complexity and model size. To address these needs, researchers have undertaken extensive research, which can be broadly categorized into two main categories: global path planning and local path planning.

[0003] Global path planning aims to find the shortest path between a starting point and a destination while avoiding known obstacles. Common algorithms include RRT, ACO, artificial potential field, and A*. Currently, RRT is highly efficient, but often fails to find the shortest path. ACO converges slowly and is prone to getting stuck in local optima, preventing the robot from reaching its destination. Artificial potential field is simple, but prone to local optima and oscillation. A* is efficient and flexible, making it an ideal choice for global planning, but its disadvantage is its high algorithmic overhead. Recent optimizations, such as combining bidirectional jump point search (JPS) with A*, have significantly reduced algorithmic overhead. This reduced algorithmic overhead allows A* to replan the global path. Local path planning involves dynamic path planning to avoid obstacles within a certain range as the robot moves along the global path. Various algorithms have been developed, including VFH, LCM, and DWA. However, existing algorithms can only perform global planning in advance. When encountering unforeseen obstacles, the static path becomes ineffective, resulting in inefficient detours for local avoidance. Summary of the Invention

[0004] The present invention aims to address the shortcomings and deficiencies of the above-mentioned prior art and proposes a global-local collaborative trajectory planning method based on deep reinforcement learning. This method constructs an autonomous strategy space through deep reinforcement learning, breaks through the static environment assumption of the traditional A* algorithm, and performs multi-objective collaborative optimization, forming a global replanning decision-making capability to cope with large unknown obstacles, significantly improving path smoothness and environmental dynamic adaptability. The neural network constructed by the present invention also optimizes the parameters of the local obstacle avoidance algorithm, adopts a maximum speed constraint in obstacle-free areas, and automatically switches to obstacle avoidance priority mode when the obstacle density exceeds a threshold. Experimental results show that the present invention improves path smoothness around large obstacles and shortens path length.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a global-local collaborative trajectory planning method based on deep reinforcement learning, which includes the following steps:

[0006] Step 1: Start the movement. When the number of steps forward is less than 200, randomly select whether to replan and the dynamic weight parameters of DWA.

[0007] Step 2: When the number of steps is greater than 200, pass S into the evaluate network to select the current action.

[0008] Step 3: After selecting the action, move forward in the environment to get the next state S'.

[0009] Step 4: Calculate the R value based on S and S', and then store [S, A, R, S'] in the memory bank.

[0010] Step 5: Extract [S, A, R, S'] from the memory bank for training, learning once every 5 steps.

[0011] Beneficial effects:

[0012] 1. This invention optimizes the traditional global static planning and local dynamic obstacle avoidance fusion algorithm to solve the problem of returning to the global route after detouring a certain distance around dynamic unknown huge obstacles. It uses reinforcement learning to perceive when the environment needs to replan the global path, achieving the results of optimizing the path, reducing time and increasing the success rate.

[0013] 2. The present invention can train robots to make correct decisions through reinforcement learning. By simply setting the reward function manually, it can cope with many complex situations, greatly reducing the workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram comparing the algorithm proposed in this invention with the general algorithm.

[0015] Figure 2 Schematic diagram of the smoothing process of the A* algorithm path.

[0016] Figure 3 Flow chart of the method of the present invention.

[0017] Figure 4 Schematic diagram of the experimental environment of the present invention.

[0018] Figure 5a A path is a diagram determined by a set of parameters.

[0019] Figure 5b The path is a schematic diagram obtained through dynamic programming of two sets of parameters.

[0020] Figure 6a Schematic diagram of obstacle avoidance trajectory without global backtracking line.

[0021] Figure 6b Schematic diagram of obstacle avoidance trajectory with global backtracking line.

[0022] Figure 7a A schematic diagram showing the speed comparison before and after adding global replanning based on reinforcement learning.

[0023] Figure 7b Schematic diagram of angular velocity comparison.

[0024] Figure 8 Schematic diagram comparing the reward values ​​of dual neural network training and single neural network training. DETAILED DESCRIPTION

[0025] The present invention will be described in further detail below with reference to the accompanying drawings.

[0026] like Figure 1 As shown, the algorithm of the present invention differs from conventional algorithms in that conventional algorithms only consider the location and movement path of obstacles, while the algorithm of the present invention also considers the shape and size of obstacles, thereby achieving global path planning for obstacle avoidance in extreme environments such as fires and earthquakes. However, when obstacles are large, unnecessary detours still occur, and the path length increases significantly. This significant increase in path length indicates that simply optimizing a local obstacle avoidance algorithm will not achieve the desired results. Therefore, for obstacle avoidance algorithms for large unknown obstacles, the present invention shifts the optimization perspective from a local to a global one, proposing a global path replanning algorithm.

[0027] This invention proposes a global path replanning method that combines the global path with local obstacle avoidance and incorporates reinforcement learning for decision-making environment identification. Simultaneously, the constructed neural network also participates in the dynamic parameter selection of local path planning to increase the success rate of obstacle avoidance. This allows for early identification of dynamic obstacles for prediction and dynamic obstacle avoidance, resolving the issues of large turning angles and lengthened paths associated with dynamic obstacle avoidance. It also allows for dynamic changes in local path planning parameters to increase the success rate of obstacle avoidance. Specifically, it includes the following:

[0028] In the algorithm proposed in the present invention, the A* algorithm is smoothed and then used for static global path planning.

[0029] The present invention first grids the obstacles on the map, and the travel path is also composed of a sequence of grids. The basic A* algorithm is as follows:

[0030] The traditional A* algorithm search principle is to add the starting point to the open list, add this point as a parent node to the close list, search for its adjacent reachable points and add them to the open list, calculate the cost of the nodes in the open list according to the evaluation function, select the node with the lowest cost as the next parent node and add it to the close list, and then search again for the lowest cost point that can be reached by this parent node until the target point is reached. The evaluation function is:

[0031] F(n)=G(n)+H(n) (1)

[0032] When traversing the grid, there are two traversal methods, 8-domain and 4-domain. Due to the need to consider the size of the main body, the present invention uses 4-domain, that is, traversing the grid in four directions: up, down, left, and right. When traversing the grid, two sets are divided into open sets and closed sets to store the traversed grids. Each traversal calculates the score function (1) of the grid, where G(n) is the length of the shortest path traversed to the grid. When traversing the grid in the open set repeatedly, if G(n) is less than the shortest path during the previous traversal, the current latest G(n) replaces the previous G(n). H(n) is the Euclidean distance between the grid and the target grid (2).

[0033]

[0034] After the calculation, all elements in the open set are traversed, and the one with the smallest F(n) is placed in the closed set, which is used as the center point for the next round of traversal. If the calculated adjacent node is already in the open list and the original cost function is smaller, the original parent node is used as the center point for the new traversal. Until the end point is also in the closed set, the parent node of each grid node is backtracked from the end point to form the final planned path.

[0035] A.DWA algorithm

[0036] The basic DWA algorithm is divided into two parts: the first is the sampling of the velocity space, and the second is the evaluation of the predicted trajectory after sampling. Among them, the limitations of velocity sampling are mainly divided into three categories. The first is the limitation of its own hardware conditions. m

[0037] V m ={(v,ω)|v∈[v min , v max ],ω∈[ω min ,ω max ]} (3)

[0038] Always min , v max are the minimum and maximum angular velocities, ω min ,ω max are the minimum and maximum angular velocities respectively. The second is the acceleration limit V d

[0039] V d ={(v,ω)|v∈[v c -a vmax Δt, v c +a vmax ·Δt],

[0040] ω∈[ω c -a wmax Δt,ω c -a wmax ·Δt]} (4)

[0041] Where Δt is the sampling time interval, and the maximum speed and angular velocity limit the sampling range. The third is the limitation of environmental obstacles V a

[0042]

[0043] (5) where dist(v,w) represents the maximum distance from the obstacle in the (v,w) state. Combining the above three constraints, the final sampling velocity space of the robot is

[0044] V s =V m ∩V d ∩V a (6)

[0045] After the sampling interval is determined, the sampling is uniformly performed at a certain sampling interval within the interval and the trajectory prediction is performed.

[0046]

[0047] (6) In the formula, k represents the number of iterations.

[0048] After sampling the speed, the predicted trajectory is evaluated and the optimal speed is selected as the output speed of the robot.

[0049] G(v,ω)=σ(α·heading(v,ω)+σ(β·dist(v,ω)+σ(γ·velocity(v,ω))(8)

[0050] G(v, ω) represents the evaluation function of the trajectory, where heading(v, ω) evaluates the error Δθ between the angle between the trajectory endpoint and the line connecting the target point. The larger the Δθ, the lower the score, so π-Δθ is used for evaluation. dist(v, ω) evaluates the distance between the simulated trajectory and the obstacle. When the distance is greater than a threshold, dist(v, ω) is a fixed value instead of infinity. Velocity(v, ω) is the evaluation function of speed. The faster the speed, the higher the evaluation function. α, β, and γ are the coefficients of the evaluation function, which can determine which condition has a greater weight, that is, which evaluation function score is more important to ensure the accuracy and effectiveness of path planning.

[0051] B. Improved A* and DWA algorithms

[0052] For the smoothing process of the A* algorithm, the method of the present invention is to optimize the redundant points by quadratic interpolation. Figure 2 For example, starting from point S, connect the nodes (n1, n2...n m ) until the minimum distance between the line connecting the two points and the obstacle does not meet the minimum distance for the robot to pass. m , n m-1 Find a distance n m The nearest point makes the line connecting it and the starting point far enough away from the obstacle, and then takes the point n closest to the obstacle on the line m+1 And use this point as the starting point for the next optimization of redundant points, and repeat the above steps until the end point G.

[0053] To optimize the DWA algorithm, a fully connected neural network is built to analyze the robot's environment and perform reinforcement learning. This allows the parameters of the DWA algorithm's scoring function to adapt to environmental changes. For example, when there are many obstacles, increasing the β parameter makes obstacle avoidance more precise, thereby increasing the success rate. When obstacles are sparse, the robot can travel at maximum speed, reducing obstacle avoidance time. The specific algorithm integrated with reinforcement learning is described in the next section.

[0054] A. Algorithm Flow

[0055] The basic elements of traditional reinforcement learning are environment, state, action, reward and strategy. In this algorithm, the environment is as follows: Figure 4As shown in the figure, after knowing the global static obstacles, dynamic unknown obstacles within a certain range are sensed. The current state matrix S is determined based on the relationship between the obstacles, the agent, and the target point. After obtaining S, an action A is randomly selected at the beginning of training. After a certain number of learning cycles, S is passed into the target neural network to obtain the q value of each action. The action corresponding to the highest q value is selected and input into the environment to obtain the next state S'.

[0056] Action A is [a1, a2], where both a1 and a2 can be 0 or 1. The value of a1 determines whether to perform global static re-planning, and the value of a2 determines which set of DWA parameters to use for local obstacle avoidance. After obtaining the next state S', it is compared with S to obtain the reward R for the current action. The [S, A, R, S'] obtained in the above steps is transferred to the memory bank, and the data in the memory bank is used for extraction and training during neural network learning.

[0057] After having the memory bank, N previously stored [S, A, R, S'] are extracted from the memory bank at regular intervals for training. Among them, S is passed to the evaluate MLP, and the output is the q value corresponding to A; S' is passed to the target MLP, and the output is the maximum q value of the four actions q.max. The q target value of the action A is calculated based on R and q.max, and the evaluate MLP parameters are updated by gradient descent. Finally, the parameters of the evaluate MLP are copied to the target MLP at regular intervals to make the two neural networks exactly the same. After learning is completed, the target MLP can be used

[0058] The MLP makes the decisions.

[0059] B. Environment Construction

[0060] This invention not only rasterizes the map to facilitate the A* global path planning algorithm, but also facilitates the quantification of obstacle coordinates and shapes. Quantifying environmental information and feeding it into the neural network is crucial for the DQN to identify environmental states and make decisions. The proposed algorithm considers obstacle density within a certain range, obstacle shape, and the degree of deviation from the global path to determine whether to replan the global path.

[0061] s = [n, angle, goal_angle] T (11)

[0062] The state matrix is ​​shown in (11). Since the present invention uses a fully connected neural network, there is no need to quantize the transition parameters and they can be directly passed into the neural network. Where n is the sum of the number of known static obstacles and unknown dynamic obstacles within a certain range.

[0063]

[0064] Formula (12) describes the degree of occlusion of the target point by an unknown dynamic obstacle.

[0065] θ i For the direction of progress After the position dynamic obstacle is rasterized, the angle between the line connecting one grid obstacle and the robot and the line connecting the next target point and the robot is obtained.

[0066] The scanning process is as follows: when the line connecting the next target point and the robot passes through the obstacle interval, the obstacles are scanned to both sides with the line as the center line; until the ray emitted from the robot's observation point does not pass through the obstacle interval, the angle of the line connecting the last obstacle and the robot is taken as θ i Because the degree of occlusion is greatest when the line connecting the target point and the robot passes through the center of a large obstacle, the closer the area is to the edge of the obstacle, the less occlusion there is. Therefore, the degree of occlusion is calculated by adding the two angles and subtracting their variance. When the unknown dynamic obstacles within a certain range of the robot's travel direction are small or highly dispersed, the angle value is smaller. Conversely, the larger the unknown dynamic obstacles, the greater the degree of occlusion on the travel path, and the larger the angle value.

[0067] goal_angle is the angle between the line connecting the next goal and the previous goal and the line connecting the next goal and the agent vertex. A larger value for goal_angle indicates that the agent has deviated further from the original planned path.

[0068] If a global static path is to be planned, unknown obstacles and known static obstacles within the sensing range are taken into consideration in the planning.

[0069] C. Definition of Reward Function

[0070] For the definition of the reward function, the following four aspects are mainly considered.

[0071] R=R1+R2+R3+R4+R5 (13)

[0072] Among them, R1 is related to reaching the target and collision.

[0073]

[0074] R2 is related to the degree of occlusion and its reduction after the decision is made.

[0075]

[0076] Among them, Δangle is the change in the next angle value after the decision is made. If it decreases, a positive reward is given; when a1 is 0, no replanning is performed, and when it is 1, replanning is performed. Replanning has a certain computational cost based on negative rewards.

[0077] R3 is related to the distance to the next target.

[0078]

[0079] R4 is related to the distance to the obstacle.

[0080]

[0081] R5 is related to the degree of deviation from the global path.

[0082] R5=-a1×20-Δgoal_angle (18)

[0083] When the deviation is small, no re-planning is required. When the deviation is large, the reduction in goal_angle increases, and R5 changes from a penalty to a reward and increases as the reduction in goal_angle increases.

[0084] D.Learning process

[0085] To address the diverse environments in which robot path planning occurs, a neural network is used to maintain the Q-table in the Q-learning algorithm for reinforcement learning. The target value of the Q-network is determined by the reward for taking an action in each environmental state and the maximum Q-value of the next state.

[0086] y t =Q(s t , a t )+α[r t +γmaxQ(s t+1 , a t+1 )-Q(s t , a t )] (9)

[0087] Formula (9) is used as the target value of the gradient descent of the fully connected neural network, where α and γ are the learning rate and discount rate respectively. t For s t The reward obtained after taking the action, maxQ(s t+1 , a t ) is the maximum value of the next state Q.

[0088] During the training process of the algorithm of the present invention, two neural networks with identical structures are replicated. One neural network, the evaluation network, performs gradient descent and backpropagation. The other neural network, the target network, is used to calculate the Q value of the next state in (9). The parameters of this network are copied from the other neural network at regular intervals. This prevents overfitting and improves the stability and convergence speed of the training process.

[0089] At the same time, [s, a, r, s'] is treated as a set of data and experience is put back in. Initially, when there is no experience in the memory bank, the evaluation network is not trained. During training, a certain number of experiences are randomly drawn from the memory bank and fed into the evaluation network.

[0090] During the learning process, when the robot begins moving (i.e., when the number of steps forward is less than 200), it randomly selects whether to replan and the dynamic weight parameters of DWA. When the number of steps exceeds 200, S is passed to the evaluation network, and the current action A = [a1, a2] is selected. After the action is selected, the robot moves forward one step in the environment to determine the next state S'. Based on S and S', the value R is calculated. The value [S, A, R, S'] is then stored in the memory bank for subsequent retrieval and training. Learning is performed every five steps.

[0091] In order to increase the stability of learning and reduce overestimation, the neural network to be learned is divided into evaluation network and target network. In learning, S is passed into the evaluation network to obtain the corresponding [S t , A t ] is used for the subsequent gradient descent calculation. The target network is used to calculate the target value. After extracting the batch size [S, A, R, S'], S' is passed to the target network to calculate the maximum Q value of all actions taken in the S' state, and then calculate the target value q_target.

[0092] q_target=R+γ×Q_value max (19)

[0093] After q_target is obtained as (19), it is compared with Q[S t , A t ] to obtain Loss.

[0094] Loss=MSE(q_eval, q_target) (20)

[0095] Gradient descent is then performed based on the loss to update the parameters of the evaluation network. Finally, the evaluation network's parameters are copied to the target network at regular intervals. When training is complete, the two neural networks are identical. After training, the target network is used to calculate the Q value of the action in actual decision-making.

[0096] The simulation experiments and data of the present invention include the following:

[0097] A. Determination of parameters and evaluation functions

[0098] TABLE 1. Parameter value

[0099]

[0100] To better evaluate the performance of obstacle avoidance algorithms, this paper considers the time to target, the angle of rotation, and the distance traveled. An metric, I, is assigned to better quantify the performance of obstacle avoidance algorithms. I decreases with increasing trajectory length and angle of rotation, and increases with increasing time to target. A higher I indicates a better algorithm.

[0101] I=W A (1-A norm )+W L (1-L norm )+W T ·T norm (twenty one)

[0102] In formula (21), A norm is the normalized average rotation angle, L norm is the normalized average trajectory length, T norm is the normalized average time.

[0103] B. DWA algorithm with dynamic parameter selection

[0104] After dynamic parameter planning, the present invention provides the robot with two sets of selectable scoring function parameter values. One set is for when obstacles are sparse, with a high speed evaluation ratio and a low direction evaluation ratio and an obstacle distance evaluation ratio, and gives priority to the fastest path; the other set is for when obstacles are dense, with a low speed evaluation ratio and a high direction evaluation ratio and an obstacle distance evaluation ratio, and gives priority to the path that can effectively avoid obstacles to increase the pass rate.

[0105] Figure 5a 、 5b In 6a and 6b, black blocks represent static obstacles and yellow blocks represent dynamic unknown obstacles. Figure 5a 、 5bAs shown in the figure below, when only one set of data is used, if the first set of parameters is used, the path with higher speed will be given priority. When there are many obstacles, the speed will be given priority. As shown in the left figure, the accuracy and success rate of obstacle avoidance will be reduced. If the second set of parameters is used,

[0106] Whenever there is an obstacle, the speed will be greatly reduced to avoid it.

[0107] This results in an increase in travel time.

[0108] Table 2. Comparison of indicators

[0109] like Figure 6a 、 6b As shown in the figure, when encountering a large dynamic obstacle, without static global replanning, the path becomes longer and less smooth in order to return to the original global path, and it takes longer to reach the target. Global static replanning based on visible dynamic obstacles and known static obstacles can make the path smoother and the distance shorter.

[0110] like Figure 7a 、 7b As shown, the time to reach the target point is longer without global static replanning than with global static planning. Furthermore, the smoother path with global static replanning is also reflected in the fact that the speed can be maintained at a predetermined maximum. Furthermore, the change in angular velocity is significantly greater without global static replanning than with global static replanning, indicating that the optimized obstacle avoidance algorithm has a smoother trajectory and a smaller turning angle.

[0111] As shown in Table 2, the final optimized dynamic parameter DWA and the reinforcement learning global static reclassification fusion algorithm have obvious advantages in terms of rotation angle, average speed, average time, evaluation index I, and success rate.

[0112]

[0113] TABLE 2

[0114] C. DQN trained with dual neural networks

[0115] like Figure 8 As shown, when the present invention uses two neural networks for training, the reward increases significantly more per epoch than when training with a single neural network. This suggests that two neural networks converge faster than a single neural network. Furthermore, having one network provide the target value while the other learns helps stabilize training, prevents a single network from overfitting the current policy or value function, and improves system robustness.

[0116] For those skilled in the art, the present invention is not limited to the exemplary implementation described above. Any technician familiar with this technical field who replaces or changes the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A global-local collaborative trajectory planning method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Start the movement. When the number of steps forward is less than 200, randomly select whether to replan and the dynamic weight parameters of DWA; Step 2: When the number of steps is greater than 200, pass S into the evaluate network to select the current action; Step 3: After selecting the action, move forward in the environment to get the next state S'; Step 4: Calculate the R value based on S and S', and then store [S, A, R, S'] in the memory bank; Step 5: Extract [S, A, R, S'] from the memory bank for training, learning once every 5 steps.

2. A global-local collaborative trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that: The method is to optimize the redundant points by quadratic interpolation, starting from point S to connect the nodes (n1, n2...n m ) until the minimum distance between the line connecting the two points and the obstacle does not meet the minimum distance for the robot to pass. m , n m-1 Find a distance n m The nearest point makes the line connecting it and the starting point far enough away from the obstacle, and then takes the point n closest to the obstacle on the line m+1 And use this point as the starting point for the next optimization of redundant points, and repeat the above steps until the end point G.

3. The global-local collaborative trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that: The method considers the density of obstacles within a certain range, the shape of the obstacles, and the degree of deviation from the global path to decide whether to replan the global path: s=[n,angle,goal_angle] T (11) The state matrix is ​​shown in (11). A fully connected neural network is used, and the transition parameters do not need to be quantified and can be directly input into the neural network. Where n is the sum of the number of known static obstacles and unknown dynamic obstacles within a certain range; Formula (12) describes the degree of occlusion of the target point by an unknown dynamic obstacle; θ i For the direction of progress After the position dynamic obstacle is rasterized, the angle between the line connecting the obstacle and the robot and the line connecting the next target point and the robot; The scanning process is as follows: when the line connecting the next target point and the robot passes through the obstacle interval, the obstacles are scanned to both sides with the line as the center line; until the ray emitted from the robot's observation point does not pass through the obstacle interval, the angle of the line connecting the last obstacle and the robot is taken as θ i , because the degree of occlusion is greatest when the line connecting the target point and the robot passes through the center of a large obstacle; the closer the area passed through is to the edge of the obstacle, the smaller the degree of occlusion; therefore, the occlusion degree value is obtained by adding the left and right angles and subtracting the variance of the two angles. When the unknown dynamic obstacles within a certain range of the robot's travel direction are small or highly dispersed, the angle value is smaller; conversely, the larger the unknown dynamic obstacles are, the higher the degree of occlusion on the travel route, and the larger the angle value; goal_angle is the angle between the line connecting the next goal and the previous goal and the line connecting the next goal and the agent vertex. The larger the value of goal_angle, the further the agent deviates from the original planned path.

4. The global-local collaborative trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that: The method defines the reward function as follows: R=R1+R2+R3+R4+R5 (13) Among them, R1 is related to reaching the target and collision: R2 is related to the degree of occlusion and its reduction after the decision is made: Among them, Δangle is the change in the next angle value after the decision is made. If it decreases, a positive reward is given; when a1 is 0, no replanning is performed, and when it is 1, replanning is performed. Replanning has a certain computational cost based on negative rewards; R3 is related to the distance to the next target: R4 is related to the distance to the obstacle; R5 is related to the degree of deviation from the global path: R5=-a1×20-Δgoal_angle (18) When the degree of deviation is small, no re-planning is required. When the degree of deviation is large, the reduction in goal_angle increases, and R5 changes from a penalty to a reward and increases as the reduction in goal_angle increases.

5. The global-local collaborative trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that: The method uses a neural network to maintain the Q table in the Q-learning algorithm for reinforcement learning in response to the diversity of robot path planning environments. The target value of the Q network is determined by the reward for taking an action in each environmental state and the maximum Q value of the next state: and t =Q(s t ,to t )+α[r t +γmaxQ(s t+1 ,to t+1 )-Q(s t ,to t )] (9) Formula (9) is used as the target value of the gradient descent of the fully connected neural network, where α and γ are the learning rate and discount rate respectively, r t For s t The reward obtained after taking the action, maxQ(s t+1 , a t ) is the maximum value of the next state Q.

6. A global-local collaborative trajectory planning method based on deep reinforcement learning according to claim 1, characterized in that: The method copies two neural networks with identical structures during the training process. One of the neural networks, the evaluation network, performs gradient descent and back propagation. The other neural network, the target network, is used to calculate the Q value of the next state in (9). The parameters of this network are copied from the other neural network at a certain period. This can prevent overfitting and improve the stability and convergence speed of the training process. At the same time, [s, a, r, s'] is used as a set of data for experience replacement. When there is no experience in the memory bank at the beginning, the evaluation network is not trained. During training, a certain number of experiences are randomly extracted from the memory bank and put into the evaluation network. For the learning process, when the robot starts moving, that is, when the number of steps forward is less than 200, it randomly selects whether to replan and the dynamic weight parameters of DWA. When the number of steps is greater than 200, S is passed to the evaluation network to select the current action A = [a1, a2]. After the action is selected, it moves forward one step in the environment to obtain the next state S'. The R value is calculated based on S and S', and then [S, A, R, S'] is stored in the memory bank for subsequent extraction training, and learning is performed every 5 steps.

Images