Dynamic obstacle avoidance path planning method and device in energy limited environment, and medium
By using randomization and PRM algorithms to generate path networks in energy-constrained environments, and combining DQN model to optimize paths, the problem of insufficient adaptability of path planning in the existing technology is solved, and efficient path planning and energy in complex environments are achieved manage.
Patent Information
- Application Number
- CN202411084741.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-08
AI Technical Summary
Existing path planning methods are difficult to effectively plan paths in dynamic environments with energy constraints, especially in complex and multi-objective environments.
Randomization method is used to generate random nodes and obstacles in the environment map, the initial path network is built using the PRM algorithm, and the path is optimized through the DQN model, combined with the KD-Tree algorithm to optimize node connections and obstacle processing, considering energy supply stations and energy consumption , adopt priority experience replay strategies and importance sampling to improve the adaptability and efficiency of path planning.
It significantly improves the adaptability and flexibility of path planning, can effectively deal with complex path planning problems in energy-constrained environments, improves navigation efficiency and safety, and is suitable for dynamic environments and energy management.
Smart Images

Figure CN118915770A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robot path planning algorithms, and in particular to a dynamic obstacle avoidance path planning method, device and medium in an energy-constrained environment. Background Art
[0002] With the rapid development of artificial intelligence and robotics, as well as the growing demand for applications, the application of automated equipment in various complex environments is becoming increasingly popular. This field focuses on how to efficiently guide robots or equipment to move from one location to another while avoiding obstacles, optimizing path length, and completing target requirements. Traditional path planning methods work well in simple or static environments, but they face many challenges in dealing with multi-objective problems with limited energy and dynamically changing environments, and lack the necessary adaptability and accuracy.
[0003] Deep Reinforcement Learning (DRL), as an advanced technology that combines reinforcement learning and deep learning, has attracted great attention in the field of path planning. Through the interactive learning process between the robot and the environment, DRL can discover effective strategies to solve path planning problems. However, despite its great potential in the field of path planning, there are still two major challenges: (1) Decision-making efficiency in complex environments. DRL needs to make effective decisions quickly in complex environments, which places higher requirements on the environment and the algorithm itself; (2) The ability to quickly adapt to environmental changes. DRL must be able to adapt to rapid changes in the environment to effectively respond to new challenges and conditions.
[0004] Therefore, there is an urgent need to develop new path planning methods that can effectively handle path planning problems in complex, multi-objective environments while overcoming the limitations of traditional methods and the challenges of existing deep reinforcement learning methods. Summary of the invention
[0005] The present invention aims to at least solve the technical problem in the prior art that existing planning methods have great limitations in path planning in a dynamic environment with limited energy and are difficult to make effective planning.
[0006] To this end, a first aspect of the present invention provides a dynamic obstacle avoidance path planning method in an energy-constrained environment.
[0007] A second aspect of the present invention provides a computer device.
[0008] A third aspect of the present invention provides a computer-readable storage medium.
[0009] The present invention provides a dynamic obstacle avoidance path planning method in an energy-constrained environment, comprising:
[0010] Within a given environment map framework, random nodes, additional obstacles, and energy supply stations representing possible path points are generated in the environment using a randomization method; wherein the given environment map framework includes the distribution of fixed obstacles and the determined start and end points;
[0011] By using the PRM algorithm (probabilistic roadmap method), connections between nodes are established according to the random nodes to form an initial path network; wherein the KD tree algorithm is used to optimize the search process of neighboring nodes, and the geometric collision detection method is used to evaluate the accessibility between nodes and the effectiveness of the path;
[0012] Adjust obstacle constraints and path nodes, and use incremental PRM to re-evaluate and update the connections between nodes to update the path network structure of the map;
[0013] The generated node paths are analyzed using the DQN model (a reinforcement learning algorithm that combines deep learning and Q-learning) to estimate the expected reward or value of each action. The DQN model learns to select the optimal movement strategy based on environmental data and historical path effects. The DQN model uses a value function to estimate the expected return of taking action a in state s.
[0014] Based on the action-value function optimized by the DQN model, the optimal action sequence is selected, and then these actions are converted into actual movement or path decisions.
[0015] The dynamic obstacle avoidance path planning method in an energy-constrained environment according to the above technical solution of the present invention may also have the following additional technical features:
[0016] In the above technical solution, the randomization method includes:
[0017] Within the given environment map frame, several additional obstacles are placed by standard random sampling;
[0018] Within the given environment map framework, the location of the energy supply station is determined by random sampling within the area, wherein the energy supply station site does not overlap with obstacles;
[0019] Within a given environmental map framework, a first set ratio of the random nodes is generated by standard random sampling, and a second set ratio of the random nodes is generated in an obstacle-dense area by a dense sampling method; wherein the random nodes do not overlap with obstacles and energy supply stations.
[0020] In the above technical solution, the establishment of connections between nodes according to the random nodes through the PRM algorithm to form an initial path network includes:
[0021] According to the random node N = {n1, n2, .., nk}, the spatial index of the node is constructed using the KD tree algorithm to query the nearest neighbor relationship between nodes, and the nearest neighbor node set of the random node ni is defined as NN(ni);
[0022] For any random node ni, determine the edges connected to it, including:
[0023] E(ni)={(ni,nj)|nj∈NN(ni), and dist(ni,nj)≤D max , and there is no obstacle collision}
[0024] Where E(ni) represents the set of edges connected to the random node ni, dist(ni,nj) represents the Euclidean distance between the random nodes ni and nj, and D max represents the set maximum connection distance, i and j represent the node numbers respectively, and k represents the number of random nodes.
[0025] In the above technical solution, the adjustment of obstacle constraints and path nodes includes increasing or decreasing obstacles, and increasing or decreasing path nodes accordingly.
[0026] In the above technical solution, the obstacle constraints and path nodes are adjusted, and the connection between nodes is re-evaluated and updated using the incremental PRM to update the path network structure of the map, including:
[0027] Periodically add or reduce obstacles; when adding obstacles, remove the nodes covered by the newly added obstacles; when reducing obstacles, if there is a historical node covered by the reduced obstacle, re-enable the historical node; if there is no historical node, add a new path node at the center of the removed obstacle;
[0028] After node reduction, reactivation or node increase, the KD-Tree algorithm is used to improve the incremental PRM to retrieve nodes within the maximum connection distance D. max For other nodes within the range, the nodes are locally reconstructed accordingly according to the node connection strategy of the incremental PRM algorithm.
[0029] In the above technical solution, the DQN model adopts a multi-layer perceptron structure, and the DQN model is used to analyze the generated node path to estimate the expected reward or value of each action, including:
[0030] The Q value Q(s, a; θ) estimated by the DQN model represents the expected return of selecting action a in a given state s; the Q value is updated by the following formula:
[0031] Q(s,a;θ)←Q(s,a;θ)+α[r+γmax a′Q(s′, a′; θ - )-Q(s,a;θ)]
[0032] Among them, s′ represents the new state after taking the new action a, a′ represents the next action of action a, r represents the immediate reward received, γ represents the discount factor, α represents the learning rate, θ represents the parameters of the current Q network, and θ - represents the parameters of the target Q network;
[0033] The extended one-hot encoding is used as the state representation, where the first part of the state vector is the node position, and the last two bits represent the energy level and the distance to the nearest energy supply station respectively;
[0034] The reward function R(s, a) calculates the immediate reward based on the state s and action a, including:
[0035]
[0036] Among them, rewards means immediate reward; R step represents the penalty for each move, R goal (s′) represents the reward obtained when successfully reaching the target, R distance (s, s′) represents the reward based on the distance difference between the current position and the next position to the target, R fuel_station_reward (s′) represents the reward for being close to the energy supply station when energy needs to be replenished and the penalty for being far away from the energy supply station when the energy level is too low, R empyt (s′) represents the penalty of energy depletion;
[0037] All nodes reachable from the current node are taken as possible actions. If there is no connected node, a node is randomly selected as a possible action. The action with the highest long-term reward is selected from all feasible actions. The ε-greedy strategy is used for action selection and the energy consumption is calculated according to the moving distance. The rule for selecting an action under a given state s is defined as the motion strategy π(s). The motion strategy π(s) and energy consumption Energy_Consume_d are expressed as:
[0038]
[0039] Where argmax represents the maximum index function; Rand(A) represents a random action in the action set; distance_energy represents the Euclidean distance between two nodes; Max_Energy_Scale represents the maximum scale of energy consumption; D max Indicates the set maximum connection distance; ε indicates the probability parameter, which ranges from 0 to 1;
[0040] Combined with energy consumption, a priority experience replay strategy is adopted to reduce the temporal correlation between observation data; among them, the importance sampling method is used to balance the sampling distribution and set the importance weight;
[0041] When learning updates, a weighted loss function is used to reduce potential biases due to high sampling probabilities, including:
[0042] L(θ i )=(Q(s,a;θ)-(r+γ·max a′ Q(s′,a′;θ-)-c·Energy_Consumed)) 2
[0043] LOSS=∑ i W i ·L(θ i )
[0044] Among them, L(θ i ) represents the loss function on sample i, LOSS represents the weighted loss function; r represents the immediate reward received by the agent from the environment; c represents the cost coefficient of energy consumption; w i represents the importance weight of the i-th sample.
[0045] In the above technical solution, the priority experience replay strategy is adopted in combination with energy consumption to reduce the time correlation between observation data, including:
[0046] Calculate the difference TD between the predicted Q value and the actual reward error :
[0047] TD error =|r+γ·max a′ Q(s′, a′; θ - )-Q(s, a; θ)-c·Energy_Consumed|
[0048] During the training process, the experience elements are sampled according to their priority, the priority p i And the sampling probability P(i) is expressed as:
[0049]
[0050] Where z represents the priority influence parameter; β represents the exponential parameter for controlling the priority calculation; τ represents the set value to avoid the priority being zero;
[0051] Balance the sampling distribution through the importance sampling method and set the importance weight w i :
[0052]
[0053] Where N is the total number of samples in the experience replay buffer and β′ is the control importance weight.
[0054] In the above technical solution, the optimal action sequence is selected based on the action value function optimized by the DQN model, and then these actions are converted into actual movement or path decisions, including:
[0055] The path with the highest expected reward is selected from the output of the DQN model, and the selected optimal path is converted into the actual movement instructions of the robot.
[0056] A computer device provided by the present invention includes a processor and a memory, wherein a computer program is stored in the memory. When the computer program is loaded and executed by the processor, a dynamic obstacle avoidance path planning method in an energy-constrained environment as described in any one of the above technical solutions is implemented.
[0057] The present invention provides a computer-readable storage medium storing a program, which, when loaded by a processor, implements a dynamic obstacle avoidance path planning method in an energy-constrained environment as described in any one of the above technical solutions.
[0058] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are:
[0059] The present invention proposes a dynamic obstacle avoidance path planning method in an energy-constrained environment, which is designed for path planning problems in dynamic and complex environments with energy management requirements. This method first uses the PRM algorithm to generate key nodes in the environment and build a preliminary path planning framework. Then, the DQN model is used to deeply analyze and optimize these node paths, accurately estimate the potential value of each action, and thus formulate the optimal action strategy.
[0060] The core advantage of the present invention is that it has excellent processing capabilities in energy-constrained dynamic environments. Through the combination of PRM's efficient spatial sampling and DQN's advanced reinforcement learning capabilities, this method not only greatly improves the adaptability and flexibility of path planning, but also through PRM's effective sampling in obstacle-dense environments, and by using the KD-Tree algorithm to improve the incremental PRM, it further improves the efficiency of handling obstacles and map updates, significantly reducing the impact of obstacles on the DQN decision-making process, allowing DQN to focus on path optimization. In addition, the invention significantly improves the performance of the path planning strategy by continuously collecting environmental data and continuously optimizing the DQN model using the experience replay pool strategy.
[0061] The present invention also takes into account the energy management of the robot. By setting energy supply station nodes and considering energy consumption in path planning, it is possible to ensure that the robot can effectively complete tasks under limited energy supply. This not only improves the practicality of path planning, but also increases its applicability in energy-constrained environments.
[0062] Compared with traditional methods, the present invention performs well in processing large-scale dynamic spatial data and optimizing long-term path planning strategies. It has shown great potential in various application scenarios, such as robot navigation, energy coverage planning, etc. Through intelligent path planning and consideration of energy constraints, the present invention significantly improves navigation efficiency and safety in dynamic environments.
[0063] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0065] Figure 1 is a flow chart of a method for dynamic obstacle avoidance path planning in an energy-constrained environment according to an embodiment of the present invention;
[0066] Figure 2 It is a schematic diagram of the principle framework of the DQN algorithm in a dynamic obstacle avoidance path planning method in an energy-constrained environment according to an embodiment of the present invention;
[0067] Figure 3 is a network structure diagram of a DQN in a dynamic obstacle avoidance path planning method in an energy-constrained environment according to an embodiment of the present invention;
[0068] Figure 4 It is a path connection network diagram based on the PRM algorithm in a dynamic obstacle avoidance path planning method in an energy-constrained environment according to an embodiment of the present invention;
[0069] Figure 5 It is an optimal path graph generated by a DQN model in a dynamic obstacle avoidance path planning method in an energy-constrained environment according to an embodiment of the present invention. DETAILED DESCRIPTION
[0070] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0071] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0072] Refer to the following Figures 1 to 5 The following describes a dynamic obstacle avoidance path planning method in an energy-constrained environment provided according to some embodiments of the present invention.
[0073] Some embodiments of the present application provide a dynamic obstacle avoidance path planning method in an energy-constrained environment.
[0074] like Figure 1 As shown, the first embodiment of the present invention proposes a dynamic obstacle avoidance path planning method in an energy-constrained environment, including the following steps S1-S5.
[0075] S1. Within a given environment map framework, a randomization method is used to generate random nodes, additional obstacles and energy supply stations representing possible path points in the environment; wherein the given environment map framework includes the distribution of fixed obstacles and determined starting and ending points.
[0076] In some embodiments, the randomization method comprises:
[0077] Within the given environment map frame, several additional obstacles are placed by standard random sampling;
[0078] Within the given environment map framework, the location of the energy supply station is determined by random sampling within the area, wherein the energy supply station site does not overlap with obstacles;
[0079] In a given environment map framework, a first set ratio of the random nodes is generated by standard random sampling, and a second set ratio of the random nodes is generated in an obstacle-dense area by dense sampling; wherein the random nodes do not overlap with obstacles and energy supply stations. The sum of the first set ratio and the second set ratio is equal to 1.
[0080] In a specific embodiment, the execution process of step S1 is as follows:
[0081] According to the environmental characteristics, a 2500×2500 network environment was constructed. First, the positions of the starting point and the end point were determined, which were set at (40, 40) and (2460, 2460) respectively. The obstacles in the environment were determined by the center point position and the random radius size to form a rectangular obstacle area. Specifically, four fixed static fixed obstacles were set, with their center points located at (600, 600), (1300, 1500), (2000, 630), and (300, 2100). In addition, 20 additional obstacles were placed in the environment through standard random sampling, and the radius of each obstacle varied randomly between 150 and 300. The initial energy was set to 70, and the energy discount for each movement of the robot was 1.8. In order to ensure that the agent can effectively replenish energy during the task, 10 energy supply stations were arranged by random sampling in the area, while ensuring that these stations do not overlap with obstacles. In terms of node generation, a total of 300 nodes representing potential path points were generated, while ensuring that these nodes did not overlap with obstacles and energy supply stations. 70% of the nodes were generated through standard random sampling of the entire grid, while the remaining 30% were generated through dense sampling methods in areas with more obstacles to increase the complexity and realism of path planning. The neighboring selection range of each node was set to 8% of the configuration space, while the maximum connection distance was set to 10% of the configuration space. The preliminary path planning node data N = {n 1 , n 2 , ..., n 300}, each feature node contains the node's coordinate position, connection status with other nodes, and environmental information.
[0082] In some embodiments, the environment is simplified into a two-dimensional grid map, and the robot's state space is defined as coordinate points in the map, where the starting point, end point, path nodes, and energy supply stations are regarded as key points occupying a single grid, while obstacles exist in the map in the form of occupying multiple grids, together constructing a complete global state space map.
[0083] S2. Through the PRM algorithm (probabilistic roadmap method), connections between nodes are established according to the random nodes to form an initial path network; wherein, the KD tree algorithm (k-dimensional tree, KD-Tree) is used to optimize the search process of adjacent nodes, and the geometric collision detection method is used to evaluate the accessibility between nodes and the effectiveness of the path; to ensure the practicality and security of the constructed network.
[0084] In some embodiments, the step S2 of establishing connections between nodes according to the random nodes through the PRM algorithm to form an initial path network includes:
[0085] According to random nodes N = {n1, n2, ..., nk}, a spatial index of nodes is constructed using the KD tree algorithm to query the nearest neighbor relationship between nodes, and the nearest neighbor node set of random node ni is defined as NN(ni);
[0086] For any random node ni, determine the edges connected to it, including:
[0087] E(ni)={(ni,nj)|nj∈NN(ni), and dist(ni,nj)≤D max , and there is no obstacle collision}
[0088] Where E(ni) represents the set of edges connected to the random node ni, dist(ni,nj) represents the Euclidean distance between the random nodes ni and nj, and D max represents the set maximum connection distance, i and j represent the node numbers respectively, and k represents the number of random nodes.
[0089] In a specific embodiment, the KD-Tree algorithm is used to construct a spatial index for the generated 300 nodes. In a 2500×2500 network environment, for each node ni (i=1, 2, ..., 300), its coordinate (x i ,y i ) is added to the KD-Tree to generate the nearest neighbor node set defined as NN(ni), through NN(n i )={n j |dist(n i , n j )≤200, j≠i} in order to quickly query the nearest neighbor relationship between nodes.
[0090] For each node, based on its position in the KD-Tree, query its nearest neighbor node. Then, according to the preset maximum connection distance of 10% of the configuration space and the obstacle avoidance strategy, determine whether the nodes should be connected to form the initial path network E(ni):
[0091] E(ni)={(ni,nj)|nj∈NN(ni), and dist(ni,nj)≤250, and there is no obstacle collision}.
[0092] S3, adjust obstacle constraints and path nodes, and use incremental PRM to re-evaluate and update the connections between nodes to update the path network structure of the map.
[0093] In some embodiments, adjusting obstacle constraints and path nodes includes increasing or decreasing obstacles, and increasing or decreasing path nodes accordingly.
[0094] Specifically, step S3 includes:
[0095] Periodically add or reduce obstacles; when adding obstacles, remove the nodes covered by the newly added obstacles; when reducing obstacles, if there is a historical node covered by the reduced obstacle, re-enable the historical node; if there is no historical node, add a new path node at the center of the removed obstacle;
[0096] After node reduction, reactivation or node increase, the KD-Tree algorithm is used to improve the incremental PRM to retrieve nodes within the maximum connection distance D. max For other nodes within the range, the nodes are locally reconstructed accordingly according to the node connection strategy of the incremental PRM algorithm.
[0097] In a specific embodiment, an obstacle is added to the map every 1 / 10 of the total cycle, with a radius ranging from 150 to 300, and it is ensured that the obstacle is at least half the radius away from the array boundary; in addition, any node covered by the new obstacle will be removed. At the same time, an obstacle is randomly removed from the map every 1 / 6 of the total cycle. When reducing obstacles, if there are any nodes covered by the obstacle, these nodes will be re-enabled; if no node is covered, a new path node will be added at the center of the obstacle.
[0098] After the nodes are reduced, reactivated, or increased, the incremental PRM is improved by the KD-Tree algorithm to retrieve other nodes within the maximum connection distance of 250. According to the node connection strategy of the incremental PRM algorithm, the nodes are locally reconstructed accordingly.
[0099] S4. Use the DQN model (a reinforcement learning algorithm that combines deep learning and Q-learning) to analyze the generated node path to estimate the expected reward or value of each action. The DQN model learns to select the optimal movement strategy based on environmental data and historical path effects. The DQN model uses the value function to estimate the expected return of taking action a in state s.
[0100] Specifically, the overall process of establishing the DQN model is as follows Figure 2 As shown, Figure 3 The DQN model neural network structure is shown.
[0101] In some embodiments, step S4 includes the following steps S41-S46. It should be noted that the order of steps S41-S46 is not the execution order of step S4, but should be understood as the content included in step S4, that is, the components of step S4. Different steps can occur simultaneously, and the order of steps should be flexibly adjusted according to actual needs.
[0102] S41. The Q value Q(s, a; θ) estimated by the DQN model represents the expected return of selecting action a in a given state s. The Q value is updated by the following formula:
[0103] Q(s,a;θ)←Q(s,a;θ)+α[r+γmax a′ Q(s′,a′;θ-)-Q(s,a;θ)]
[0104] Among them, s′ represents the new state after taking the new action a, a′ represents the next action of action a, r represents the immediate reward received, γ represents the discount factor, α represents the learning rate, θ represents the parameters of the current Q network, and θ- represents the parameters of the target Q network.
[0105] S42. Extended one-hot encoding is used as the state representation, where the front part of the state vector is the node position (when the robot is located at a certain node, the corresponding position is 1, and the rest are 0), and the last two digits represent the energy level and the distance to the nearest energy supply station respectively.
[0106] S43, the reward function R(s, a) calculates the immediate reward according to the state s and the action a, including:
[0107]
[0108] Among them, rewards means immediate reward; R step represents the penalty for each move, R goal (s′) represents the reward obtained when successfully reaching the target, R distance (s, s′) represents the reward based on the distance difference between the current position and the next position to the target, R fuel_station_reward (s′) represents the reward for being close to the energy supply station when energy needs to be replenished and the penalty for being far away from the energy supply station when the energy level is too low, R empyt (s′) represents the penalty for energy depletion.
[0109] S44. Take all nodes reachable from the current node as possible actions. If there is no connected node, randomly select a node as a possible action. Select the action with the highest long-term reward from all feasible actions. Use the ε-greedy strategy to select actions and calculate and normalize the energy consumption according to the moving distance. Define the rule for selecting actions under a given state s as the motion strategy π(s). Then the motion strategy π(s) and energy consumption Energy_Consumed are expressed as:
[0110]
[0111] Where argmax represents the maximum index function; Rand(A) represents a random action in the action set; distance_energy represents the Euclidean distance between two nodes; Max_Energy_Scale represents the maximum scale of energy consumption; D max represents the set maximum connection distance; ε represents the probability parameter, which ranges from 0 to 1.
[0112] S45. In order to reduce the temporal correlation between observation data and make the data independent of each other, a priority experience replay strategy is used in combination with energy consumption to reduce the temporal correlation between observation data to improve the efficiency and effect of the learning process. Among them, the sampling distribution is balanced by the importance sampling method and the importance weight is set.
[0113] S46. When learning updates, use a weighted loss function to reduce potential biases due to high sampling probabilities, including:
[0114] L(θ i )=(Q(s,a;θ)-(r+γ·max a′ Q(s′, a′; θ - )-c·Energy_Consumed)) 2
[0115] LOSS=∑ i w i ·L(θ i )
[0116] Among them, L(θ i ) represents the loss function on sample i, LOSS represents the weighted loss function; r represents the immediate reward received by the agent from the environment; c represents the cost coefficient of energy consumption; w i represents the importance weight of the i-th sample.
[0117] In a specific embodiment, the learning rate and reward discount in step S41 are set to 0.0001 and 0.9 respectively, and the Q value is updated by the following formula:
[0118] Q(s,a;θ)←Q(s,a;θ)+0.0001·[r+0.9·max a′ Q(s′,a′;θ)-Q(s,a;θ)]
[0119] The reward design in step S43 is as follows:
[0120]
[0121] dist(s, goal) represents the distance from the current position to the goal, and dist(s′, goal) represents the distance from the next position to the goal.
[0122] In step S44, the ε-greedy strategy is adopted. The value of ε is initially set to 0.9. As each round proceeds, the value of ε gradually decreases at a decay rate of 0.995. The maximum scale of energy consumption is set to 3 to calculate and normalize the robot's energy consumption Energy_Consumed according to the moving distance, which can be expressed as:
[0123]
[0124] Step S45 includes:
[0125] Calculate the difference TD between the predicted Q value and the actual reward error :
[0126] TD error =|r+γ·max a′ Q(s′, a′; θ - )-Q(s, a; θ)-c·Energy_Consumed|
[0127] In a specific embodiment, the discount factor is set to 0.9 and the cost coefficient of energy consumption is set to 0.5. Then TD error It can be expressed as:
[0128] TD error =|r+0.9·max a′ Q(s′, a′; θ - )-Q(s,a;θ)-0.5·Energy_Consumed|
[0129] During the training process, the experience elements are sampled according to their priority, the priority p i And the sampling probability P(i) is expressed as:
[0130]
[0131] Wherein, z represents the priority influence parameter; β represents the exponential parameter for controlling the priority calculation; τ represents the set value to avoid the priority being zero; in a specific embodiment, a parameter z for adjusting the priority influence is set to 0.6, a very small set value τ is set to 0.01 to avoid the priority being zero, an exponential parameter β for controlling the priority calculation is set to 0.5, and the priority p is i And the sampling probability P(i) can be expressed as:
[0132]
[0133] Balance the sampling distribution through the importance sampling method and set the importance weight w i :
[0134]
[0135] Wherein, N represents the total number of samples in the experience playback buffer, and β′ represents the control importance weight. In a specific embodiment, the parameter β′ of the control importance weight is set to 0.4.
[0136] S5. Based on the action value function optimized by the DQN model, the robot selects the optimal action sequence and then converts these actions into actual movement or path decisions to achieve path planning of the robot in a complex environment.
[0137] In some embodiments, step S5 includes:
[0138] The path with the highest expected reward is selected from the output of the DQN model, and the selected optimal path is converted into the actual movement instructions of the robot. That is, the action value function Q(s, a) of the DQN model is used to evaluate the utility of taking action a in a given state s. The model continuously optimizes this function through training. For the current state s, the action a that maximizes Q(s, a) is selected. * , use a * = argmax a Q(s, a) is used to select the optimal expected action. * Transformed into actual robot movement or path decision to achieve path planning in the environment. If the optimal path does not exist, that is, the robot cannot move forward due to energy problems or needs to re-plan the path, it is necessary to re-plan the path, and if an optimal path is finally found, it will be returned. However, if the optimal path cannot be found anyway, then we will return "Not FoundPath" to indicate the difficulty of navigation in the current environment.
[0139] like Figure 4 and Figure 5 As shown, the disclosed method successfully generated a path connection network based on the PRM algorithm, and accurately identified and selected the optimal path through the DQN model.
[0140] This embodiment improves the incremental PRM by using the KD-Tree algorithm and combines its efficient spatial sampling with the powerful decision-making ability of DQN to effectively generate and optimize path planning by utilizing available resources while ensuring that the robot can reach the destination when there is enough energy. In this way, the robot can flexibly adjust its route according to real-time environmental changes to cope with emergencies or environmental changes.
[0141] Specifically, this embodiment combines the efficient spatial sampling capability of PRM with the advantages of deep reinforcement learning. PRM provides a stable framework for path planning, which is particularly suitable for dealing with complex spatial environments. By improving it into an incremental PRM, the efficiency of handling obstacles and map updates can be further improved, so that DRL can focus on path optimization and reduce the impact of complex obstacle spaces. The introduction of DQN enables the agent to learn from historical data and environmental interactions to optimize the decision-making process. This combination not only solves the application limitations of traditional DQN in complex environments, but also provides a new path planning strategy to cope with dynamically changing environments and limited energy.
[0142] A second embodiment of the present invention provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the processor, the dynamic obstacle avoidance path planning method in an energy-constrained environment as described in any of the above embodiments is implemented.
[0143] A third embodiment of the present invention provides a computer-readable storage medium storing a program, which, when loaded by a processor, implements the dynamic obstacle avoidance path planning method in an energy-constrained environment as described in any of the above embodiments.
[0144] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0145] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A dynamic obstacle avoidance path planning method in an energy-constrained environment, characterized in that: include: Within a given environment map framework, random nodes, additional obstacles, and energy supply stations representing possible path points are generated in the environment using a randomization method; wherein the given environment map framework includes the distribution of fixed obstacles and the determined start and end points; By using the PRM algorithm, connections between nodes are established according to the random nodes to form an initial path network; wherein the KD tree algorithm is used to optimize the search process of adjacent nodes, and a geometric collision detection method is used to evaluate the accessibility between nodes and the effectiveness of the path; Adjust obstacle constraints and path nodes, and use incremental PRM to re-evaluate and update the connections between nodes to update the path network structure of the map; The generated node path is analyzed using the DQN model to estimate the expected reward or value of each action, where the DQN model learns to select the optimal motion strategy based on environmental data and historical path effects. The DQN model uses the value function to estimate the expected return of taking action a in state s. Based on the action-value function optimized by the DQN model, the optimal action sequence is selected, and then these actions are converted into actual movement or path decisions.
2. The dynamic obstacle avoidance path planning method in an energy-constrained environment according to claim 1, characterized in that: The randomization method includes: Within the given environment map frame, several additional obstacles are placed by standard random sampling; Within the given environment map framework, the location of the energy supply station is determined by random sampling within the area, wherein the energy supply station site does not overlap with obstacles; Within a given environmental map framework, a first set ratio of the random nodes is generated by standard random sampling, and a second set ratio of the random nodes is generated in an obstacle-dense area by a dense sampling method; wherein the random nodes do not overlap with obstacles and energy supply stations.
3. The dynamic obstacle avoidance path planning method in an energy-constrained environment according to claim 1, characterized in that: The method of establishing connections between nodes according to the random nodes through the PRM algorithm to form an initial path network includes: According to random nodes N = {n1, n2, ..., nk}, a spatial index of nodes is constructed using the KD tree algorithm to query the nearest neighbor relationship between nodes, and the nearest neighbor node set of random node ni is defined as NN(ni); For any random node ni, determine the edges connected to it, including: E(ni)={(ni,nj)∣∣nj∈NN(ni), and dist(ni,nj)≤D max , and no obstacles collide} Where E(ni) represents the set of edges connected to the random node ni, dist(ni,nj) represents the Euclidean distance between the random nodes ni and nj, and D max represents the set maximum connection distance, i and j represent the node numbers respectively, and k represents the number of random nodes.
4. The dynamic obstacle avoidance path planning method in an energy-constrained environment according to claim 1, characterized in that: The adjusting of obstacle constraints and path nodes includes increasing or decreasing obstacles, and increasing or decreasing path nodes accordingly.
5. The dynamic obstacle avoidance path planning method in an energy-constrained environment according to claim 4, characterized in that: The obstacle constraints and path nodes are adjusted, and the connections between nodes are re-evaluated and updated using the incremental PRM to update the path network structure of the map, including: Periodically add or reduce obstacles; when adding obstacles, remove the nodes covered by the newly added obstacles; when reducing obstacles, if there is a historical node covered by the reduced obstacle, re-enable the historical node; if there is no historical node, add a new path node at the center of the removed obstacle; After node reduction, reactivation or node increase, the KD-Tree algorithm is used to improve the incremental PRM to retrieve nodes within the maximum connection distance D. max For other nodes within the range, the nodes are locally reconstructed accordingly according to the node connection strategy of the incremental PRM algorithm.
6. The dynamic obstacle avoidance path planning method in an energy-constrained environment according to claim 1, characterized in that: The DQN model adopts a multi-layer perceptron structure. The DQN model is used to analyze the generated node path to estimate the expected reward or value of each action, including: The Q value Q(s,a;θ) estimated by the DQN model represents the expected return of selecting action a in a given state s; the Q value is updated by the following formula: Q(s,a;θ)←Q(s,a;θ)+α[r+γmax a′ Q(s′,a′;θ-)-Q(s,a;θ)] Among them, s′ represents the new state after taking the new action a, a′ represents the next action of action a, r represents the immediate reward received, γ represents the discount factor, α represents the learning rate, θ represents the parameters of the current Q network, and θ - represents the parameters of the target Q network; The extended one-hot encoding is used as the state representation, where the first part of the state vector is the node position, and the last two bits represent the energy level and the distance to the nearest energy supply station respectively; The reward function R(s,a) calculates the immediate reward based on the state s and action a, including: Among them, rewards means immediate reward; R step represents the penalty for each move, R goal (s′) represents the reward obtained when successfully reaching the target, R distance (s, s′) represents the reward based on the distance difference between the current position and the next position to the target, R fuel_station_reward (s′) represents the reward for being close to the energy supply station when energy needs to be replenished and the penalty for being far away from the energy supply station when the energy level is too low, R empyt (s′) represents the penalty of energy depletion; All nodes reachable from the current node are taken as possible actions. If there is no connected node, a node is randomly selected as a possible action. The action with the highest long-term reward is selected from all feasible actions. The ε-greedy strategy is used for action selection and the energy consumption is calculated according to the moving distance. The rule for selecting an action under a given state s is defined as the motion strategy π(s). The motion strategy π(s) and energy consumption Energy_Consumed are expressed as: Where argmax represents the maximum index function; Rand(A) represents a random action in the action set; distance_energy represents the Euclidean distance between two nodes; Max_Energy_Scale represents the maximum scale of energy consumption; D max Indicates the set maximum connection distance; ε indicates the probability parameter, which ranges from 0 to 1; Combined with energy consumption, a priority experience replay strategy is adopted to reduce the temporal correlation between observation data; among them, the importance sampling method is used to balance the sampling distribution and set the importance weight; When learning updates, a weighted loss function is used to reduce potential biases due to high sampling probabilities, including: L(θ i )=(Q(s,a;θ)-(r+γ·max a′ Q(s′,a′;θ - )-c·Energy_Consumed)) 2 LOSS=Σ i w i ·L(θ i ) Among them, L(θ i ) represents the loss function on sample i, LOSS represents the weighted loss function; r represents the immediate reward received by the agent from the environment; c represents the cost coefficient of energy consumption; w i represents the importance weight of the i-th sample.
7. The method for dynamic obstacle avoidance path planning in an energy-constrained environment according to claim 6, characterized in that: The energy consumption is combined with a priority experience replay strategy to reduce the time correlation between observation data, including: Calculate the difference TD between the predicted Q value and the actual reward error : TD error =|r+γ·max a′ Q(s′,a′;θ - )-Q(s,a;θ)-c·Energy_Counsumed| During the training process, the experience elements are sampled according to their priority, the priority p i And the sampling probability P(i) is expressed as: Where z represents the priority influence parameter; β represents the exponential parameter for controlling the priority calculation; τ represents the set value to avoid the priority being zero; Balance the sampling distribution through the importance sampling method and set the importance weight w i : Where N is the total number of samples in the experience replay buffer and β′ is the control importance weight.
8. The method for dynamic obstacle avoidance path planning in an energy-constrained environment according to claim 7, characterized in that: The optimal action sequence is selected based on the action value function optimized by the DQN model, and then these actions are converted into actual movement or path decisions, including: The path with the highest expected reward is selected from the output of the DQN model, and the selected optimal path is converted into the actual movement instructions of the robot.
9. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded and executed by the processor, a dynamic obstacle avoidance path planning method in an energy-constrained environment as described in any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium, characterized in that: A program is stored, and when the program is loaded by a processor, a dynamic obstacle avoidance path planning method in an energy-constrained environment as described in any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Automatic navigation method and device for robot with body and computer equipment
CN121207184A
An automatic navigation method, device and computer equipment for a robot with a body
CN121207184B
