Robot navigation method and device, computer equipment and storage medium

Abstract

The invention provides a robot navigation method and device, computer equipment and a storage medium, and belongs to the field of robot navigation.The method comprises the steps that the target distance between a robot and a final position and the obstacle distance between the robot and a nearest obstacle are obtained in real time; two algorithms are constructed to respectively control discrete actions and continuous actions of the robot; respectively determining an attraction potential field and a repulsion potential field, and constructing an artificial potential field based on the attraction potential field and the repulsion potential field; two algorithms are combined, and a unified navigation model is constructed based on an artificial potential field; potential energy variation of the robot is determined according to the artificial field potential, the potential energy variation of each step of the robot serves as a reward function, and the robot is guided to move in the potential energy reduction direction; and under the condition that the target distance is smaller than a preset threshold value, determining that the robot completes navigation. In this way, the discrete-continuous action hybrid control mode can adapt to the complex environment, and an accurate navigation path is provided for the robot.

Description

Technical Field

[0001] The present invention belongs to the field of robot navigation, and in particular relates to a robot navigation method, device, computer equipment and storage medium. Background Art

[0002] Autonomous robots are intelligent robots capable of completing tasks in unknown environments without relying on external instructions. Research on autonomous robots lies at the intersection of artificial intelligence and automation, and they can play a vital role in many fields, including healthcare, transportation, and environmental protection. For example, autonomous robots can deliver medicine and meals in hospitals, reducing the workload of medical staff. In the transportation sector, autonomous robots can assist with tasks such as waste collection and city patrols, becoming a crucial tool for urban management.

[0003] Autonomous navigation of robots is a crucial capability for ensuring that platforms can reach their destinations from a starting point without collision in environments with numerous obstacles. Traditional navigation methods typically involve simultaneous localization and mapping (SLAM), trajectory planning, and tracking control. However, SLAM mapping is time-consuming and requires high accuracy and density of LiDAR sensors. This makes autonomous navigation difficult in environments with sparse distance information and unobstructed maps. Therefore, end-to-end online learning navigation methods based on deep reinforcement learning (DRL) have attracted widespread attention. However, the performance of traditional DRL algorithms, such as deep Q-networks (DQNs), is often affected by Q-value overestimation and reward functions, and they are limited to discrete control tasks. This results in extremely poor performance, severely impacting the accuracy and efficiency of robot navigation and poor stability. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a robot navigation method, apparatus, computer equipment and storage medium.

[0005] In order to achieve the above object, the present invention provides the following technical solutions: A method for robot navigation, comprising: Obtain the target distance between the robot and the end position and the obstacle distance to the nearest obstacle in real time; Construct the PER-D3QN algorithm and DDPG algorithm to control the robot's discrete and continuous actions respectively; Determine an attractive potential field according to the target distance, determine a repulsive potential field according to the obstacle distance, and construct an artificial potential field based on the attractive potential field and the repulsive potential field; Combining the PER-D3QN algorithm for discrete motion navigation with the DDPG algorithm for continuous motion navigation, a unified navigation model is constructed based on artificial potential fields. Determine the robot's potential energy change based on the artificial field potential, use the potential energy change of each step of the robot as a reward function, and guide the robot to move in the direction of decreasing potential energy; When the target distance is less than a preset threshold, it is determined that the robot has completed navigation.

[0006] Optionally, introducing an artificial field potential as a reward function includes: The artificial potential field expression is: ; in, and are the attractive potential field and the repulsive potential field, respectively. Represents the distance from the target robot to the target point, Characterizes the minimum distance between the target robot and the obstacle; The reward function is constructed based on the potential energy change of the artificial field potential, and the reward function expression is determined as: ; in, are the potential energies of the previous step and the current step, respectively, is the reward value obtained; is a positive reward value, indicating successful navigation. A negative reward value indicates navigation failure; is the target point threshold, is the collision threshold, when Less than When , it is determined that the navigation to the target point is successful; when Less than , it indicates that a collision has occurred and navigation has failed.

[0007] Optionally, after the unified navigation model is constructed, the unified navigation model is further trained, including: The robot navigation process is converted into a Markov decision process (MDP). The robot obtains the reward of the Markov decision process at each time step based on discrete or continuous actions, and calculates the cumulative discounted reward in the MDP process to maximize the cumulative discounted reward to update the parameters of the current unified navigation model; the cumulative discounted reward is used to quantify the quality of the path.

[0008] Optionally, the calculation formula for the cumulative discount reward is: ; in, Indicates policy-based Q(s, a) is a Q value corresponding to an action a in a state s, is a mathematical expectation calculation symbol, is a discount factor, is a parameter of the network; represents an iteration time; represents a reward obtained by the robot in a state selecting an action; and are an initial state and an initial action, respectively; An iteration formula for maximizing cumulative discounted rewards is: ; wherein, t is a time step of iteration in a Markov decision process, represents a reward obtained by the robot in a state selecting an action, represents an optimal Q value corresponding to an action in a state , represents a maximum Q value in a state .

[0009] An apparatus for robot navigation comprises: An acquisition module is configured to acquire a target distance between the robot and a terminal position and an obstacle distance between the robot and a nearest obstacle in real time. A construction module is configured to construct a PER-D3QN algorithm and a DDPG algorithm for controlling discrete actions and continuous actions of the robot respectively, to determine an attractive potential field according to the target distance, to determine a repulsive potential field according to the obstacle distance, to construct an artificial potential field based on the attractive potential field and the repulsive potential field, and to construct a unified navigation model based on the artificial potential field by combining the PER-D3QN algorithm for discrete action navigation and the DDPG algorithm for continuous action navigation. A navigation module is configured to determine a potential energy change amount of the robot according to the artificial potential field, to take the potential energy change amount of the robot at each step as a reward function, and to guide the robot to move in a direction in which the potential energy is reduced.

[0010] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the robot navigation method.

[0011] A computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the robot navigation method when executing the program.

[0012] The robot navigation method provided by the application has the following beneficial effects: The proposed navigation method significantly improves the performance and stability of robot navigation by combining the PER-D3QN and DDPG algorithms and introducing an artificial potential field dynamic reward mechanism. This method first uses the PER-D3QN to control discrete actions, leveraging its priority sampling and dual-Q network architecture to effectively mitigate the Q-value overestimation problem. DDPG is also employed to process continuous actions, overcoming the limitations of traditional DRL algorithms in continuous control tasks. An artificial potential field is constructed by acquiring target and obstacle distances in real time. The attractive potential field accelerates approach to the target, while the repulsive potential field dynamically avoids obstacles. The potential energy change is used as a reward function, directly linked to the navigation goal (minimizing potential energy), avoiding the bias of manually designed reward functions. This hybrid discrete-continuous action control scheme is adaptive to complex environments. Combined with the smooth gradient guidance provided by the potential field, it reduces training oscillations and improves convergence efficiency. When the target distance falls below a threshold, the system autonomously determines navigation is complete, forming a closed-loop control loop. This method significantly addresses the performance issues of traditional DRL algorithms caused by Q-value overestimation, improper reward function design, and limited action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] To more clearly illustrate the embodiments of the present invention and its design, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0014] Figure 1 The figure is a flow chart of a robot navigation method provided according to an exemplary embodiment of the present invention.

[0015] Figure 2 The figure is a schematic diagram of a DQN algorithm structure provided according to an exemplary embodiment of the present invention.

[0016] Figure 3 The figure is a schematic diagram of the DDPG algorithm structure provided according to an exemplary embodiment of the present invention.

[0017] Figure 4 A schematic diagram of averaged state-action estimation value training provided by the present invention according to an exemplary embodiment.

[0018] Figure 5 The figure is a flowchart of a reward calculation provided according to an exemplary embodiment of the present invention.

[0019] Figure 6 A schematic diagram of a Markov decision process provided by the present invention according to an exemplary embodiment.

[0020] Figure 7A schematic diagram of knowledge transfer training provided according to an exemplary embodiment of the present invention.

[0021] Figure 8 A schematic diagram of PER-D3QN algorithm training is provided according to an exemplary embodiment of the present invention.

[0022] Figure 9 The figure is a schematic diagram of an execution process of PER provided according to an exemplary embodiment of the present invention.

[0023] Figure 10 The present invention provides a schematic diagram of a policy according to an exemplary embodiment. Among them, a is a schematic diagram of a static obstacle simulation scene, and b is a schematic diagram of a dynamic obstacle simulation scene.

[0024] Figure 11 The present invention is a block diagram of a robot navigation device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solution of the present invention and to be able to implement it, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0026] This paper applies the PER-D3QN algorithm to discrete robot navigation and the DDPG algorithm to continuous robot navigation. The design of the reward function is crucial throughout the training process. Given the flexibility and ease of use of artificial potential fields, this paper incorporates them into the reward function. The change in potential energy with each step of the mobile robot is used as a reward to guide the mobile robot toward the target point.

[0027] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A schematic flow chart of a robot navigation method provided by the present invention, specifically as follows Figure 1 As shown, the following steps are included: S101. Obtain the target distance between the robot and the end position and the obstacle distance to the nearest obstacle in real time.

[0029] In this step, the current state of the robot needs to be obtained in real time, including the sensing information of the lidar and the position of the robot.

[0030] For example, 24 pieces of distance information obtained uniformly in all directions by the lidar; the angle between the robot's orientation and the line connecting the robot to the target point; the target distance between the robot and the end position; the distance to the nearest obstacle detected by the lidar, etc.

[0031] S102. Construct the PER-D3QN algorithm and the DDPG algorithm to control the discrete and continuous actions of the robot respectively.

[0032] In this step, firstly, the prioritized experience replay technology PER is introduced on the basis of the dual competitive deep Q network D3QN to construct the PER-D3QN algorithm, which is applied to the discrete action navigation of the target robot.

[0033] In one embodiment, in order to realize the navigation of the robot, Double DQN, Dueling DQN and PER are combined to design the PERD3QN algorithm. First, for the DQN algorithm, such as Figure 2 As shown in Figure 2, the inclusion of a maximum operation when estimating action values ​​can cause the robot to overestimate Q-values. To address this issue, the Dual DQN was proposed, eliminating the overestimation problem by decoupling the steps of selecting the action corresponding to the target Q-value and calculating the target Q-value. In the case of the Competitive DQN, a competitive architecture is introduced into the network, which consists of a value function and an advantage function. The value is related only to the state, while the advantage is related to both the state and the action. The network's final output is a combination of value and advantage, which can effectively improve the network's convergence efficiency. Regarding PER, the temporal difference error is used as a priority for samples when storing experience. With the introduction of PER, unlike the random sampling in DQN, PER extracts samples from the experience replay buffer based on priority, which can accelerate model training.

[0034] In this step, the second aspect is to use the DDPG algorithm for robot continuous motion navigation. Among them, the DDPG algorithm structure is as follows Figure 3 As shown in the figure, the averaged state-action estimate is the average of the K previously learned Q-value estimates in the target network as the estimated value of the current action, which reduces the target approximation error and thus achieves the effect of optimizing the training process. Figure 4 The averaging process is as follows:

[0035] in, represents the discount factor, which is used to measure the importance of future rewards. Represents the reward value, Indicates status, Indicates action, represents the action-value function.

[0036] The summation of a series of discounted returns, are multiple previously learned The average value of the estimated value is calculated after a certain calculation, and the two are added together to obtain the final action estimate.

[0037] Furthermore, to further reduce Q-value estimation bias, this paper builds on the existing actor-critic network architecture of Deep Deterministic Policy Gradient (DDPG) by introducing averaged state-action estimates and Prioritized Experience Replay (PER) to construct the AVG-DNPER-DDPG algorithm for continuous action navigation in robots. Compared to DDPG, the AVG-DNPER-DDPG algorithm uses a larger amount of data for forward propagation through the Q network, which helps the agent better predict its next action. Backward propagation uses the same amount of data for updates.

[0038] S103 : determining an attractive potential field according to the target distance, determining a repulsive potential field according to the obstacle distance, and constructing an artificial potential field based on the attractive potential field and the repulsive potential field.

[0039] In this step, an artificial field potential is introduced as a reward function, and the potential energy change of the mobile robot at each step is used as a reward to guide the mobile robot towards the target point. The expression of the designed artificial potential field is:

[0040] ; in, and are the attractive potential field and the repulsive potential field, respectively. Represents the distance from the target robot to the target point, This represents the minimum distance between the target robot and an obstacle. When obstacles are known, the present invention needs to consider all obstacles near the object when calculating the repulsive potential field. However, the navigation task in this invention is based on an unknown environment. Therefore, the method used is to calculate the repulsive potential field based on the distance from the object to the nearest obstacle (i.e., the minimum distance detected by the lidar).

[0041] During robot navigation, after the robot selects and executes an action, the environment provides corresponding feedback, which is determined by a reward function to determine whether the action is good or not. Reward functions are often designed based on the environment and task. A reward is a scalar quantity, with positive values ​​representing rewards and negative values ​​representing penalties. For robot navigation problems, reward functions guide the robot's behavior, and their design is crucial to the effectiveness and efficiency of model training. Traditional reward functions typically provide a fixed positive reward upon task completion and a fixed negative reward upon task failure. Rewards are only given at the end of the task, leading to sparse rewards. To address the slowness of model training caused by traditional reward functions, this paper optimizes the reward function and proposes a new reward function form. The proposed reward function consists of two parts: the first part provides a fixed reward when the robot reaches the final state. The robot receives a large positive reward after successfully navigating to the target point. If the robot collides, navigation fails, and the reward function provides a large negative reward. The second part provides a small reward based on the change in state after the robot selects an action and reaches the next state. The present invention proposes to introduce an artificial potential field and use the potential energy change of the robot at each step as a reward to guide the robot towards the target point.

[0042] S104. Combining the PER-D3QN algorithm for discrete action navigation and the DDPG algorithm for continuous action navigation, a unified navigation model is constructed based on the artificial potential field.

[0043] Based on the above, the present invention introduces the artificial field potential to construct the reward function of the navigation model as follows: ; in, are the potential energies of the previous step and the current step, respectively, is the reward value obtained; is a positive reward value, indicating successful navigation. A negative reward value indicates navigation failure; is the target point threshold, is the collision threshold, when Less than When , it is determined that the navigation to the target point is successful; when Less than When a collision occurs, it indicates that navigation fails. In addition, and When considering the size of the robot, An additional error value is required. and There are two aspects to consider when determining the value of : 1) Should be slightly larger than , which ensures that the robot actively learns to reach the target point. 2) The magnitude of the potential field change should also be considered. If and If they are too large, the effect of the potential field will be greatly weakened, and if they are too small, the ability to guide the robot to avoid obstacles and converge to the target point will be weakened.

[0044] The reward calculation flow chart is as follows Figure 5 As shown, target and collision are the signs of reaching the end position and colliding with the obstacle respectively.

[0045] In this way, this step introduces the artificial field potential as the reward function through the above three approaches, and combines the PER-D3QN algorithm and the AVG-DNPER-DDPG algorithm to build a navigation model.

[0046] S105. Determine the potential energy change of the robot based on the artificial field potential, use the potential energy change of each step of the robot as a reward function, and guide the robot to move in a direction with reduced potential energy.

[0047] In this step, after the unified navigation model is constructed, the unified navigation model is also trained.

[0048] For example, the robot navigation process is converted into a Markov decision process, and the training of the navigation model is completed based on the Markov decision process.

[0049] Among them, Markov Decision Processes (MDP) such as Figure 6 As shown in the figure, the agent in this invention is a robot, and the environment is the robot's environment. The robot's navigation process is converted into a Markov decision process (MDP). The robot obtains the reward of the MDP at each time step based on discrete or continuous actions, and calculates the cumulative discounted reward during the MDP process to maximize the cumulative discounted reward and update the parameters of the current unified navigation model; this cumulative discounted reward is used to quantify the quality of the path.

[0050] MDP can use quintuple Indicates that and They are the state space and action space of the system respectively; represents the reward function; Indicates that you are selecting an action The state transition probability of the system afterwards; is a discount factor that reflects the diminishing impact of current decisions on the future.

[0051] In one embodiment, the current state of the robot is input into the current value network of the navigation model, and the output of the current value network is selected according to the maximum Q value. The method comprises the following steps: taking the corresponding current action and obtaining the reward and the state at the next moment according to the current action; treating the current state, current action and state at the next moment as a Markov decision process of one time step; obtaining the reward of the Markov decision process of each time step, and calculating the cumulative discounted reward to update the parameters of the current value network to maximize the cumulative discounted reward; updating the target value network according to the parameters of the current value network at every preset time step, and determining the Q value of the current value network and the target value network; calculating the loss function through the Q value of the current value network and the target value network, and then adjusting the parameters of the current value network according to the back propagation of the loss function to obtain the trained navigation model.

[0052] Specifically, in the dual network structure that combines the current value network and the target value network, the current state Q value will be affected by the Q value of the next state, which will aggravate the instability of the overall training process. Among them, the architecture of the current value network and the target value network is the same. In order to alleviate this problem, while using the current network parameters to estimate the next state Q value, these parameters will be updated regularly. The updated new network will reduce the instability of the target and time, thereby effectively controlling the error. That is, to avoid the same network weights Under these conditions, the predicted value and the target value have a large difference. The target network is treated as an independent network to calculate the target value. The error is reduced by calculating the loss function of the target network and the current value network and minimizing the loss function, so that the network's prediction gradually approaches the actual situation. The calculation formula of the loss function is:

[0053] ; in, is the reward for the current state, is a discount factor, is the current value of the network in the network parameter The Q value of action a based on state s, The network parameters are the target value network Based on the status Next action Q value.

[0054] Based on the loss function, the target value network is updated, and the update formula is: ; ; in, and are the parameters of the current network, and are the parameters of the target network, is the update coefficient, which is used to control the magnitude of the update.

[0055] In one possible implementation, the mapping from states to actions in an MDP is based on a policy. The best strategy is the one that gets the highest reward from any initial state. , the Q value is defined as the state Take action The expected value of the cumulative discount reward is calculated as follows:

[0056] ; in, Indicates policy-based The Q value corresponding to action a in state s is, is the calculation symbol of mathematical expectation, is a discount factor that measures the importance of future rewards, are the parameters of the network; represents the iteration time; Indicates that the agent is in state The reward for selecting an action in and They are the initial state and initial action respectively.

[0057] The goal of MDP is to find a strategy that maximizes the cumulative discounted reward. Therefore, the Q-learning algorithm can find a near-optimal strategy by iterating to the optimal Q value through the Bellman formula. That is, the iterative formula for maximizing the cumulative discounted reward is:

[0058] ; Where t is the iterative time step in the Markov decision process, Indicates that the agent is in state The reward for selecting an action in Representation and Status The best Q value corresponding to the action in, Indicates status The maximum Q value when

[0059] In another possible implementation, through a deterministic strategy Decision-making and action , and fit the state-action function through the value network. The objective function can be defined as the cumulative return with a discount factor, as follows: ; is the strategy parameter, Indicates rewards at different times, is the discount factor.

[0060] In this invention, the state information consists of LiDAR sensing information and the robot's position (the defined state space includes: 24 pieces of distance information uniformly acquired in all directions by LiDAR; the angle between the robot's orientation and the line connecting the robot to the target point; the distance between the robot and the target point; the minimum distance detected by LiDAR; and the index corresponding to the minimum distance detected by LiDAR). The action space is composed of displacements in five different directions (large left turn, small left turn, forward, small right turn, and large right turn). By designing these state and action spaces, the goal is to find a near-optimal strategy through iterative network training under the guidance of a reward function, thereby obtaining higher rewards and improving the success rate of navigation.

[0061] In addition, a knowledge transfer training method that uses prior knowledge can be introduced for the training of navigation models to improve the convergence speed of the algorithm in relatively complex environments, such as Figure 7 As shown in the figure, the model usually needs to be trained from scratch when facing different environments. However, DRL training is very time-consuming, and the model may have difficulty converging in relatively complex environments. To address this problem, the present invention proposes a knowledge transfer training method. After the robot is trained in a relatively simple environment, its model and parameters are transferred to a similar complex environment and training starts from this point, thereby accelerating training in complex environments.

[0062] Specifically, the navigation model is first trained in a relatively simple environment, where it is easier for the model to learn some basic navigation knowledge and patterns. The network parameters of the trained navigation model are then extracted and used as training parameters. (The network parameters of the model trained in the simple environment are extracted. The parameters contain the information learned by the model in the simple environment and are used as parameters for subsequent training.) The navigation model is then transferred to a more complex environment and trained again using these trained parameters as the initial weights of the network. The goal is to leverage the knowledge and parameters learned in the simple environment to help the model learn and adapt faster and better in complex environments.

[0063] In another embodiment, the PER-D3QN algorithm is used as a navigation model for discrete robot actions, and its training process can be as follows: Figure 8 As shown. The robot inputs the state information into the current value network and selects the maximum Q value according to the output of the current value network. The robot performs the action and gets rewarded. Then, the current state , the selected action the state at the next time and the reward The samples are put into the experience replay buffer as experiences. During the network training process, the sample data is selected according to the sample priority, and the parameters of the current value network are copied to the target value network every n time steps. Finally, the Q values of the current and target value networks are used to calculate the loss function. After sufficient training, the parameters of the current value network will be close to the optimal. The execution process of PER is shown in FIG. 3. Figure 9

[0064] In another embodiment, the AVG-DNPER-DDPG algorithm needs more accurate parameters to support the robot to take more accurate navigation paths as a navigation model of continuous actions of the robot. Therefore, the AVG-DNPER-DDPG algorithm needs to be further trained.

[0065] In an implementable manner, in the critic's current value network, the iterative update of the network parameters is based on the minimum value of the loss function variance, and the mean square error is as follows: ; is the network parameter, represents the number of samples, is the expected value, is the actually calculated value.

[0066] For the actor's current value network, the network parameters are iteratively updated according to the loss gradient of the policy, and the specific process is as follows: ; is the policy parameter, is the number of samples, represents a certain quantity related to the policy and the Q value, represents the gradient of the policy.

[0067] ​Another possible implementation involves adding a critic network to the existing actor-critic network architecture of DDPG to assist in training. During each update, the smallest Q-value from the two critic networks is used as the action estimate, thereby reducing the probability of local optima. A dual-actor network is then introduced to mitigate the underestimation caused by the dual-critic network. The action with the highest value from the two actor networks is selected for update to stabilize the algorithm's training process. By analyzing the causes of target approximation error in the policy network, this method averages the estimates of multiple critic networks for the same action and uses the average as the action estimate to reduce the impact of this error. Traditional PER tends to collect empirical data with large absolute values ​​of temporal difference error (TD-error), but this approach results in inadequate sample partitioning, leading to slower algorithm training. Empirical data with small absolute values ​​of TD-error but high reward can also improve the robot's learning efficiency, but this data is not given high priority, making it less likely to be collected. The present invention divides the sample priority by the sum of the absolute values ​​of two TD-errors and the current action reward generated by two critic networks, and processes the two value functions in the dual critic network through the averaging method, and then further optimizes the algorithm by combining the dual actor network.

[0068] For example, a dual actor-dual critic network architecture is introduced to obtain the minimum Q value of the two critic networks at each update as the estimated Q value of the action. The calculation formula is: ; ; ; Indicates at time Rewards, It is the state of the next moment. Indicates According to the strategy parameters Get the action, and The action value functions, i.e., Q values, calculated by two different critic networks are respectively. By calculating the value estimates under two different paths and , then take the minimum value , which can reduce the overestimation error and make the estimation of future actions more accurate.

[0069] The dual critic network produces the minimum Q-value to reduce the overestimation error, while the dual actor network selects the action with the highest value among the two actions under the minimum Q-value to reduce the impact of Q-value underestimation and improve the robustness of the network structure.

[0070] On the basis of the above, PER is combined with training. The core idea of ​​PER is to train the neural network in DQN by randomly sampling a certain number of learning experiences, rather than relying solely on the last action information. In the reinforcement learning environment, in state The agent under Act on the environment and get feedback , current status It will also transform into a new state The process information is in the form of a four-tuple > in the experience replay buffer, which represents the agent's learning experience. The agent's experiences over a continuous period of time are interrelated. Random sampling from the experience replay buffer reduces the correlation between required learning experiences, helping the agent learn more comprehensively and effectively.

[0071] Specifically, the current state , Current Action , the state at the next moment and rewards As experience data, it is put into the experience replay buffer pool; then training samples are extracted from the experience replay buffer pool. Considering that directly calculating the priority of all sample data in the experience pool will seriously affect the efficiency of the algorithm, the present invention divides the sampling process into two steps: first, randomly sampling from the experience replay buffer pool data; where n represents the number of priority samples required by the network. If n is 1, it indicates that the current sampling method is uniform sampling, and size represents the size of the batch data. Secondly, from n samples are collected from the data according to the priority as the training samples required by the current value network; the AVG-DNPER-DDPG algorithm is trained by the training samples, and the parameters of the AVG-DNPER-DDPG algorithm after training are determined.

[0072] The priority of the experience data is determined based on the sum of the absolute value of the bi-temporal difference error generated by the two critic networks in the AVG-DNPER-DDPG algorithm and the reward of the current state, as follows: and are the state and action at time t, are network parameters, and are the difference and the sum of rewards between the two critic networks, is the reward for the current state, is the updated critic network in state and actions Next, based on the parameters The calculated action value estimate, The critic network before the update is in state and actions Next, based on the parameters The calculated action value estimate, is a specific constant, Prioritize empirical data.

[0073] S106: When the target distance is less than a preset threshold, determine that the robot has completed navigation.

[0074] In this step, an obstacle environment was constructed based on a three-dimensional simulator to verify robot navigation.

[0075] Two 3D simulators were built in Gazebo. Obstacle environment, such as Figure 10 As shown: Figure 10 In the figure a, it is a static obstacle environment (scene one). Figure 10 Figure b shows a dynamic obstacle environment (Scene 2). A cylinder with a radius of 0.15m represents an obstacle. The cylinder in the dynamic obstacle environment moves counterclockwise around the center of the environment at a constant speed. The square frame represents a barrier; hitting the barrier is equivalent to a collision with the obstacle. The square within the barrier is the target point. The line within the barrier represents the laser beam emitted by the LiDAR (Light Detection and Ranging) system.

[0076] During the training simulation, the robot starts from different starting points and navigates to randomly generated target points in an open area of ​​the environment. Through the interaction between the robot and the environment and the constraints of the navigation rules, the robot's decision-making quality is continuously improved. Finally, the navigation path is determined based on the optimized decision.

[0077] The proposed method first controls discrete actions using PER-D3QN, leveraging its priority sampling and dual-Q network architecture to effectively mitigate the Q-value overestimation problem. Continuous actions are then processed using DDPG, overcoming the limitations of traditional DRL algorithms in continuous control tasks. An artificial potential field is constructed by acquiring real-time target and obstacle distances. The attractive potential field accelerates approach to the target, while the repulsive potential field dynamically avoids obstacles. The change in potential energy is used as a reward function, directly linking it to the navigation goal (minimizing potential energy), thus avoiding the bias of manually designed reward functions. This hybrid discrete-continuous action control scheme is adaptive to complex environments. Combined with the smooth gradient guidance provided by the potential field, it reduces training oscillations and improves convergence efficiency. When the target distance falls below a threshold, the system autonomously determines that navigation is complete, forming a closed-loop control loop. This significantly addresses the performance issues of traditional DRL algorithms caused by Q-value overestimation, improper reward function design, and limited action space.

[0078] Figure 11 A block diagram of a robot navigation device provided by the present invention, specifically as follows Figure 11 Shown, including: The acquisition module 201 is used to obtain the target distance between the robot and the end position and the obstacle distance to the nearest obstacle in real time.

[0079] Construction module 202 is used to construct the PER-D3QN algorithm and the DDPG algorithm to control the discrete and continuous movements of the robot respectively; determine the attractive potential field according to the target distance, determine the repulsive potential field according to the obstacle distance, and construct an artificial potential field based on the attractive potential field and the repulsive potential field; combine the PER-D3QN algorithm for discrete movement navigation and the DDPG algorithm for continuous movement navigation, and construct a unified navigation model based on the artificial potential field.

[0080] The navigation module 203 is used to determine the change in the robot's potential energy based on the artificial field potential, use the change in the potential energy of each step of the robot as a reward function, and guide the robot to move in the direction of decreasing potential energy; when the target distance is less than a preset threshold, it is determined that the robot has completed navigation.

[0081] Using this device, discrete actions are first controlled using PER-D3QN, leveraging its priority sampling and dual-Q network architecture to effectively mitigate the Q-value overestimation problem. Continuous actions are simultaneously processed using DDPG, overcoming the limitations of traditional DRL algorithms in continuous control tasks. An artificial potential field is constructed by acquiring target and obstacle distances in real time. The attractive potential field accelerates approach to the target, while the repulsive potential field dynamically avoids obstacles. The change in potential energy is used as a reward function, directly linking it to the navigation goal (minimizing potential energy), thus avoiding the bias of manually designed reward functions. This hybrid discrete-continuous action control scheme is adaptive to complex environments. Combined with the smooth gradient guidance provided by the potential field, it reduces training oscillations and improves convergence efficiency. When the target distance falls below a threshold, the system autonomously determines that navigation is complete, forming a closed-loop control loop. This significantly addresses the performance issues of traditional DRL algorithms caused by Q-value overestimation, improper reward function design, and limited action space.

[0082] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provided are methods for robot navigation.

[0083] This specification also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Provided are methods for robot navigation.

[0084] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0086] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0088] It should be noted that the above detailed description and examples, while indicating certain embodiments of the application, are given by way of illustration only, and are not by way of limitation of the application. Thus, while the present application has been described in detail with respect to its specific embodiments, it will be apparent that modifications and variations of the detailed implementation can be effected by those skilled in the art, without departing from the spirit and scope of the application. Accordingly, it should be noted that the description of the application is intended for purposes of illustration only and should not be construed in any way whatsoever as limiting the application. No reference to a particular alternative is therefore to be construed as a disavowal of the generic set of alternatives that the reference describes or relates to.

Claims

1. A robot navigation method, characterized in that: include: Obtain the target distance between the robot and the end position and the obstacle distance to the nearest obstacle in real time; Construct the PER-D3QN algorithm and DDPG algorithm to control the robot's discrete and continuous actions respectively; Determine an attractive potential field according to the target distance, determine a repulsive potential field according to the obstacle distance, and construct an artificial potential field based on the attractive potential field and the repulsive potential field; Combining the PER-D3QN algorithm for discrete motion navigation with the DDPG algorithm for continuous motion navigation, a unified navigation model is constructed based on artificial potential fields. Determine the robot's potential energy change based on the artificial field potential, use the potential energy change of each step of the robot as a reward function, and guide the robot to move in the direction of decreasing potential energy; When the target distance is less than a preset threshold, it is determined that the robot has completed navigation.

2. A robot navigation method according to claim 1, characterized in that: The introduction of artificial field potential as a reward function includes: The artificial potential field expression is: ； in, and are the attractive potential field and the repulsive potential field, respectively. Represents the distance from the target robot to the target point, Characterizes the minimum distance between the target robot and the obstacle; The reward function is constructed based on the potential energy change of the artificial field potential, and the reward function expression is determined as: ； in, are the potential energies of the previous step and the current step, respectively, is the reward value obtained; is a positive reward value, indicating successful navigation. A negative reward value indicates navigation failure; is the target point threshold, is the collision threshold, when Less than When , it is determined that the navigation to the target point is successful; when Less than , it indicates that a collision has occurred and navigation has failed.

3. A robot navigation method according to claim 2, characterized in that: After the unified navigation model is constructed, the unified navigation model is further trained, including: The robot navigation process is converted into a Markov decision process (MDP). The robot obtains the reward of the Markov decision process at each time step based on discrete or continuous actions, and calculates the cumulative discounted reward in the MDP process to maximize the cumulative discounted reward to update the parameters of the current unified navigation model; the cumulative discounted reward is used to quantify the quality of the path.

4. A robot navigation method according to claim 3, characterized in that: The calculation formula for cumulative discount rewards is: ； in, Indicates policy-based The Q value corresponding to action a in state s is, is the calculation symbol of mathematical expectation, is a discount factor, are the parameters of the network; represents the iteration time; Indicates that the robot is in state The reward for selecting an action in and are the initial state and initial action respectively; The iterative formula for maximizing the cumulative discounted reward is: ； in, t is the iterative time step in the Markov decision process, Indicates that the robot is in state The reward for selecting an action in Representation and Status The best Q value corresponding to the action in, Indicates status The maximum Q value when 5. A robot navigation device, characterized in that: include: The acquisition module is used to obtain the target distance between the robot and the end position and the obstacle distance to the nearest obstacle in real time; A construction module is used to construct a PER-D3QN algorithm and a DDPG algorithm to control discrete and continuous actions of the robot respectively; determine an attractive potential field according to the target distance, determine a repulsive potential field according to the obstacle distance, and construct an artificial potential field based on the attractive potential field and the repulsive potential field; Combining the PER-D3QN algorithm for discrete motion navigation with the DDPG algorithm for continuous motion navigation, a unified navigation model is constructed based on artificial potential fields. The navigation module is used to determine the change in the robot's potential energy based on the artificial field potential, use the change in the potential energy of each step of the robot as a reward function, and guide the robot to move in the direction of decreasing potential energy; when the target distance is less than a preset threshold, the robot is determined to have completed navigation.

6. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

Images