A method, apparatus, computer device, and storage medium for robot navigation.

Abstract

This invention provides a method, apparatus, computer device, and storage medium for robot navigation, belonging to the field of robot navigation. The method includes: real-time acquisition of the target distance between the robot and its destination position, and the obstacle distance to the nearest obstacle; constructing two algorithms to control the robot's discrete and continuous actions respectively; determining the attractive and repulsive potential fields respectively, and constructing an artificial potential field based on the attractive and repulsive potential fields; combining the two algorithms to construct a unified navigation model based on the artificial potential field; determining the robot's potential energy change based on the artificial potential field, using the potential energy change of the robot at each step as a reward function to guide the robot to move in the direction of decreasing potential energy; and determining that the robot has completed navigation when the target distance is less than a preset threshold. Thus, this discrete-continuous action hybrid control mode can adapt to complex environments and provide the robot with a precise navigation path.

Description

Technical Field

[0001] This invention belongs to the field of robot navigation, and specifically relates to a method, apparatus, computer equipment, and storage medium for robot navigation. Background Technology

[0002] Autonomous navigation robots are intelligent robots capable of completing tasks autonomously in unknown environments without relying on external instructions. Research on autonomous navigation robots is an interdisciplinary field combining artificial intelligence and automation, and it can play a vital role in many areas such as healthcare, transportation, and environmental protection. For example, autonomous navigation robots can handle tasks such as delivering medicine and meals in hospitals, thereby reducing the workload of medical staff; in the transportation sector, autonomous navigation robots can assist in tasks such as waste collection and city patrols, becoming an important tool for urban management.

[0003] Autonomous navigation of robots is a crucial capability that ensures a platform reaches its destination from a starting point in an environment with numerous obstacles without collisions. Traditional navigation methods typically involve Simultaneous Localization and Mapping (SLAM), trajectory planning, and tracking control. However, SLAM mapping is time-consuming and requires high accuracy and density from LiDAR, making autonomous navigation difficult even in environments with sparse distance information and unobstructed maps. Therefore, end-to-end online learning navigation methods based on Deep Reinforcement Learning (DRL) have attracted widespread attention. However, the performance of traditional DRL algorithms (such as Deep Q-Networks (DQN)) is often affected by Q-value overestimation and the reward function, and they are limited to discrete control tasks, resulting in extremely poor algorithm performance that severely impacts the accuracy and efficiency of robot navigation, and exhibits poor stability. Summary of the Invention

[0004] To address the aforementioned problems, the present invention provides a method, apparatus, computer device, and storage medium for robot navigation.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for robot navigation, comprising:

[0007] Real-time acquisition of the target distance between the robot and the endpoint, and the distance to the nearest obstacle;

[0008] The PER-D3QN algorithm and the DDPG algorithm are constructed to control the discrete and continuous movements of the robot, respectively.

[0009] An attractive potential field is determined based on the target distance, and a repulsive potential field is determined based on the obstacle distance. An artificial potential field is constructed based on the attractive and repulsive potential fields.

[0010] By combining the PER-D3QN algorithm for discrete motion navigation and the DDPG algorithm for continuous motion navigation, a unified navigation model is constructed based on an artificial potential field.

[0011] The potential energy change of the robot is determined based on the artificial field potential, and the potential energy change of the robot at each step is used as a reward function to guide the robot to move in the direction of decreasing potential energy.

[0012] If the target distance is less than a preset threshold, the robot is confirmed to have completed navigation.

[0013] Optionally, the introduction of an artificial potential as a reward function includes:

[0014] The expression for the artificial potential field is:

[0015] ;

[0016] in, and These are the attractive potential field and the repulsive potential field, respectively. Characterizes the distance from the target robot to the target point. Characterizes the minimum distance between the target robot and the obstacle;

[0017] A reward function is constructed based on the potential energy change of the artificial field potential, and the expression of the reward function is determined as follows:

[0018] ;

[0019] in, These are the potential energies from the previous step and the current step, respectively. The reward value obtained; A positive reward value indicates successful navigation. A negative reward value indicates navigation failure. The threshold for the target point. For the collision threshold, when Less than When the navigation to the target point is successfully reached; Less than When this occurs, it indicates a collision and navigation failure.

[0020] Optionally, after constructing the unified navigation model, the unified navigation model is also trained, including:

[0021] The robot navigation process is transformed into a Markov Decision Process (MDP). The robot obtains the reward of the MDP at each time step based on discrete or continuous actions, and calculates the cumulative discount reward in the MDP process to maximize the cumulative discount reward to update the parameters of the current unified navigation model; the cumulative discount reward is used to quantify the quality of the path.

[0022] Optionally, the formula for calculating cumulative discount rewards is as follows:

[0023] ;

[0024] in, Indicates policy-based The Q-value corresponding to action a in state s. It is the symbol for calculating mathematical expectation. It is a discount factor. These are network parameters; Indicates the iteration time; Indicates the robot's state The reward obtained by choosing an action; and These are the initial state and the initial action, respectively.

[0025] The iterative formula for maximizing cumulative discount rewards is:

[0026] ;

[0027] in, t For the time step of the iteration in the Markov decision process, Indicates the robot's state The reward obtained by choosing an action. Representation and State The optimal Q value corresponding to the action in the middle, Representing state The maximum Q value at that time.

[0028] A device for robot navigation, comprising:

[0029] The acquisition module is used to acquire the distance between the robot and the target position at the destination and the distance between the robot and the nearest obstacle in real time;

[0030] The module is used to construct the PER-D3QN algorithm and the DDPG algorithm to control the robot's discrete and continuous actions, respectively; to determine the attractive potential field based on the target distance and the repulsive potential field based on the obstacle distance; and to construct an artificial potential field based on the attractive and repulsive potential fields; and to construct a unified navigation model based on the artificial potential field by combining the PER-D3QN algorithm for discrete action navigation and the DDPG algorithm for continuous action navigation.

[0031] The navigation module is used to determine the robot's potential energy change based on the artificial field potential, and uses the potential energy change of the robot at each step as a reward function to guide the robot to move in the direction of decreasing potential energy; when the target distance is less than a preset threshold, the robot is determined to have completed navigation.

[0032] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for robot navigation.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method for robot navigation.

[0034] The robot navigation method provided by this invention has the following beneficial effects:

[0035] The navigation method proposed in this invention, by combining the PER-D3QN and DDPG algorithms and introducing a dynamic reward mechanism based on an artificial potential field, significantly improves the performance and stability of robot navigation. This method first uses PER-D3QN to control discrete actions, effectively mitigating Q-value overestimation through its priority sampling and dual-Q network structure. Simultaneously, DDPG is employed to handle continuous actions, overcoming the limitations of traditional DRL algorithms in continuous control tasks. An artificial potential field is constructed by acquiring the target distance and obstacle distance in real time. The attractive potential field accelerates the approach to the target, while the repulsive potential field dynamically avoids obstacles. The change in potential energy is used as the reward function, directly correlated with the navigation target (minimizing potential energy), avoiding biases inherent in manually designed reward functions. This discrete-continuous action hybrid control mode can adapt to complex environments. Combined with the smooth gradient guidance provided by the potential field, it reduces training oscillations and improves convergence efficiency. When the target distance is less than a threshold, the system autonomously determines navigation completion, forming a closed-loop control. This method significantly addresses the performance problems of traditional DRL algorithms caused by Q-value overestimation, improper reward function design, and limited action space. Attached Figure Description

[0036] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic flowchart of a robot navigation method provided by the present invention according to an exemplary embodiment.

[0038] Figure 2This is a schematic diagram of a DQN algorithm structure provided by the present invention according to an exemplary embodiment.

[0039] Figure 3 This is a schematic diagram of a DDPG algorithm structure provided by the present invention according to an exemplary embodiment.

[0040] Figure 4 This is a schematic diagram of an averaged state-action estimate training method provided by the present invention according to an exemplary embodiment.

[0041] Figure 5 This is a schematic diagram of a reward calculation process provided by the present invention according to an exemplary embodiment.

[0042] Figure 6 This is a schematic diagram of a Markov decision process provided by the present invention according to an exemplary embodiment.

[0043] Figure 7 This is a schematic diagram of a knowledge transfer training method provided by the present invention according to an exemplary embodiment.

[0044] Figure 8 This is a schematic diagram illustrating the training of the PER-D3QN algorithm according to an exemplary embodiment of the present invention.

[0045] Figure 9 This is a schematic diagram illustrating the execution process of a PER according to an exemplary embodiment of the present invention.

[0046] Figure 10 This is a schematic diagram of a policy according to an exemplary embodiment of the present invention. Wherein, a is a schematic diagram of a static obstacle simulation scene, and b is a schematic diagram of a dynamic obstacle simulation scene.

[0047] Figure 11 This is a block diagram of a robot navigation device according to an exemplary embodiment of the present invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0049] This invention employs the PER-D3QN algorithm for discrete motion navigation of robots and the DDPG algorithm for continuous motion navigation. The design of the reward function is crucial throughout the training process. Considering the flexibility and ease of use of artificial potential fields, this invention incorporates artificial potential fields into the reward function. The potential energy change of the mobile robot at each step is used as the reward to guide the robot towards the target point.

[0050] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0051] Figure 1 This invention provides a flowchart illustrating a robot navigation method, specifically as follows: Figure 1 As shown, it includes the following steps:

[0052] S101. Real-time acquisition of the target distance between the robot and the endpoint position, and the obstacle distance between the robot and the nearest obstacle.

[0053] In this step, it is necessary to obtain the robot's current status in real time, including the sensor information of the LiDAR and the robot's position.

[0054] For example, 24 distance information lines uniformly obtained by LiDAR in all directions; the angle between the robot's orientation and the line connecting the robot to the target point; the target distance between the robot and the endpoint; the distance to the nearest obstacle detected by LiDAR, etc.

[0055] S102. Construct the PER-D3QN algorithm and the DDPG algorithm to control the robot's discrete and continuous actions, respectively.

[0056] In this step, firstly, the Priority Experience Replay technique (PER) is introduced on the basis of the Dual Competition Deep Q-Network (D3QN) to construct the PER-D3QN algorithm, which is then applied to the discrete motion navigation of the target robot.

[0057] In one embodiment, to achieve robot navigation, the PERD3QN algorithm is designed by combining Double DQN, Dueling DQN, and PER. First, for the DQN algorithm, as... Figure 2 As shown, the inclusion of maximum computation in action value estimation can cause the robot to overestimate its Q-value. To address this issue, a dual DQN is proposed, eliminating overestimation by decoupling the steps of selecting the action corresponding to the target Q-value and calculating the target Q-value. In the case of competitive DQN, a competitive architecture is introduced into the network, comprising a value function part and an advantage function part. The value is only related to the state, while the advantage is related to both the state and the action. The network's final output is a combination of value and advantage, effectively improving the network's convergence efficiency. Regarding PER, temporal difference error is used as the priority of samples when storing experience. Unlike random sampling in DQN, PER extracts samples from the experience replay buffer according to priority, accelerating model training.

[0058] In this step, the second aspect involves using the DDPG algorithm for continuous robot motion navigation.

[0059] The DDPG algorithm structure is as follows: Figure 3 As shown; the averaged state-action estimate uses the average of K previously learned Q-value estimates in the target network as the estimate of the current action to reduce target approximation error and thus optimize the training process, as... Figure 4 As shown. The averaging process is as follows:

[0060]

[0061] in, This represents a discount factor used to measure the importance of future rewards. Indicates the reward value. Indicates state, Indicates an action, This represents the action value function.

[0062] Summing over a series of discounted returns, It is multiple previously learned in the target network The average of the estimated values ​​is calculated, and the two are added together to obtain the final action estimate.

[0063] Furthermore, to further reduce Q-value estimation bias, this invention introduces Averaged State-Action Estimation (ASE) and Priority Experience Replay (PER) techniques into the existing actor-commentator network architecture of Deep Deterministic Policy Gradient (DDPG), constructing the AVG-DNPER-DDPG algorithm for continuous motion navigation of robots. Compared to DDPG, the AVG-DNPER-DDPG algorithm uses a larger amount of data for forward propagation through the Q-network, which helps the agent better predict the next action, while maintaining the same amount of data for backpropagation updates.

[0064] S103. Determine the attractive potential field based on the target distance, determine the repulsive potential field based on the obstacle distance, and construct an artificial potential field based on the attractive and repulsive potential fields.

[0065] In this step, an artificial potential field is introduced as the reward function, using the potential energy change of the mobile robot at each step as the reward to guide the robot toward the target point. The expression for this artificial potential field is:

[0066] ;

[0067] in, and These are the attractive potential field and the repulsive potential field, respectively. Characterizes the distance from the target robot to the target point. The minimum distance between the target robot and the obstacle is characterized. When the obstacles are known, this invention needs to consider all obstacles near the object when calculating the repulsive potential field. However, the navigation task in this invention is based on an unknown environment. Therefore, the method used is to calculate the repulsive potential field based on the distance from the object to the nearest obstacle (i.e., the minimum distance detected by the lidar).

[0068] During robot navigation, after the robot selects and executes an action, the environment provides corresponding feedback information based on a reward function to determine whether the action was good or bad. The reward function is often designed according to the environment and the task. The reward is a scalar, with positive values ​​representing rewards and negative values ​​representing penalties. For robot navigation, the reward function guides the robot's behavior, and the robot's design is crucial to the effectiveness and efficiency of model training. With traditional reward functions, the robot typically receives a fixed positive reward value when the task is completed, and a fixed negative reward value when the task fails. In this respect, rewards are only given at the end of the task, leading to a sparse reward problem. To address the slow model training caused by traditional reward functions, this invention aims to optimize the reward function by proposing a new form. The proposed reward function consists of two parts: the first part is a fixed reward given when the robot reaches the final state. The robot receives a large positive reward after successfully navigating to the target point. When the robot collides, navigation fails, and the reward function gives a large negative reward. The second part is a small reward value given based on the state change after the robot selects an action and reaches the next state. This invention proposes to introduce an artificial potential field and use the robot's potential energy change at each step as a reward to guide the robot toward the target point.

[0069] S104. Combining the PER-D3QN algorithm for discrete motion navigation and the DDPG algorithm for continuous motion navigation, a unified navigation model is constructed based on an artificial potential field.

[0070] Based on the above, this invention introduces the following reward function for constructing the navigation model using an artificial field potential:

[0071] ;

[0072] in, These are the potential energies from the previous step and the current step, respectively. The reward value obtained; A positive reward value indicates successful navigation. A negative reward value indicates navigation failure. The threshold for the target point. For the collision threshold, when Less than When the navigation to the target point is successfully reached; Less than When this occurs, it indicates a collision and navigation failure. Furthermore, in determining... and When doing so, the size of the robot itself needs to be considered. An error value also needs to be added. For and The value of can be determined from two aspects: 1) It should be slightly larger This ensures that the robot actively learns to reach the target point. 2) The magnitude of the potential field change should also be considered. If and If the potential field is too large, its effect will be greatly weakened; if it is too small, its ability to guide the robot to avoid obstacles and converge to the target point will be weakened.

[0073] The flowchart for reward calculation is as follows: Figure 5 As shown, the target and collision are indicators of reaching the endpoint and colliding with an obstacle, respectively.

[0074] Thus, this step introduces an artificial field potential as a reward function through the above three approaches, and constructs a navigation model by combining the PER-D3QN algorithm and the AVG-DNPER-DDPG algorithm.

[0075] S105. Determine the potential energy change of the robot based on the artificial field potential, and use the potential energy change of the robot at each step as a reward function to guide the robot to move in the direction of decreasing potential energy.

[0076] In this step, after constructing the unified navigation model, the unified navigation model is also trained.

[0077] For example, the robot navigation process is transformed into a Markov decision process, and the navigation model is trained based on the Markov decision process.

[0078] Among them, Markov Decision Processes (MDPs) are as follows: Figure 6 As shown in the figure, the agent in this invention is a robot, and the environment is the environment in which the robot is located. The robot navigation process is transformed into a Markov Decision Process (MDP). The robot obtains the reward of the MDP for each time step based on discrete or continuous actions, and calculates the cumulative discounted reward in the MDP process to maximize the cumulative discounted reward to update the parameters of the current unified navigation model; this cumulative discounted reward is used to quantify the quality of the path.

[0079] MDP can be used with quintuples. It means that among them and These are the system's state space and action space, respectively. Represents the reward function; Indicates selecting an action The subsequent state transition probability of the system; It is a discount factor that reflects the decreasing impact of current decisions on the future.

[0080] In one embodiment, the robot's current state is input into the current value network of the navigation model, and the maximum Q value is selected based on the output of the current value network. The corresponding current action is determined, and a reward and the state at the next moment are obtained based on the current action. The current state, current action, and state at the next moment are treated as a Markov decision process with a time step. The reward of the Markov decision process at each time step is obtained, and the cumulative discount reward is calculated to maximize the cumulative discount reward and update the parameters of the current value network. The target value network is updated according to the parameters of the current value network every preset time step, and the Q-values ​​of the current value network and the target value network are determined. The loss function is calculated using the Q-values ​​of the current value network and the target value network, and then the parameters of the current value network are adjusted by backpropagation according to the loss function to obtain the trained navigation model.

[0081] Specifically, in the dual-network structure combining the current value network and the target value network, the Q-value of the current state is affected by the Q-value of the next state, which exacerbates the instability during the overall training process. The current value network and the target value network have the same architecture. To mitigate this problem, while using the current network parameters to predict the Q-value of the next state, these parameters are periodically updated. The updated network reduces the instability of the target and time, thereby effectively controlling the error. That is, to avoid the same network weights... If the predicted value differs significantly from the target value, the target network is treated as an independent network to calculate the target value. By calculating the loss function of both the target network and the current value network, and minimizing this loss function, the error is reduced, allowing the network's predictions to gradually approach the true situation. The formula for this loss function is:

[0082] ;

[0083] in, It is the reward for the current state. It is a discount factor. For the current value of the network parameters The Q-value of action a based on state s. For the target value network in network parameters State-based The following action The Q value.

[0084] The target value network is updated based on this loss function, and the update formula is as follows:

[0085] ;

[0086] ;

[0087] in, and These are the parameters of the current network. and These are the parameters of the target network. It is the update coefficient, used to control the magnitude of the update.

[0088] In one feasible approach, the mapping from state to action in an MDP is policy-based. The best strategy is to obtain the highest reward from any initial state. Given a strategy... The Q value is defined as the state... Take action The expected value of the cumulative discount reward is calculated using the following formula:

[0089] ;

[0090] in, Indicates policy-based The Q-value corresponding to action a in state s. It is the symbol for calculating mathematical expectation. It is a discount factor used to measure the importance of future rewards. These are network parameters; Indicates the iteration time; Indicates the agent's state The reward obtained by choosing an action; and These are the initial state and the initial action, respectively.

[0091] The goal of MDP is to find a strategy that maximizes the cumulative discount reward. Therefore, the Q-learning algorithm can find a near-optimal strategy by iterating to the optimal Q-value using the Bellman formula; that is, the iterative formula for maximizing the cumulative discount reward is:

[0092] ;

[0093] Where t is the time step of the iteration in the Markov decision process. Indicates the agent's state The reward obtained by choosing an action. Representation and State The optimal Q value corresponding to the action in the middle, Representing state The maximum Q value at that time.

[0094] In another feasible approach, through a deterministic strategy Make a decision and take action The state-action function is fitted through a value network, and the objective function can be defined as the cumulative return with a discount factor, as follows:

[0095] ;

[0096] These are strategy parameters. This indicates rewards at different times. It is a discount factor.

[0097] In this invention, the state information consists of the sensing information from the LiDAR and the robot's position (the defined state space includes: 24 distance information lines uniformly obtained by LiDAR in all directions; the angle between the robot's orientation and the line connecting the robot to the target point; the distance between the robot and the target point; the minimum distance detected by LiDAR; and the index corresponding to the minimum distance detected by LiDAR). The action space consists of displacements in five different directions (large left turn, small left turn, forward, small right turn, and large right turn). By designing the above state and action spaces, the goal is to find a near-optimal strategy through iterative training of the network, guided by the reward function, thereby obtaining higher rewards and improving the navigation success rate.

[0098] In addition, a knowledge transfer training method utilizing prior knowledge can be introduced to improve the convergence speed of the algorithm in relatively complex environments, such as... Figure 7 As shown. Because models typically need to be trained from scratch when facing different environments. However, DRL training is very time-consuming, and the model may struggle to converge in relatively complex environments. To address this issue, this invention proposes a knowledge transfer training method that allows the robot to be trained in a relatively simple environment, then its model and parameters can be transferred to a similar complex environment, and training can begin from that point, thereby accelerating training in complex environments.

[0099] Specifically, the navigation model is first trained in a relatively simple environment. In a simple environment, the model can more easily learn basic navigation knowledge and patterns. The network parameters of the trained navigation model are then used as training parameters. (The network parameters of the model trained in the simple environment contain the information learned in that environment and are used as parameters for subsequent training.) The navigation model is then transferred to a more complex environment, and these training parameters are used as the initial weights for further training. The goal is to leverage the knowledge and parameters learned in the simple environment to help the model learn and adapt more quickly and effectively in complex environments.

[0100] In another embodiment, the PER-D3QN algorithm serves as a navigation model for the robot's discrete actions, and its training process can be as follows: Figure 8 As shown, the robot inputs state information into the current value network and selects the maximum Q value based on the output of the current value network. The corresponding action. The robot performs the action and receives a reward. Then, the current state is... The selected action The state at the next moment and rewards Samples used as experience are placed in the experience replay buffer. During network training, sample data is selected according to sample priority, and the parameters of the current value network are copied to the target value network every n time steps. Finally, the Q-values ​​of the current and target value networks are used to calculate the loss function. After sufficient training, the parameters of the current value network will be close to optimal. The execution process of PER is as follows: Figure 9 As shown.

[0101] In another embodiment, the AVG-DNPER-DDPG algorithm, as a navigation model for continuous robot movements, requires more precise parameters to support more accurate navigation paths. Therefore, the AVG-DNPER-DDPG algorithm needs further training.

[0102] In one feasible approach, within the commentator's current value network, the network parameters are iteratively updated based on the minimum of the variance of the loss function, whose mean squared error is as follows:

[0103] ;

[0104] These are network parameters. Indicates the number of samples. It is expected value, It is the actual calculation value.

[0105] For the current value network of the actor, the network parameters are iteratively updated according to the loss gradient of the policy, as shown below:

[0106] ;

[0107] These are strategy parameters. It is the sample size. This represents a quantity related to the strategy and the Q-value. This represents the gradient with respect to the policy.

[0108] In another feasible approach, a critic network is added to the existing actor-critic network architecture of DDPG to assist training. During each update, the minimum Q-value between the two critic networks is used as the action estimate, thereby reducing the probability of local optima. Then, a dual-actor network is introduced to mitigate the value underestimation caused by the dual-critic network. The action with the highest value between the two actor networks is selected for updating, stabilizing the algorithm's training process. By analyzing the causes of target approximation errors in the policy network, this invention averages the estimates from multiple critic networks for the same action, using the average as the action estimate to reduce the impact of errors. Traditional PER tends to collect empirical data with large absolute values ​​of temporal difference errors, but this approach leads to insufficient sample partitioning and slow algorithm training. Some empirical data with small absolute values ​​of TD-error but high reward values ​​can also improve the robot's learning efficiency, but these data have low priority, resulting in a low probability of being collected. This invention generates two TD-error values ​​by using two Critic networks and sums them with the current action reward to prioritize samples. It then uses an averaging method to process the two value functions in the dual Critic networks and further optimizes the algorithm by combining them with a dual actor network.

[0109] For example, by introducing a dual-actor-dual-commentator network architecture, the minimum Q-value between the two commentator networks at each update is obtained as the estimated Q-value of the action. The calculation formula is as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] Indicates at time The reward It is the state in the next moment. Indicates in Based on strategy parameters The action obtained, and These are the action-value functions, or Q-values, calculated by two different commentator networks. The value estimates are then calculated for the two different paths. and Then take the minimum value. This can reduce overestimation errors and make the estimation of future actions more accurate.

[0114] The dual-commenter network generates the minimum Q-value to reduce overestimation error, while the dual-actor network selects the action with the highest value among the two actions at the minimum Q-value to reduce the impact of underestimation of Q-value and improve the robustness of the network structure.

[0115] Building upon the above, we further incorporate PER (Performance-Based Learning) for training. The core idea of ​​PER is to train the neural network in DQN by randomly sampling a certain number of learning experiences, rather than relying solely on the information from the last action. In a reinforcement learning environment, the state... The intelligent agent under the command, through actions It acts on the environment and obtains feedback values. Current state It will also transform into a new state. The process information is presented in quadruplets. The information, stored in the form of >, is the agent's learning experience in the experience replay buffer. The agent's experiences over a continuous period are interconnected. By randomly sampling from the experience replay buffer, the correlation between the desired learning experiences can be reduced, thus helping the agent learn comprehensively and effectively.

[0116] Specifically, the current state Current action The state at the next moment and rewards The empirical data is placed into the experience replay buffer; then training samples are drawn from the experience replay buffer. Considering that directly calculating the priority of all samples in the experience pool would severely impact the algorithm's efficiency, this invention divides the sampling process into two steps: First, random sampling is performed from the experience replay buffer. There are 1 data point; where n represents the number of priority samples required by the network. If n is 1, it indicates that the current sampling method is uniform sampling. Size represents the size of the batch data. Secondly, from... From the data, n samples are collected according to priority as the training samples required by the current value network; the AVG-DNPER-DDPG algorithm is trained using these training samples to determine the parameters of the trained AVG-DNPER-DDPG algorithm.

[0117] Based on the sum of the absolute values ​​of the dual temporal differences generated by the two commentator networks in the AVG-DNPER-DDPG algorithm and the reward of the current state, the priority of empirical data is determined as follows:

[0118]

[0119]

[0120]

[0121] and These represent the state and action at time t. These are network parameters. and These represent the sum of the difference between the two critic networks and the reward, respectively. It is the reward for the current state. The updated critics' network is in status and actions Below, based on parameters The calculated estimated value of the action, The critics' network before the update is in state and actions Below, based on parameters The calculated estimated value of the action, For a specific constant, Prioritize empirical data.

[0122] S106. If the target distance is less than a preset threshold, determine that the robot has completed navigation.

[0123] In this step, an obstacle environment was constructed based on a 3D simulator to verify robot navigation.

[0124] Two were created in the 3D simulator Gazebo. Obstacle environments, such as Figure 10 As shown: Figure 10 In the middle, 'a' represents a static obstacle environment (Scenario 1). Figure 10 In the middle b, the dynamic obstacle environment (Scenario 2) is represented. In the environment, the cylinder with a radius of 0.15m represents an obstacle, and the cylinder in the dynamic obstacle environment moves counterclockwise around the center point of the environment at a certain speed; the square frame represents a fence, and hitting the fence is equivalent to colliding with an obstacle; the square in the fence is the target point; the line in the fence represents the laser beam emitted by the LiDAR (Light Detection and Ranging) device.

[0125] During the training simulation, the robot starts from different initial points and navigates to randomly generated target points in an open area of ​​the environment. Through the interaction between the robot and the environment, and the constraints of rules during navigation, the robot's decision-making quality is continuously improved, and finally, a navigation path is determined based on the optimized decisions.

[0126] The above method first uses PER-D3QN to control discrete actions, effectively mitigating Q-value overestimation through its priority sampling and dual-Q network structure. Simultaneously, DDPG is employed to handle continuous actions, overcoming the limitations of traditional DRL algorithms in continuous control tasks. An artificial potential field is constructed by acquiring the target and obstacle distances in real time. The attractive potential field accelerates approach to the target, while the repulsive potential field dynamically avoids obstacles. The change in potential energy is used as the reward function, directly correlated with the navigation target (minimizing potential energy), avoiding biases inherent in manually designed reward functions. This discrete-continuous action hybrid control mode can adapt to complex environments. Combined with the smooth gradient guidance provided by the potential field, it reduces training oscillations and improves convergence efficiency. When the target distance is less than a threshold, the system can autonomously determine navigation completion, forming closed-loop control. This approach significantly addresses the performance issues of traditional DRL algorithms caused by Q-value overestimation, improper reward function design, and limited action space.

[0127] Figure 11 A block diagram of a robot navigation device provided by the present invention is shown below. Figure 11 As shown, it includes:

[0128] The acquisition module 201 is used to acquire in real time the target distance between the robot and the endpoint position and the obstacle distance between the robot and the nearest obstacle.

[0129] Module 202 is used to construct the PER-D3QN algorithm and the DDPG algorithm to control the robot's discrete and continuous actions, respectively; to determine the attractive potential field based on the target distance and the repulsive potential field based on the obstacle distance; and to construct an artificial potential field based on the attractive and repulsive potential fields. By combining the PER-D3QN algorithm for discrete action navigation and the DDPG algorithm for continuous action navigation, a unified navigation model is constructed based on the artificial potential field.

[0130] The navigation module 203 is used to determine the potential energy change of the robot based on the artificial field potential, and uses the potential energy change of the robot at each step as a reward function to guide the robot to move in the direction of decreasing potential energy; when the target distance is less than a preset threshold, the robot is determined to have completed navigation.

[0131] Using the aforementioned device, discrete actions are first controlled by PER-D3QN, whose priority sampling and dual-Q network structure effectively alleviate the problem of Q-value overestimation. Simultaneously, DDPG is employed to handle continuous actions, overcoming the limitations of traditional DRL algorithms in continuous control tasks. An artificial potential field is constructed by acquiring the target distance and obstacle distance in real time. The attractive potential field accelerates approach to the target, while the repulsive potential field dynamically avoids obstacles. The change in potential energy is used as the reward function, directly correlated with the navigation target (minimizing potential energy), avoiding the bias of manually designed reward functions. This discrete-continuous action hybrid control mode can adapt to complex environments. Combined with the smooth gradient guidance provided by the potential field, it can reduce training oscillations and improve convergence efficiency. When the target distance is less than a threshold, the system can autonomously determine that navigation is complete, forming closed-loop control. This approach significantly addresses the performance problems of traditional DRL algorithms caused by Q-value overestimation, improper reward function design, and limited action space.

[0132] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method for robot navigation.

[0133] This specification also provides a computer device, which, at the hardware level, includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above-mentioned functions. Figure 1 The provided method for robot navigation.

[0134] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] It should be noted that the above-described specific embodiments enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for robot navigation, characterized in that, include: Real-time acquisition of the target distance between the robot and the endpoint, and the distance to the nearest obstacle; The PER-D3QN algorithm is constructed to control the discrete movements of the robot, and the DDPG algorithm is constructed to control the continuous movements of the robot; wherein, the PER-D3QN algorithm is an algorithm formed by introducing the priority playback technique PER on the basis of the dual-competitive deep Q network D3QN. An attractive potential field is determined based on the target distance, and a repulsive potential field is determined based on the obstacle distance. An artificial potential field is constructed based on the attractive and repulsive potential fields. By combining the PER-D3QN algorithm for discrete motion navigation and the DDPG algorithm for continuous motion navigation, a unified navigation model is constructed based on an artificial potential field. The potential energy change of the robot is determined based on the artificial field potential, and the potential energy change of the robot at each step is used as a reward function to guide the robot to move in the direction of decreasing potential energy. If the target distance is less than a preset threshold, the robot is determined to have completed navigation; Using the artificial field potential as a reward function includes: The expression for the artificial potential field is: ； in, and These are the attractive potential field and the repulsive potential field, respectively. Characterizes the distance from the target robot to the target point. Characterizes the minimum distance between the target robot and the obstacle; A reward function is constructed based on the potential energy change of the artificial field potential, and the expression of the reward function is determined as follows: ； in, These are the potential energies from the previous step and the current step, respectively. The reward value obtained; A positive reward value indicates successful navigation. A negative reward value indicates navigation failure. The threshold for the target point. For the collision threshold, when Less than When the navigation to the target point is successfully reached; Less than When this occurs, it indicates a collision and navigation failure.

2. The robot navigation method according to claim 1, characterized in that, After constructing the unified navigation model, the unified navigation model is also trained, including: The robot navigation process is transformed into a Markov Decision Process (MDP). The robot obtains the reward of the MDP at each time step based on discrete or continuous actions, and calculates the cumulative discount reward in the MDP process to maximize the cumulative discount reward to update the parameters of the current unified navigation model; the cumulative discount reward is used to quantify the quality of the path.

3. A robot navigation device, characterized in that, The method applied to claim 1 includes: The acquisition module is used to acquire the distance between the robot and the target position at the destination and the distance between the robot and the nearest obstacle in real time; The module is used to construct discrete actions of the robot controlled by the PER-D3QN algorithm and continuous actions of the robot controlled by the DDPG algorithm; it determines the attractive potential field based on the target distance and the repulsive potential field based on the obstacle distance, and constructs an artificial potential field based on the attractive and repulsive potential fields; it combines the PER-D3QN algorithm for discrete action navigation and the DDPG algorithm for continuous action navigation to construct a unified navigation model based on the artificial potential field. The navigation module is used to determine the robot's potential energy change based on the artificial field potential, and uses the potential energy change of the robot at each step as a reward function to guide the robot to move in the direction of decreasing potential energy; when the target distance is less than a preset threshold, the robot is determined to have completed navigation.

4. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 2.

5. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 2.

Images