A method to accelerate robot navigation efficiency

By combining external and intrinsic rewards, this robot navigation method solves the problems of reward sparsity and low exploration efficiency in deep reinforcement learning, and achieves efficient and safe navigation in complex environments. It is suitable for mobile robot navigation in scenarios such as hotels, airports, and restaurants.

CN117490696BActive Publication Date: 2024-11-12GUANGZHOU CHUANGYUAN ROBOT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311373021.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-23
Publication Date
2024-11-12
Estimated Expiration
2043-10-23

AI Technical Summary

Technical Problem

Existing deep reinforcement learning-based robot navigation algorithms suffer from reward sparsity and low exploration efficiency in complex environments, leading robots to adopt overly conservative navigation strategies and reducing navigation efficiency.

Method used

Combining external and internal rewards, external rewards incentivize the robot to maintain a safe distance and avoid collisions, while internal rewards encourage exploration of new states through a curiosity module. Markov decision processes and deep learning are used to optimize strategies, and reward functions are set to improve navigation efficiency and safety.

Benefits of technology

Improving robot navigation efficiency and safety in complex environments involves accelerating the journey to the goal through external rewards and promoting the exploration of new strategies through intrinsic rewards, resulting in faster learning speeds and higher success rates.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117490696B_ABST
    Figure CN117490696B_ABST
Patent Text Reader

Abstract

The invention relates to the field of artificial intelligence application technology, and discloses a method for accelerating robot navigation efficiency, which specifically includes three steps: problem description and reinforcement learning modeling, reward function setting and training process design. The navigation efficiency and safety of the robot in a complex environment are maximized through the setting of rewards and the design of training processes. Through the design of external rewards, the robot will be externally encouraged to navigate to a target point as quickly as possible, thereby accelerating the navigation speed. Through the design of maintaining a safe distance and avoiding collisions, the safety of navigation is improved. Through the application of intrinsic rewards, the robot is rewarded for exploring new states, thereby prompting it to actively explore and discover new navigation strategies, thereby improving the robot's exploration efficiency. By using a deep learning algorithm to optimize the reward function and update the network weights, the learning efficiency and accuracy of the robot's navigation algorithm are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence application, and in particular relates to a method for accelerating robot navigation efficiency. Background Art

[0002] In recent years, with the vigorous development of artificial intelligence and the robotics industry, more and more robots have appeared in people's vision, such as hotels, airports, restaurants, shopping malls, etc. It is precisely because of the expansion of the application scope and user groups of robots that robots and people have to share the same environment, especially when robots are used in service scenarios. As a necessary and core function of mobile robots, navigation is particularly important in this scenario. Especially when robots navigate in dense crowds and complex environments, how to shuttle safely and freely among the crowds, and how to be efficient and abide by social etiquette is a big challenge, because human intentions are changeable and unknown.

[0003] Traditional robot navigation methods focus on obstacle avoidance, treating humans as obstacles and ignoring the interaction between humans and robots. Some of these methods set the same motion rules for humans and robots, such as RVO (Reciprocal Velocity Obstacles) and SFM (Social Force Model). However, these artificially designed rules may cause the robot to freeze when facing a crowded environment, because traditional methods only have passive obstacle avoidance and do not understand the environment.

[0004] Recently, deep reinforcement learning (DRL) has been used to solve this problem, which is better than previous model-based (traditional) algorithms. Many of these works use information about other agents in the environment as input, including their positions or velocities. For example, an algorithm was introduced in the multi-agent obstacle avoidance scenario: CADRL (Obstacle Avoidance Based on Deep Reinforcement Learning), which uses the DRL method in the multi-agent obstacle avoidance scenario. There are also some other methods that use long short-term memory (LSTM) to gain a better understanding of the environment. In social attention-based reinforcement learning (SARL), human-human interactions are modeled, and attention models are used to process information to enable robots to better understand the environment. There are also other methods that predict the weights of attention using graph neural networks, achieving better performance than previous motion prediction-based algorithms.

[0005] For the DRL algorithm, the processes of exploration (adding randomness to decisions when the strategy is unknown) and exploitation (improving the strategy based on knowledge learned from the environment) are crucial. However, most of the existing navigation algorithms based on deep reinforcement learning have the following two problems: First, due to the sparsity of rewards in reinforcement learning, it is very difficult to learn a good strategy; second, when navigating in a real environment, unknown states will appear at any time during navigation, not just at the beginning of training. Moreover, in some states that do not seem optimistic, very valuable actions may also be achieved. For example, in a state close to humans, very valuable operations (bypassing pedestrians) may occur. Therefore, when the robot adopts an overly conservative navigation strategy, the robot's navigation efficiency will be reduced, that is, it will fall into a local optimal solution. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art, the present invention proposes a method for accelerating the efficiency of robot navigation by combining external rewards and intrinsic rewards to overcome the problems of sparse rewards and low exploration efficiency in previous algorithms, and to obtain a strategy with faster learning speed, higher rewards and higher success rate in a shorter time.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for accelerating robot navigation efficiency, specifically comprising the following steps:

[0008] S1. Problem description and reinforcement learning modeling: The multi-agent obstacle avoidance problem is formulated as a sequential decision problem with partially observable states. There are n agents in the environment, including robots and pedestrians. Represents the state of the robot, including the observable state and unobservable state of the robot at time t. The observable state of the robot at time t includes: position (p x , p y ), speed (V x , V y ), radius r; the unobservable states of the robot at time t include: target point (g x , g y ) and the preferred speed v pref ; represents the observable state of pedestrian i; represents the observable state of the population in the environment;

[0009] Robots execute strategies Generate action a l From the status Navigate to your destination The robot does not know the destination and strategy of other pedestrians. Therefore, the state of all agents at time t is defined as:

[0010]

[0011] in Reason and The goal is to learn a policy Enable the robot to navigate safely to its destination;

[0012] The problem is defined as a Markov decision process; where the goal is to find the optimal policy π * ; The optimal strategy is the one that maximizes expected returns:

[0013]

[0014] S2. Reward function setting: In order to achieve this goal, a new reward function is set. The reward function is divided into external rewards and intrinsic rewards.

[0015] External rewards ex Defined as:

[0016]

[0017] where Δd g is the distance between the robot and the target, 0.2 is the minimum safe distance during navigation; μ i represents the distance between the i-th person and the robot, and N represents the number of humans;

[0018] The intrinsic reward adopts the intrinsic curiosity module, which uses the feature network φ to transform the current state s and the next state s of the environment t+1 Encoded into feature space φ(s) and φ(s t+1 ); the agent's horizontal state is converted to a state defined by a feature vector, which is the output of the network; the state in the feature space is used to predict the action to be taken The actual action a in the current state φ(s) i It is used to predict the next state in the feature space The network is then trained to minimize a i and predict actions error;

[0019] Therefore, the intrinsic reward r in is φ(s t+1 )and The mean squared error (MSE) between the two is calculated. When the agent visits unknown or unpredictable states, the mean squared error will be higher.

[0020] By combining the two rewards, the efficiency and safety of the robot's navigation in the crowd can be improved; the total reward function is: intrinsic reward r in Multiply by the hyperparameter β that controls its influence, plus the external reward r ex ; The formula is as follows:

[0021]

[0022] S3. Training process design: The training process is as follows:

[0023] 1. Initialize the environment, network, robot status and strategy;

[0024] 2. Obtain environmental information through the robot's built-in sensors;

[0025] 3. Extract the acquired environmental information and encode it;

[0026] 4. Determine whether the robot has reached the target point.

[0027] Preferably, in step S1, the optimal strategy π * To select the best action a for state s at a specific time t l , n * :s l →a i .

[0028] Preferably, in step S2, the external reward r ex represents the reward obtained from the external environment, which aims to encourage the agent to navigate towards the goal while avoiding collisions and keeping a safe distance from humans.

[0029] Preferably, the intrinsic reward in step S2 is used to encourage the agent to explore new states or reduce the uncertainty of predicting the consequences of actions.

[0030] Preferably, in step S33, the process of extracting and encoding the acquired environmental information is as follows:

[0031] 33.1. The robot outputs action a according to the initial strategy;

[0032] 33.2. The curiosity module outputs the predicted action a' based on the environmental information;

[0033] 33.2.1. Predict the next state s' of the environment based on action a', and wait for the actual environment s to be generated.

[0034] Preferably, in step S34, the process of determining whether the robot has reached the target point is as follows:

[0035] 34.1. If the target point has been reached, an external reward will be given and the next target point will be generated;

[0036] 34.1.2. Get the next state s of the environment;

[0037] 34.1.3. Calculate the MSE of s and s' generated by 33.2.1 and give an intrinsic reward;

[0038] 34.1.4. Update network weights by combining intrinsic rewards and extrinsic rewards;

[0039] 34.2. If the target point is not reached, determine whether the maximum step limit has been reached or a collision has occurred;

[0040] 34.2.1. If the maximum step limit is reached or a collision occurs, an external reward is given and the environment is reset.

[0041] Return to step 34.1.2;

[0042] 34.2.2. If the maximum number of steps is not reached or a collision occurs, further determine whether the maximum number of training rounds has been reached;

[0043] 34.2.2.1. If the maximum number of training rounds has not been reached, continue to update the network weights according to the total reward value and return to step 32;

[0044] 34.2.2.2. If the maximum number of training rounds is reached, the optimal strategy π is output and training ends.

[0045] The technical effects and advantages of the present invention are as follows: the navigation efficiency and safety of the robot in a complex environment are maximized through the setting of rewards and the design of training processes; through the design of external rewards, the robot is externally encouraged to navigate to the target point as quickly as possible, thereby speeding up the navigation speed; the distance between the robot and other pedestrians is taken into account in the external rewards, and the safety of navigation is improved by maintaining a safe distance and avoiding collisions; through the application of intrinsic rewards, the robot is rewarded for exploring new states, thereby prompting it to actively explore and discover new navigation strategies, thereby improving the robot's exploration efficiency.

[0046] By adopting Markov decision process modeling and solving the optimal strategy to maximize the expected return, the reinforcement learning algorithm can be fully utilized to realize the intelligent navigation of the robot. By using the deep learning algorithm to optimize the reward function and update the network weight, the learning efficiency and accuracy of the robot navigation algorithm are improved.

[0047] This method can be applied to different practical scenarios, such as hotels, airports, restaurants, shopping malls, etc. It can be applied to various mobile robots, such as the navigation functions of sweeping robots and service robots, and has broad practical application prospects. Compared with the deep reinforcement learning method, this method has a lower training cost and does not require too much hardware equipment, so it is more practical and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a training flow chart of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The following is combined with Figure 1 Provide further details of this application;

[0051] A method for accelerating robot navigation efficiency, comprising the following steps:

[0052] S1. Problem description and reinforcement learning modeling: The multi-agent obstacle avoidance problem is formulated as a sequential decision problem with partially observable states. There are n agents in the environment, including robots and pedestrians. Represents the state of the robot, including the observable and unobservable states of the robot at time t. The observable states include: position (p x , p y )、speed(v x , v y ), radius r; unobservable states include: target point (g x , g y ) and the preferred speed v pref .

[0053] represents the observable state of pedestrian i;

[0054] represents the observable state of the population in the environment;

[0055] Robots execute strategies Generate action a l From the status Navigate to your destination The robot does not know the destination and strategy of other pedestrians. Therefore, the state of all agents at time t is defined as:

[0056]

[0057] in Reason and The goal is to learn a policy Enable the robot to navigate safely to its destination;

[0058] The problem is defined as a Markov decision process; where the goal is to find the optimal policy π * , that is, at a specific time t, for state s i Choose the best action a i , π * :s i →a i ; The optimal strategy is the one that maximizes expected returns:

[0059]

[0060] S2. Reward function setting: In order to achieve this goal, a new reward function is set. The reward function is divided into external rewards and intrinsic rewards.

[0061] External rewards ex Represents the reward obtained from the external environment, which is intended to encourage the agent to navigate towards the goal. Agent: Intelligent agent, often described in reinforcement learning as a robot or other interactive person in the environment, while avoiding collisions and keeping a safe distance from humans, is defined as follows:

[0062]

[0063] where Δd g is the distance between the robot and the target, 0.2 is the minimum safe distance during navigation; μ i represents the distance between the i-th person and the robot, and N represents the number of humans;

[0064] Intrinsic rewards are used to encourage the agent to explore new states or reduce the uncertainty of predicting the consequences of actions. The intrinsic curiosity module (ICM) is used, which uses a feature network φ to transform the current state s and the next state s of the environment. t+1 Encoded into feature space φ(s) and φ(s t+1 ); the agent's horizontal state is converted to a state defined by a feature vector, which is the output of the network; the state in the feature space is used to predict the action to be taken The actual action a in the current state φ(s)l It is used to predict the next state in the feature space The network is then trained to minimize a l and predict actions error;

[0065] Therefore, the intrinsic reward r in is φ(s t+1 )and The mean squared error (MSE) between the two is calculated. When the agent visits unknown or unpredictable states, the mean squared error will be higher.

[0066] By combining the two rewards, the efficiency and safety of the robot's navigation in the crowd can be improved; the total reward function is: intrinsic reward r in Multiply by the hyperparameter β that controls its influence, plus the external reward r ex ; The formula is as follows:

[0067]

[0068] S3. Training process design: refer to Figure 1 , the training process is as follows:

[0069] 31. Initialize the environment, network, robot status and strategy;

[0070] 32. Obtain environmental information through the robot’s own sensors;

[0071] 33. Extract the acquired environmental information and encode it;

[0072] 33.1. The robot outputs action a according to the initial strategy;

[0073] 33.2. The curiosity module outputs the predicted action a' based on the environmental information;

[0074] 33.2.1. Predict the next state s' of the environment based on action a' and wait for the actual environment s to be generated;

[0075] 34. Determine whether the robot has reached the target point;

[0076] 34.1. If the target point has been reached, an external reward will be given and the next target point will be generated;

[0077] 34.1.2. Get the next state s of the environment;

[0078] 34.1.3. Calculate the MSE of s and s' generated by 33.2.1 and give an intrinsic reward;

[0079] 34.1.4. Update network weights by combining intrinsic rewards and extrinsic rewards;

[0080] 34.2. If the target point is not reached, determine whether the maximum step limit has been reached or a collision has occurred;

[0081] 34.2.1. If the maximum step limit is reached or a collision occurs, an external reward is given and the environment is reset.

[0082] Return to step 34.1.2;

[0083] 34.2.2. If the maximum number of steps is not reached or a collision occurs, further determine whether the maximum number of training rounds has been reached;

[0084] 34.2.2.1. If the maximum number of training rounds has not been reached, continue to update the network weights according to the total reward value and return to step 32;

[0085] 34.2.2.2. If the maximum number of training rounds is reached, the optimal strategy π is output and training ends.

[0086] The present invention aims to solve the problem of sparse rewards in reinforcement learning by using external rewards and internal rewards, and to improve the learning efficiency of the algorithm by using deep learning, and to apply it in robot navigation to solve the problem of poor navigation ability of robots in complex crowd scenes in real environments;

[0087] Currently, the best method is the robot navigation method based on deep reinforcement learning, which has problems such as high training cost, high hardware equipment requirements, and difficulty in actual implementation. This solution is a strategy based on the external rewards of traditional reinforcement learning plus the internal rewards of deep learning. It optimizes unnecessary deep learning parameter redundancy and is superior to traditional algorithms in terms of training speed, success rate, navigation time, etc. At the same time, due to the drive of the two rewards, the formed strategy is more in line with the actual scenario, so it has more advantages in actual implementation; in addition, the technology application product can be transplanted as a strategy algorithm in mobile robots for use as a navigation function, including but not limited to sweeping robots, service robots, etc.; application scenarios include but are not limited to hotels, airports, restaurants, and shopping malls.

[0088] In this invention, external rewards and intrinsic rewards are mainly combined to overcome the problems of sparse rewards and low exploration efficiency in previous algorithms. Therefore, in theory, all methods that can solve the problems of sparse rewards and low exploration efficiency can replace this solution to achieve the purpose of the invention. For example: reinforcement learning methods combined with imitation learning, sample enhancement methods, inverse reinforcement learning, etc.

[0089] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for accelerating robot navigation efficiency, characterized in that: The specific steps include: S1. Problem description and reinforcement learning modeling: The multi-agent obstacle avoidance problem is formulated as a sequential decision problem with partially observable states. There are n agents in the environment, including robots and pedestrians. Represents the state of the robot, including the observable state and unobservable state of the robot at time t. The observable state of the robot at time t includes: position (p x , p y ), speed (V x , V y ), radius r; the unobservable states of the robot at time t include: target point (g x , g y ) and the preferred speed v prof ; represents the observable state of pedestrian i; represents the observable state of the population in the environment; Robots execute strategies Generate action a l From the status Navigate to your destination The robot does not know the destination and strategy of other pedestrians. Therefore, the state of all agents at time t is defined as: in Reason and The goal is to learn a policy Enable the robot to navigate safely to its destination; The goal is to find the optimal strategy π * : S2. Reward function setting: The reward function is divided into external rewards and intrinsic rewards; External rewards ex Defined as: where Δd g is the distance between the robot and the target, 0.2 is the minimum safe distance during navigation; μ i represents the distance between the i-th person and the robot, and N represents the number of humans; The intrinsic reward adopts the intrinsic curiosity module, which uses the feature network φ to transform the current state s and the next state s of the environment t+1 Encoded into feature space φ(s) and φ(s t+1 ); the agent's horizontal state is converted to a state defined by a feature vector, which is the output of the network; the state in the feature space is used to predict the action to be taken The actual action a in the current state φ(s) l It is used to predict the next state in the feature space Then train the network to reduce a l and predict actions error; Therefore, the intrinsic reward r ir is φ(s t+1 )and The mean square error between By combining the two rewards, the efficiency and safety of the robot's navigation in the crowd can be improved; the total reward function is: intrinsic reward r in Multiply by the hyperparameter β that controls its influence, plus the external reward r ex ; The formula is as follows: S3. Training process design:

31. Initialize the environment, network, robot status and strategy; 32. Obtain environmental information through the robot’s own sensors; 33. Extract the acquired environmental information and encode it; 34. Determine whether the robot has reached the target point.

2. A method for accelerating robot navigation efficiency according to claim 1, characterized in that: In step S1, the optimal strategy π * At a specific time t, for state s i Choose the best action a l , π * :s i →a i .

3. A method for accelerating robot navigation efficiency according to claim 1, characterized in that: In step S2, the external reward r ex represents the reward obtained from the external environment, which aims to encourage the agent to navigate towards the goal while avoiding collisions and keeping a safe distance from humans.

4. A method for accelerating robot navigation efficiency according to claim 1, characterized in that: The intrinsic reward in step S2 is used to encourage the agent to explore new states or reduce the uncertainty of predicting the consequences of actions.

5. The method for accelerating robot navigation efficiency according to claim 1, characterized in that: In step S33, the process of extracting and encoding the acquired environmental information is as follows: 33.

1. The robot outputs action a according to the initial strategy; 33.

2. The curiosity module outputs the predicted action a' based on the environmental information; 33.2.

1. Predict the next state s' of the environment based on action a', and wait for the actual environment s to be generated.

6. A method for accelerating robot navigation efficiency according to claim 1, characterized in that: In step S34, the process of determining whether the robot has reached the target point is as follows: 34.

1. If the target point has been reached, an external reward will be given and the next target point will be generated; 34.1.

2. Get the next state s of the environment; 34.1.

3. Calculate the MSE of s and s' generated by 33.2.1 and give an intrinsic reward; 34.1.

4. Update network weights by combining intrinsic rewards and extrinsic rewards; 34.

2. If the target point is not reached, determine whether the maximum step limit has been reached or a collision has occurred; 34.2.

1. If the maximum number of steps is reached or a collision occurs, an external reward is given and the environment is reset, and then return to step 34.1.2; 34.2.

2. If the maximum number of steps is not reached or a collision occurs, further determine whether the maximum number of training rounds has been reached; 34.2.2.

1. If the maximum number of training rounds has not been reached, continue to update the network weights according to the total reward value and return to step 32; 34.2.2.

2. If the maximum number of training rounds is reached, the optimal strategy π is output and training ends.